Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis
Authors: Yuxi Ren
Xin Xia
Yanzuo Lu
Jiacheng Zhang
Jie Wu
Pan Xie
XING WANG
Xuefeng Xiao
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Model
Consistency Distillation
Human-Feedback Learning
Primary area: diffusion_based_models

Abstract: Recently, a series of diffusion-aware distillation algorithms have emerged to alleviate the computational overhead associated with the multi-step inference process of Diffusion Models (DMs). Current distillation techniques often dichotomize into two distinct aspects: i) ODE Trajectory Preservation; and ii) ODE Trajectory Reformulation. However, these approaches suffer from severe performance degradation or domain shifts. To address these limitations, we propose Hyper-SD, a novel framework that synergistically amalgamates the advantages of ODE Trajectory Preservation and Reformulation, while maintaining near-lossless performance during step compression. Firstly, we introduce Trajectory Segmented Consistency Distillation to progressively perform consistent distillation within pre-defined time-step segments, which facilitates the preservation of the original ODE trajectory from a higher-order perspective. Secondly, we incorporate human feedback learning to boost the performance of the model in a low-step regime and mitigate the performance loss incurred by the distillation process. Thirdly, we integrate score distillation to further improve the low-step generation capability of the model and offer the first attempt to leverage a unified LoRA to support the inference process at all steps. Extensive experiments and user studies demonstrate that Hyper-SD achieves SOTA performance from 1 to 8 inference steps for both SDXL and SD1.5. For example, Hyper-SDXL surpasses SDXL-Lightning by +0.68 in CLIP Score and +0.51 in Aes Score in the 1-step inference.

Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis
Yuxi Ren
Xin Xia
Yanzuo Lu Xing Wang
Jiacheng Zhang Xuefeng Xiao
Jie Wu
Pan Xie
ByteDance
Project Page: https: http://hpperrrree github........... io/
Hyper SOXL
(NFE=1)
HYPER
SOXL SOXL-Bo Base ONFE- 50)
SDXL- Lightnnin (NFE=1)
SDXL- Turbo (NFE=1)
HYPER 6E
Figure 1. The visual comparison between our Hyper-SDXL and other methods. From the first to the fourth column, the prompts for these images are (1) dog wearing white t-shirt, with the word "hyper" written (2) abstract beauty, approaching perfection, pure form, golden ratio, minimalistic, unfinished.... (3) a crystal heart laying on moss in serene zen garden ... (4) anthropomorphic art of scientist stag, victorian inspired clothing by krenz cushart... respectively.
Abstract
Recently, a series of diffusion-awarr aware distillation algorithms have emerged to allevi- ate the computational overhead associated with the multi-step inference process Diffusion Models (DMs). Current distillation techniques often dichotomize into two distinct aspects: i) ODE Trajectory Preservation; and ii) ODE Trajectory Reformulation. However, these approaches suffer from severe performance degra- dation domain shifts. To address these limitations, propose Hyper-SD,
Project Lead. Correspondence to <xiaoxucfeng ailab hiwpreffeggaabb@eeyyycce....cmmm 3
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

novel framework that synergistically amalgamates the advantages of ODE Trajec- tory Preservation and Reformulation, while maintaining near-lossless performance during step compression. Firstly, we introduce Trajectory Segmented Consistency Distillation to progressively perform consistent distillation within pre-defined time- step segments, which facilitates the preservation of the original ODE trajectory from a higher-order perspective. Secondly, we incorporate human feedback learn- ing to boost the performance of the model in a low-step regime and mitigate the performance loss incurred by the distillation process. Thirdly, we integrate score distillation to further improve the low-step generation capability of the model and offer the first attempt to leverage a unified LoRA to support the inference process at all steps. Extensive experiments and user studies demonstrate that Hyper-SD achieves SOTA performance from to 8 inference steps for both SDXL and SD1 .... For example, Hyper-SIDXI. surpasses SDXI--Lightning by +0.68 in CLIP Score and +0.5/ in Aes Score in the 1-step inference.
Introduction
Diffusion models (DMs) have gained significant prominence in the field of Generative Al
9 25
20 24], but they are burdened by the computational requirements(36 12] associated with multi-step inference procedures [27 10). To overcome these challenges and fully exploit the capabilities of
DMs, several distillation methods have been proposed [27 32 46 [0 16 29 44 40 28). which can
be categorized into two main groups: sngjectory-preservie distillation and
distillation.
Trajectory-preeeriing distillation techniques are designed to maintain the original trajectory of an ordinary differential equation (ODE) (27 46). The primary objective of these methods is to enable student models to make further predictions on the flow and reduce the overall number of inference steps. These techniques prioritize the preservation of similarity between the outputs of the distilled model and the original model. Adversarial losses can also be employed to enhance the accuracy of supervised guidance in the distillation process [14]. However, it is important to note that, despite their benefits, enjjectory-preserve distillation approaches may suffer from a decrease in generation quality due to inevitable errors in model fitting.
rrfooulagy methods directly utilize the endpoint of the ODE flow or real images as the primary source of supervision, disregarding the intermediate steps of the trajectory [16] 29 28 By reconstructing more efficient trajectories, these methods can also reduce the number inference steps.
approaches enable the exploration of the model's potential within limited number of steps, liberating it from the constraints of the original trajectory. However, it can lead to inconsistencies between the accelerated model and the original model's output domain, often resulting in undesired effects.
To navigate these hurdles and harness the full potential of DMs, we present an advanced framework that adeptly combines Firstly, we proposed trajectory segmented consistency distillation (TSCD), which divides the time steps into segments and enforces consistency within each segment while gradually reducing the number of segments to achieve all-time consistency. This approach addresses the issue of suboptimal consistency model performance caused by insufficient model fitting capability and accumulated errors in inference. Secondly, we leverage human feedback learning techniques [3 44 233 to optimize the accelerated model, modifying the ODE trajectories to better suit few-step inference. This results in significant performance improvements, even surpassing the capabilities of the original model in some scenarios. Thirdly, we enhanced the one-step generation performance using score distillation 35 achieving the idealized all-time consistent model via unified LoRA.
peeettori- and eujectory-reformullan distillation techniques.
In summary, our main contributions are summarized as follows: 1) Accelerate: we propose TSCD that achieves a more fine-gr -eniiird and high-order consistency distillation approach for the original score-based model. 2) Boost: we incorpoate human feedback learning to further enhance model performance in low-steps regime. 3) Unify: we provide unified LORA as the all-time consistency model and support inference at all NTEs. 4) Performance: Hyper-SD achieves SOTA performance in low-steps inference for both SDXL and SD1.5.

---

2 Preliminaries
For completeness, the preliminaries diffusion model are provided in Appendix A
2.1 Diffusion Model Distillation
As mentioned in Sec. | current techniques for distilling Diffusion Models (DMs) can be broadly categorized into two approaches: one that preserves the Ordinary Differential Equation (ODE)
trajectory 27 32 46
10, and another that reformulates it [29 40 28)
Here, we provide concise overview of some representative categories of methods. For clarity, define the teacher model as fteas the student model as fstu. noise e, prompt condition off-the-shelf ODE Solver as "(,,,,),, the total training timesteps as T, the num of inference timesteps as NN the noised trajectory point I; and the skipping-step s, where to < t1 < t T, t 1 = 8, n uniformly distributed over (1,2...., N 1}..
Progressive Distillation. Progressive Distillation (PD) 27 trains the student model fata approximate the subsequent flow locations determined by the teacher model frea over sequence of steps.
Considering a 2-step PD for illustration, the target prediction ãt,, by frea is obtained through the following calculations:
Bin-1 = (Iee Seea (Jen la c)2 (x=-));
iin-2 = V (St Stea (ir, -n 11 C)2 12);
(2)
And the training loss is
LPD ||:t,_-- vD (xt,, Satu (Dt,, bny c), 22)2
(3)
Consistency Distillation. Consistency Distillation (CD)
32 directly maps It, along the ODE
trajectory to its endpoint zig The training loss defined as
LCD = (Ttt, L, c), 0) 1/ ( 1 Sotu (fe, in c), 0)
where fai, is the exponential moving average(EMA) of far and i.._, is the next flow location estimated by frea with the same function as Eq. (
The Consistency Trajectory Model (CTM) [O was introduced to minimize accumulated estimation errors and discretization inaccuracies prevalent in multi-step consistency model sampling. Diverging from targeting the endpoint Zo, CTM targets any intermediate point Itand within the range S tend I tn-1, thus redefining the loss function as:
LCTM (3r,,, ( In c)2 bend) - (fr,,, 13 J (Br,, l c)2 lend)
(5)
Adversarial Diffusson Distillation. In contrast to PD and CD, Adversarial Distillation (ADD), proposed in SDXL-Turbo [2 and SD3-Turbo 28), bypasses the ODE trajectory and directly focuses
on the original state 2g using adversarial objective. The generative and discriminative loss components
are computed as follows:
CADD = -B(((V((,,, Sstu bn 0))
(6)
CRDD = (DD,,,,, (It,, tn c), 0))] E [D
(7)
where D denotes the discriminator, tasked with differentiating between 20 and V (xt,, Jitn::sttuae (1t,, bm; c), 0). The target zip can be sampled from real synthesized data.
Score Distillation Sampling. Score distillation Compling(SDS)) 21 was integrated into diffusion distillation in SDXL-Turbo 22) and Diffusson Matching Distillation (MM)) SDXL-Turbo (9) utilizes frea to estimate the score to the real distribution, while [MD further introduced a fake distribution simulator fake to calibrate the score direction and uses the output distribution of the original model as the real distribution, thus achieving one-step inference.
Leveraging the DMD approach, the gradient of the Kullback-Leibler (KL) divergence between the real and fake distributions is approximated by the equation:
VDKL
E
(-(reat((z) - ftake (fttu)
(8)
(0.
(x)
where is a random latent variable sampled from standard normal distribution. This methodology enables the one-step diffusion model to refine generative process, minimizing the KI. divergence to produce images that progressively closer to the teacher model's distribution.

---

2.2 Human Feedback Learning
ReFL [37 has been proven be an effective method learn from human feedback designed for diffusion models. It primarily includes two stages: (1) reward model training and (2) preference fine-tuning. the first stage, given the human preference data pair, Tw (preferred generation) and æ (unpreferred one), reward model ro is trained via the loss:
(()) -E( (log(a
ro(c,):)))
where D denotes the collected feedback data, a(-) represents the sigmoid function, and corresoonds to the text prompt. The reward model re is optimized to produce reward scores that align with human preferences. In the second stage, ReFL starts with an input prompt c, and a randomly initialized latent 27 = 2. The latent is then iteratively denoised until reaching a randomly selected timestep tn [tteft, tright]. when denoised image x is directly predicted from zt,,. The tie ft and fright are predefined boundaries. The reward model is then applied to this denoised image, generating the expected preference score re x)), which used fine-tuned the diffusion model:
[(0)reft Bes Pepteeooreg c)]
(10)
3 Method
In this study, we have integrated both the ODE-preserve and ODE-reformulate distillation techniques into unified framework, yielding significant advancements in accelerating diffusion models. In Sec.3.11 we propose an innovative approach to consistency distillation that employs a time-steps segmentation strategy, thereby facilitating trajectory segmented consistency distillation. Sec. 3.2 we incorporate human feedback learning techniques to further enhance the performance of accelcrated diffusion models. In Sec. 3.3 we achieve all-time consistency including one-step by utilizing the score-based distribution matching distillation.
3.1 Trajectory Segmented Consistency Distillation
Both Consistency Distillation (CD) 32 and Consistency Trajectory Model (CTM) 10] aim to transform a diffusion model into consistency model across the entire timestep range (0, T) through single-stage distillation. However, these distilled models often fall short of optimality due limita- tions in model fitting capacity. Drawing inspiration from the soft consistency target introduced in CTM, we refine the training process by dividing the entire time-steps range [o, T" into segments and performing segment- wise consistent model distillation progressively.
In the first stage, set k = 8 and the original diffusion model to initiate fstu and frea. The starting timesteps in are uniformly and randomly sampled from (ti, f2...., -1- We then sample ending timesteps tend [to, tn where is computed as:
bn
T
to
X
(11)
A
and the training loss is calculated as:
LAscd
(Ftn En, c), tend), V (iita-1) (ftar -ott-i.nn-- In tn-1, ten
(12)
where - refers to Eq. ( and Saru denotes the Exponential Moving Average (EMA) of Jstu-
Subsequently, we resume the model weights from the previous stage and continue to train fatas progressively reducing & to |4, 2, 1]. It is noteworthy that & = 1 corresponds to the standard CTM training protocol. For the distance metric d, we employ hybrid of adversarial loss, as proposed in sdxl-lightning [4], and Mean Squared Error (MSE) Loss. Empirically, we observe that MSE Loss more effective when the predictions and target values are proximate (e.g., for ie 8, 4), whereas adversarial loss proves more precise as the divergence between predictions and targets increases (e.g., for = 2, 1). Accordingly, we dynamically increase the weight of the adversarial loss and diminish that of the MSE loss across the training stages. Additionally, we have integrated noise perturbation mechanism g to reinforce training stability. Take the two-stage Trajectory Segmented Consistency Distillation (TSCD) process as example. As shown in Fig. 2 the first stage exccutes

---

TSCD
0BI Salver
Skg
TSC
Feur- Seur-Su
Comiistout ODE
C.LODE Solver
ODDE
TSCD
Consittett ODF
ODE Subver
112
Fext
10
Eext
82
Figure 2. An illustration of the two-stage Trajectory Segmented Consistency Distillation. The first stage involves consistency distillation two separate time segments: [0. F] and [5. I] to obtain two segments consistency ODE. Then, this ODE trajectory is adopted to train a global consistency model in the subsequent stage.
independent consistency distillations within the time segments [0, 31 and [3.7]. Based on the previous two-segment consistency distillation results, global consistency trajectory distillation is then performed.
The TSCD method offers two principal advantages: Firstly, fine-grained segment distillation reduces model fitting complexity and minimizes errors, thus mitigating degradation in generation quality, Secondly, it ensures the preservation of the original ODE trajectory. Models from each training stage can be utilized for inference at corresponding steps while closely mirroring the original model's generation quality. We illustrate the complete procedure of Trajectory Segmented Consistency Distil- lation in Appendix [ It is worth noting that, by utilizing Low-Rank Adaptation(LoRA) technology. we train TSCD models as plug-and-pla plugins that can be used instantly.
3.2 Human Feedback Learning
In addition to the distillation, we propose to incorporate feedback learning further to boost the performance of the accelerated diffusion models. In particular, we improve the generation quality of the accelerated models by exploiting the feedback drawn from both human aesthetic preferences and existing visual perceptual models. For the feedback aestheties, utilize the LAION aesthetic predictor and the aesthetic preference reward model provided by ImageReward 37 toward the higher aesthetic generation as:
to steer the model
((0)aes aes
Σ Beg Aplag [ReLU(aa c))l
(13)
where ra is the aesthetic reward model, including the aesthetic predictor of the LAION dataset and ImageReward model, e the textual prompt and as together with ReLU function works hinge loss.
Beyond the feedback from aesthetic preference, notice that the existing visual perceptual model embedded in rich prior knowledge about the reasonable image can also serve as good feedback provider. Empirically,
found that the instance segmentation model guide the model to generate entities with reasonable structure. To be specific, instead of starting from a random initialized latent, we first diffuse the noise on an image xo in the latent space to Iy according to Eq. (16), and then, we execute denoise iteratively until specific timestep de and directly predict To similar [37]. Subsequently, we leverage perceptual instance segmentation models to evaluate the performance of structure generation by examining the perceptual discrepancies between the ground truth image instance annotation and the predicted results on the denoised image as:
C(I) percep
E Ho al(m)((
Linetance ((m GT
(14)
where my is the instance segmentation model(e.g. SOLO (4) can capture the structure defect of the generated image more accurately and provide more targeted noteworthy that besides the instance segmentation model, other perceptual models are also applicable and we are actively investigating the utilization of advanced large visual perception models(e.g. SAM) to provide enhanced feedback learning. Such perceptual models can work as complementary feedback for the subjective aesthetic focusing more on the objective
The instance segmentation model
feedback signal.
5

---

generation quality. Therefore, we optimize the diffusion models with the feedback signal as:
[(0) feedback L(0)aes +
(15)
Human feedback learning can improve model performance but may unintentionally alter the output domain, which always desirable. Therefore, we also trained human feedback learning knowl- edge as plugin using LoRA technology. By employing the LoRA merge technique with the TSCD LoRAs discussed in Section3.1I we can achieve flexible balance between generation quality and output domain similarity.
3.3 One-step Generation Enhancement
One-step generation within the consistency model framework not ideal due to the inherent limitations of consistency loss. As analyzed in Fig. the consistency distilled model demon- strates superior accuracy in guiding towards the trajectory endpoint zu at position Zy. Therefore, score distillation is suitable and efficient way to boost the one-step generation of our TSCD
XT
GT average score
Pred average score
Estimation
Error
Pred score
models.
Specifically, we advance one-step generation with an optimized Distribution Matching Dis- tillation (DMD) technique [40]. DMD en-
Figure 3. Score distillation comparison between score- based model and consistency model. The estimated score produced by the score-hased model may exhibit greater estimation error than the consistency model.
hances the model's output by leveraging two dis- tinct score functions: (x) from the teacher model's distribution and frake(2) from the fake
model. We incorporate a Mean Squared Error (MSE) loss alongside the score- score-based distillation to promote training stability. The human feedback learning technique mentioned in Sec. 3.2 is also integrated, fine-tuning our models to efficiently produce images of exceptional fidelity.
After enhancing the one-step inference capability of the TSCD model, consistency model. Employing the TCD scheduler(46), the enhanced model can perform inference from to 8 steps. Our approach eliminates the need for model conversion x0-prediction [I]]), enabling the implementation of the one-step LoRA plugin. We demonstrated the effectiveness of our one-step LoRA in Sec 4.3 Additionally, smaller time-step inputs can enhance the credibility of the one-step diffusion model in predicting the noise (7]. Therefore, we also employed this technique to train a dedicated model for single-step generation.
obtain an ideal global
Table 1. Comparison with other acceleration approaches.
4 Experiments
Support Arch.
CFG
One-Step
Method
Step
Free
UNet
LoRA
4.1
Implementation Details
PeRFlow
38
4+
SD15
No
No
No
Dataset. We use a subset of the
TCD 46
2+
SD15/XL
Yes
No
No
LAION (30 and COYO 6 datasets fol-
LCM 19
2+
SD15/XL
Yes
No
No
lowing SDXL-lightning training procedure Sec 3.1 and Sec
during the
Turha 29
1+
SD21/XI.
Yes
Yes
No
Lightning 14
1+
SDXL.
Yes
Yes
No
Ours
1+
SD15/XL
Yes
Yes
Yes
For the Human Feedback Learning in 3.2 we utilized the COC02017 train split dataset with instance annotations and captions for structure optimization.
Training Setting, For TSCD in See 3.1 we progressively reduced the time-steps segments number as 8 + 4 + 2 - in four stages, employing 512 batch size and learning rate 1e - 6. We take the SOLO 34 the instance segmentation model achieve feedback learning in Sec 3.2 Our training per stage costs around 200 1100 GPU hours. We trained LoRA instead of UNet for all the distillation stages for convenience, and the corresponding LoRA is loaded to process the human feedback learning optimization in Sec 3.2 For one-step enhancement in Sec 3.3 we trained the unified all-timesteps consistency LoRA with time-step inputs T 999 and the dedicated model for single-step generation with T 800.

---

Baseline Models. We conduct our experiments on the siiee--fffsiooo----- (SD15) [25] and stable- diffusion-xl-vy x---bb.-bbese (SDXL) 20]. To demonstrate the superiority of our method in acceleration, we compared our method with various existing acceleration schemes as shown in Tab.
Evaluation Metrics. We use the aesthetic predictor pre-trained on the LAION dataset and CLIP score(ViT--/32) to evaluate the visual appeal of the generated image and the text-to-image align- ment. We further include some recently proposed metrics, such as ImageReward score [37], and Pickscore M to offer a more comprehensive evaluation of the model performance. In addition to these, due to the inherently subjective nature of image generation evaluation, we conduct an extensive user study to evaluate the performance more accurately.
Table 2. Quantitative comparisons with state-of-the-art, SD15 and SDXL architectures. The best result is highlighted in bold.
4.2 Main Results
CLIP Score
Image Score Reward
Aes
Pick
Quantitative Comparison. We quan- titatively compare our method with both the baseline and inffusion-based distillation approaches in terms of ob- jective metrics. The evaluation is performed on COCO-Sk [1S dataset with both SDIS (512px) and SDXL (1024px) architectures. As shown in Tab. [ our method significantly out- performs the state-of-the-ae art across all metrics on both resolutions. In par- ticular, compared to the two baseline models, we achieve better aesthetics (including AesScore, ImageReward and PickScore) with only LoRA and fewer steps. As for the CLIPScore that evaluates image-text matching, we outperform other methods by +0.1 1 faithfully and are also closest to the baseline model, which demonstrates the effectiveness of our human feedback learning.
Model
Step
Type
Score
SD15-Base
25
25
UNet 31.88
5.26
0.18
0.217
SD15-LCM
19
4
LoRA 30,36
5.66
-0.37
0.212
SDI5-TCD
46
4
LoRA 30.62
5.45
-0.15
0.214
PeRFlow
38
4
UNet 30.77
5.64
-0.35
0.208
Hyper-SD15
LoRA 30.87
5.79
0.29
0.215
SDXL-Base
25
25
UNet 33.16
5.54
0.87
0.229
SDXL-LCM
19
LoRA 32.43
5.42
0.48
0.224
SDXL- TCD
46
4
LoRA 32.45
5.42
0.67
0.226
SDXL SDXL-Lightning [14]
4
LoRA 32.40
5.63
0.72
0.229
HypL
4
LoRA 32.56
5.74
0.93
0.232
SDrb-Turbo 29
UNet 32.33
5.33
0.78
0.228
SDXL-Lighniin [14]
UNet 32.17
5.34
0.54
0.223
Hyper-SDXL 1 UNet 32.85
5.85
1.19
0.231
Qualitative Comparison. We present comprehensive visual comparison with recent approaches, including LCM [1)), TCD [46), PeRFLow [38), Turbo (29 and Lightning [14. Our observations can be summarized as follows. (1) Thanks to the fact that SDXL has almost 2.6B parameters, the model is able to synthesis decent images in 4 steps after different distillation algorithms. Our method further utilizes its huge model capacity to compress the number of steps required for high-quality outcomes to step only, and far outperforms other methods in terms of style (a), aesthetics (b-c) and image-text matching (d) as indicated in Fig. 4 (2) On the contrary, limited by the capacity of SD15 model, the images generated by other approaches tend to exhibit severe quality degradation. While our Hyper-SD consistently yields better results across different types of user prompts, including photographic (a), realistic (b-c) and artstyles (d) as depicted in Appendix C.I (3) To further release the potential of our methodology, we also conduct experiments on the fully fine-tuning of SDXL model following previous works [4 29 As shown in Appendix C.2 our 1-Step UNet again demonstrates superior generation quality that far exceeds the rest of the opponents. Both in terms of colorization (a-b) and details (c-d), our images are more presentable and attractive when it comes to the real-world application scenarios.
User Study. To verify the effectiveness of our proposed Hyper-SD, we conduct extensive user study across various settings and approaches. As presented in Fig. 5 our method (red in left) obtains significantly more user preferences than others (blue in right). Specifically, our Hyper-SD15 has achieved more than a two-thirds ad advantage against the same architectures. The only exception is that SD2I-Turbo 2 was able to get significantly closer to our generation quality one-step inference by means of larger training dataset of SD21 model as well as fully fine-tuning. Notably, we found that we obtained a higher preference with less inference steps compared both the baseline SD15 and SDXL models, which once again confirms the validity of our human feedback learning. Moreover, our 1-Step UNet shows a higher preference than LoRA against the same UNet-based approaches (i.e. SDXI (29 and SDXL-LLihhtnig [14), which is also consistent with the analyses of previous quantitative and qualitative comparisons. This demonstrates the excellent scalability of our method when more parameters are fine-tuned.
7

---

SDXL-Base SONFE, CFG7.5 25 Steps
SDXL-LCM No CTG Steps
SDXL-ICD No CFG 4 Steps
SDXL-Lightning
Hyper-SDXL No CTG Step (LoRA)
No CFG 4 Steps
Steps
(a) baby Swan gruffiti
(b) close-up an Asian lady with sunglasses
(c) monkey making latte art
VI
RASCS
START
START
(d) The word 'START
Figure 4. Qualitative comparisons with LoRA-based approaches on SDXL architecture.
24.2%
Seps
Pap Lone
Hapin 4100
9100 LAbA
Lolki
55.4%
35.09
15.09
23.25
epparsuss
Pondiou
Uppan St0
Sum Lig wariing
See LMW
Stezs Wtte
12 UA1A)
21.%%
Step JRNi
S035-LCM
U3t0 JM
16.0%
Step LSF
3100 Jeat
30.0%
Stip Uiidd
41.1%
Step.
308
50
60
208
Proference Raia(()
Preferenio Rata(s)
Figure 5. The user study about the comparison hetween our method and other methods.
4.3
Ablation Study
In Tab. we provide ablation studies on TSCD and human feedback with quantitative evaluation.
Effect of TSCD. Without the utilization of human feedback, the results show that our proposed TSCD outperforms the baseline TCD [46] significantly when inference step lower, while the performance approaches the same as the step increases. This demonstrates the effectiveness of our progressive strategy that alleviates the training difficulties when the step extremely low.
Effeet of Human Feedback. verify the benefit of incorporating reward models, we also conduct experiments ablating on human feedback lcarning of different steps. As shown in Tab. 3 the

---

performance degradation caused by distillation process is well compensated after human feedback learning. Moreover, the image-text alignment (i.e. CLIPScore) and aesthetics (i.e. Others) evaluated on different steps are very similar, which better matches the definition of consistency model 332
Effect of One-Step Enhancement. In T we also re-implement the DMD [40] with human feedback. The results show that our TSCD model indeed exhibit less es- timation error than the original score- coore-based model with similar text-to- rext-to-image alignment and better aesthetic metrics, demonstrating higher generation quality and utility.
Table Ablation studies of TSCD and human feedback.
CLIP
Aes Score
Image Reward
Pick Score
Method
Step
Score
SDXL Architecture (w/o Human Feedback)
TCD [46]
2
32.36
5.62
0.29
0.217
TSCD (Ours)
2
32.49
5.58
0.64
0.222
TCD 4d
4
32.45
5.42
0.67
0.226
Unified LoRA. In addition the differ- ent steps of LoRAs proposed above, we note that our one-step LoRA can be consid- ered as a unified approach, since it is able reason about different number of steps (e.g. 1,2,4,8 as shown in Appendix C.3 and consistently generate high-qualine re- sults under the effect of consistency distilla- tion. For completeness, Tab. lab. alsso presents the quantitative results of different steps when applying the 1-Step unified LoRA. We can observe that there is no difference in image-text matching between different steps the CLIPScore evaluates, which means that user prompts are well adhered to. And as the other metrics show, the aes- thetics rise slightly as the step increases, which is expected after all the user choose based on the needs for efficiency. This would be of great convenience and practicality in real-world deployment sce- narios, since generally only model can be loaded per instance.
TSCD (Ours)
4
32.53
5.66
0.78
0.229
TCD (46)
8
32.47
5.78
0.76
0.229
TSCD (Ours)
8
32.46
5,85
0.77
0.229
SDXL Architecture (w/ Human Feedback)
DMD 40
1
32.70
5,58
0.82
0.223
TSCD (Ours)
32.59
5.69
1.06
0.226
TSCD (Ours)
2
32.61
5.84
1.04
0.232
TSCD (Ours)
4
32.56
5.74
0.93
0.232
TSCD (Ours)
8
32.56
5.89
0.93
0.232
Table 4. Quantitative results on unified LoRAs.
CLIP Score
Aes Score
Image Reward
Pick Score
Arch.
Step
8
30.73
5.47
0.53
0.224
SD15
4
31.07
5.55
0.53
0.224
512px
2
31.21
5,93
0.45
0.222
30.87
5.79
0.29
0.215
32.54
5.83
1.14
0.233
SDXL 1024px
4
32,51
5.52
1.15
0.234
2
32.59
5.71
1.15
0.234
1
32.59
5.69
1.06
0.226
Compatibility with ControlNet. Ap- pendix shows that our models are also compatible with ControlNet (45] We test the one-step unified SD15 and SDXL LoRAs the scribble [4] and canny [I] control images, respectively. And we can observe the conditions are well followed and the consistency of our unified LoRAs can still be demonstrated, where the quality of generated images under different inference steps always guaranteed.
Compatibility with Base Model. Appendix C.5 shows that our LoRAs can be applied to different base models. Specifically, we conduct experiments on anime 22, realistic [3] and artstyle [5] base models. The results demonstrate that our method has a wide range of applications, and the lightweight LoRA also significantly reduces the cost of acceleration.
Conclusion
We propose Hyper-SD, a unified framework that maximizes the few-step generation capacity of diffusion models, achieving new SOTA performance based on SDXL and SD15. By employing trajectory- segmeeted consistency distillation, we enhanced the trajectory preservation ability during distillation, approaching the generation proficiency of the original model. Then, human feedback learning and variational score distillation stimulated the potential for few-step inference, resulting in more optimal and efficient trajectory for generating models. We have open-sourced LoRAs for SDXL and SD15 from to steps inference, along with dedicated one-step SIXL. model, aiming to further propel the development of generative AI community. More discussions refer Appendix

---

References
hugging face.
https//hugcinnfcco.o///ffffrrrr
Contzolintt-camy-sdkll
Dreamshaper xl v2.1 turbo dpm++ sde stable diffusion checkpoint civitai.. httpa://civitai.com
modela/112902
Juggernaut xl jugg_x_
wtt___unnnfffioooyyoooo stable diffusion checkpoint civitai, https://civitai
com/mmodels/133005
[4] Ilyasvie//conttrl vilp:o_i__ii___d_el__ll_llll_ sd15 scribble hugging face.
http:://hugguugaaoo.///////vvv
control_.iipp_sdd__ecao sccribble
[5] Zavychromaxl v7.0 stable diffusion checkpoint civitai, https///eivtaallcc//oo//l///////
[6] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. http:///g.hhhhaacc///aaaaaaaaaaa
2022.
[7] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-\sigmaa, Weak-to-stroot training of diffusion transformer for 4k text-to-imas image generation, 2024.
[8] Nicholas Guttenberg. Diffusion with offset noise, 2023.
URL https://www. crosslabs.
btg//bboggdiffussonnnwiile ofli:iiiiiooiiiii----s--------ss
[9] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion prohabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[10] Dongjun Kim, Chich-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida. Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023.
[11] Yuval Kirstain, Adam Polyak, Uriel Singer. Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset user preferences text-to-image generation. 2023.
[12] Jiashi Li, Xin Xia, Wei Li, Huixia Li, Xing Wang, Xuefeng Xiao, Rui Wang. Min Zheng, and Xin Pan. Next-vit: Next generation vision transformer for efficient deployment realistic industrial scenarios, 2022.
[13] Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, and Chen Chen. Controlnct++: Improving conditional controls with efficient consistency feedback, 2024.
[14] Shanchuan Lin, Anran Wang, and Xiao Yang. Sdxl-lightning: Progressive adversarial diffusion distillation. arXiv preprint arXiv:2402. 139299 2024.
[15] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCY 2014: 13th European Conference. Zurich, Switzerland, September 6-12. 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[16] Xingchao Liu, Chengyue Gong, and Qiang Liu, Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03N: 2022.
[17] Yanzuo Lu, Meng Shen, Andy Ma, Xiaohua Xie, and Jian- Huang Lai. Minet: Mutual learning network with neighborhood invariance for universal domain adaptation. In AAAI, volume 38, pages 3900-3908, 2024.
[18] Yanzuo Lu, Manlin Zhang, Yiqi Lin, Andy Ma, Xiaohua Xie, and Jianhuang Lai. Improving pre-trained masked autoencoder via locality enhancement for person re-identification. In PRCV, volume 13535, pages 509-521, 2022.
[19] Simian Luo, Yigin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arX0v:2310.04378 2023.
[20] Dustin Podell, Zion English, Kyle Laccy, Andreas Blattmann, Tim Dockhor, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023.
[21] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint ar XiX:v:2209.111. 14988, 2022.
10

---

[22] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. International conference on machine learning, pages 8748-8763, PMLR. 2021.
[23] Yuxi Ren, Jie Wu, Yanzuo Lu, Huafeng Kuang. Xin Xia, Xionghui Wang. Qianqian Wang. Yixing Zhu, Pan Xie, Shiyin Wang, et al. Byteedit: Boost, comply and accelerate generative image editing, arXiv preprint arXiv:2404.00 048600 2024.
[24] Yuxi Ren, Jie Wu, Xuefeng Xiao, and Jianchao Yang. Online multi-granularity distillation for gan compression. Proceedings of the JEEE/CVF International Conference on Computer Vision (ICCV),
pages 6793-66666000 6803, October 2021.
[25] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10688 2022.
[26] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth:
Fine tuning text-to-image diffusion models for subject-driven generation. Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-22510, 2023.
[27] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffussio models. arXiv preprint arXiv:2202.00ST 2022.
[28] Axel Sauer. Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rom- bach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. arXiv preprint arXiv:2403. 12015, 2024.
[29] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint uxXiv22111.1044 2023.
[30] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large- scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294, 2022.
[31] Meng Shen, Yanzuo Lu, Yanxu Hu, and Andy J. Ma. Collaborative learning of diverse experts for source-free universal domain adaptation. In ACM MM, pages 2054-2065, 2023.
[32] Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. arXiv preprint arXiv:2303.0146) 2023.
[33] Yang Song, Jascha Sohl-Dickstein. Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 1rX1v:2011.13456, 2020.
[34] Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li. Solo: Segmenting objects by locations, 2020.
[35] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
[36] Xuefeng Xiao, Lianwen Jin, Yafeng Yang. Weixin Yang, Jun Sun, and Tianhai Chang. Building fast and compact convolutional neural networks for offfine handwritten chinese character recognition. Pattern Recognition, 72:72-81, 81, 2017.
[37] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
[38] Hanshu Yan, Xingehao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng. Perflow; Accelerating diffusion models with piecewise rectified flows. 2024.
[39] Binxin Yang. Shuyang Gu, Bo Zhang, Ting Zhang. Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar based image editing with diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18381-18391, 2023.
11

---

[40] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint urXiv:2311.18822 2023.
[41] Jiacheng Zhang. Jiaming Li, Xiangru Lin, Wei Zhang. Xiao Tan, Junyu Han, Errui Ding. Jingdong Wang. and Guanbin Li. Decoupled pseudo-labeling for semi-supervised monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16923-
16932, 2024.
[42] Jiacheng Zhang, Xiangru Lin, Minyue Jiang, Yue Yu, Chenting Gong, Wei Zhang, Xiao Tan, Yingying Li, Errui Ding, and Guanbin Li. A multi guanti-ga retrieval system for natural language- based vehicle retrieval. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 3216-3225, 2022.
[43] Jiacheng Zhang, Xiangru Lin, Wei Zhang, Kuo Wang, Xiao Tan, Junyu Han, Errui Ding, Jingdong Wang. and Guanbin Li. Semi-detr: semi-supervised object detection with detection transformers. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 23809-238118 2023.
[44] Jiacheng Zhang, Jie Wu, Yuxi Ren, Xin Xia, Huafeng Kuang, Pan Xic, Jiashi Li, Xuefeng Xiao, Weilin Huang, Min Zheng, Lean Fu, and Guanbin Li. Unifl: Improve stable diffusion via unified feedback learning, 2024.
[45] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IREE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[46] Jianbin Zheng, Minghui Hu, Zhongyi Fan, Chaoyue Wang, Changxing Ding, Dacheng Tao, and Tat-Jen Cham. Trajectory consistency distillation. arXiv preprint arXiv:2402. 19159, 2024.
12

---

A Preliminaries of Diffusion Model
Diffusion models (DMs), introduced by Ho et al. 19, consist of a forward diffusion process, described by a stochastic differential equation (SDE) [3), and reverse denoising process. The forward process gradually adds noise to the data, transforming the data distribution Paatu (2) into known distribution, typically Gaussian. This process is described by:
dæ, p(z, t)dt ott)dur.
(16)
where t E [0, TT, We represents the standard Brownian motion, p(::-) and (-)) are the drift and diffusion coefficients respectively. The distribution of x, sampled during the diffusion process is denoted as (x)), with the empirical data distribution pa(x) = Paaaa (2), and pr(x) being approximated by tractable Gaussian distribution.
This SDE is proved to have the same solution trajectories as an ordinary differential equation (ODE) 33 dubbed as Probability Flow (PF) ODE, which is formulated as
1 dæ. = [ 1) - (1) log pe de.
(17)
Therefore, the DM (2 t) is trained to estimate the score function Va. (x.). Then the estimation be used to approximate the above PF ODE by empirical PF ODE. Although various
efficient methods [27 32 46 TO 16 29 14 40, [8 have been proposed to solve the ODE, the quality of the generated images 20 is still not optimal when using relatively large dt steps. This underlines the necessity for multi-step inference in DMs and presents a substantial challenge to their wider application. For example, several customized diffusion models (26] 392 still require 50 inference steps to generate high-quality images although the overhead has been greatly reduced during training.
B Pseudo Code of TSCD
Algorithm Trajectory Segmented Consistency Distillation (TSCD)
Input: dataset D, initial model parameters O learning rate n,, ODE solver V noise schedule functions att) and (t), guidance scale range [wiinn; Wmax), the total segment count list klia- the skipping-step as 8, total training timesteps T, the num inference timesteps list NLss and encoder function F..).
2: Initialize: Set the EMA of model parameters θ <- O
3: for (i, in enumerate(kLae)) do
4: Compute the num of inference timesteps = [isss
5: for each training iteration do
6: Sample batch (z, c) from dataset D, and guidance scale w from U ['[minn- Wmax].
7:
Compute the training timesteps (to, ti......... fN-1) such that to < t1 < tN-1 = T with a uniform step size s, where tn -t tn-1 = s for n uniformly distributed over {1, (1.2.....4. N
1).
8:
Sample starting timestep i, uniformly from(t1. 12, i. -N-
9:
Calculate the segment boundary l using equation: 4b -
x [EI.
[
Sample ending timestep fend uniformly from [to, fn-1].
11:
Sample random noise z from the normal distribution (0, 1).
Sample the noised latent xt., . (a(tt))) o² (t,))).
13:
Compute the target at.-, using Eq. (1)
Compute the TSCD loss LTSCD using Eq. 12
15:
Apply gradient descent to update Q + θ 7VeLTscD.
Update the EMA of model parameters e - + stopgrad(we) + +11-)))).
17:
end for
end for
19:
Output: Refined model parameters O.
13

---

Qualitative Results
C.1 SD15 Architecture with LoRA training
SD15-Base
SDIS-LCM No CFG Steps
SDIS-TCD No CFG Steps
SDI5-PoRFlow SNFE, CFG7.5 4 Steps
Hyper-SD15
SONFE, CFG7.5
No CFG
25 Steps
Steps
ARCIEE
VEL
(+) A father and poster design, long shot
(b) girl with a hairband performing song with her guitar on warm evening local market, children's story book
man, by Anderson
Yohji Yamumoto
Figure 6. Qualitative comparisons with LoRA-based approaches on SD15 architecture.
C.2 SDXL Architecture with UNet training
SDXI.-Turbo 512px, CFG
SDXL- -igh--Igg
SDXL-Hyper No Step (UNet)
SDXL-Turbo 512px, No CPG Step
SDXL-Lighhning
SDXL-Hyper
CFG I Step
No Step
No CFG Step (UNet)
Step
girl, by Sorayama
(c) sculpture
(b) A portruit, Fauvist
(d) Baby playing with toys the
Figure Qualitative comparisons with UNet-based approaches on SDXL. architecture.
14

---

C.3 Unified LoRA
Hyper-SDDS
SDXL Unitted LoRA, 1020ps, No CPG
Unilled LoRA, 512ps, No CPG
Step
2 Steps
Steps
8 Steps
Step
2 Stepe
Stepe
8 Steps
perches quielly twisted ancient shorp yellow eyes watchful
(b) A photographer holding camen, squating by a like captriing the reflection of the mountains early morning
micng transpurent showcasing design against background
(4) A tranquil park lumished with rows benches made of marble
Figure Qualitative results unified LoRAs
C.4 Compatibility with ControlNet
ControlNet
Hyper-SDDS-Seriin
Control Image
Uniiied LoRA, S12ps, CPG
Example
Stop
2 Steps
4 Steps
8 Steps
ControlNet Control Image Examplc
Hyper-SDXL-Camny Unitied LoRA, 1024px, Step
2 Steps
4 Steps
8 Steps
Figure Our unified LoRAs are compatible with ControlNet. The examples are conditioned on either scribble or canny images.
15

---

C.5 Compatibility with Base Model
Base Models 50NFE, CFG7.5 25 Steps
Hyper-SDXL No CFG Step
Steps
4 Steps
Anime DreamShaper XL
Realistic Juggernaut XL
Artstyle ZavyChromaXL
Figure 10. Our LoRAs with different steps can be applied to different base models and consistently generate high-quality images.
16

---

D More ablation studies against TCD
To prove the effectiveness of TSCD against TCD, we conduct extra experiments on TCD+RLHF
and TCD+DMD in Tab. 5 The results show that our TSCD demonstrates superior performance
consistently over different training settings, which prove the robustness of TSCD with reduced training difficulty and less accumulation errors.
Table 5. More ablation studies against TCD.
CLIP Score
Aes Score
Image Reward
Pick
Method
Step
Score
DMD
32.35
5.81
0.35
0.217
TCD+DMD
32.50
5.67
0.31
0.215
32.58
5.69
0.85
0.222
TSCD+DMD
(+0.08)
(+0.02)
(+0.54)
(40,007)
TCD
2
32.36
5.62
0,29
0.217
32.49
5.58
0.64
0.222
TSCD
2
(+0.13)
(-0.04)
(+0.35)
(+0.005)
TCD4RLHF
32.38
5.63
0.59
0.221
32.61
5.84
1.04
0,232
TSCD+RLHF
(+0.23)
(+0.21)
(+0.45)
(40.011)
TCD
32.45
5.42
0.67
0.226
32.53
5.66
0.78
0,229
TSCD
(+0.08)
(+0.24)
(+0.11)
(+0.003)
TCD+RLHF
32.50
5.62
0.85
0.229
32.56
5.74
0.93
0.232
TSCD+RLHF
(+0.06)
(+0.12)
(+0.08)
(+0.003)
E
Discussion and Limitation
Hyper-SD demonstrates promising results in generating high-quality images with few inference steps and could benefit various downstream tasks such as semi-supervised learning [43] 41], do-
main adaptation [I [7], retrieval 42 18), etc. However, there several avenues for further
improvement:
the CFG properties of diffusion models allow for improving model performance and mitigating explicit content, such as pornography, by adjusting negative prompts. 14
Classifier Free Guidance:
40 28) including ours, eliminated the CFG characteristics, restricting the utilization negative cues and imposing usability limitations. Therefore, in future work, aim to retain the functionality of negative cues while accelerating the model, enhancing both generation effectiveness and security.
However, most diffusion acceleration methods [22 46 29
Customized Human Feedback Optimization: this work employed the generic reward models for feedback learning. Future work will focus on customized feedback learning strategies designed specifically for accelerated models to enhance their performance.
Diffusion Transformer Architecture: Recent studies have demonstrated the significant potential of DIT in image generation, we will focus on the DIT architecture to explore superior few-steps generative diffusion models in future work.
17

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In both the abstract and the introduction we break down the contributions of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitation of this paper well discussed in Appendix E
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
The formulation of our proposed TSCD algorithm is presented in Algorithm
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Implementation details can be found in Sec. 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: All checkpoints and code to reproduce our results would publicly available in project page.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https /pips.ccc/pulic/guuui guwde//ddee/ee/ Coc//ooooeeessseeeeeeo hwtpbbbmsssiooooooooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Implementation details can be found in Sec. 4.1 All checkpoints and code to reproduce our results would be publicly available in our project page.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The evaluation metrics adopted in this paper do not measure accuracy but text-to-image alignment and aesthetics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
20

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Implementation details can be found in Sec. 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This paper definitely do not violate any Code Of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used in this paper are explicitly mentioned.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All checkpoints and code reproduce results would be publicty available in our project page.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: This paper involved user study for image quality evaluation in Sec 4.2 The full text the instructions given to participants consists of only one sentence: "Given the two images below, which one do you prefer?"
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [No]
Justification: This paper does not have potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
23


Reviews:
[{"review_id": "tI2red3K0I", "reviewer": "Reviewer_XyiH", "summary": "This paper studies the distillation problem of diffusion models. Specifically, it introduces Trajectory Segmented Consistency Distillation to progressively perform consistent distillation within pre-defined time-step segments, which facilitates the preservation of the original ODE trajectory. Besides, the human feedback learning and Distribution Matching Distillation (DMD) technique are also included in the proposed model.", "questions": "Please see the weaknesses for specific questions.", "limitations": "The author discusses the limitations of the proposed method in the appendix, although it focuses more on the potential future works and follow up directions, instead of limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ This paper is well written and easy to follow.\n\n+ The studied problem is interesting and the proposed technique follows up the previous works to extend to a more general training strategy.\n\n+ The user study is informative to demonstrate the superior performance of the proposed model.", "weaknesses": "- The proposed method of TSCD sounds interesting but some details may be questionable. First, intuitively, the randomness of time segments and the choices of t_end may introduce the randomness in the model training process. How can we guarantee the model will always converge to the same results given these different design choices? Second, comparing with CTM, how can we guarantee the multi-step training with different time steps can essentially introduce the advantages? For example, the multi-stage training may also introduce accumulated errors.\n\n- It is interesting to add human feedback as a guidance. The paper also mentioned that human feedback learning may distort the output distribution. This is quite important. How can we know how human feedback learning is changing the distribution? How to detect and evaluate that? It is not very obvious why employing the LoRA merge technique with the TSCD can achieve a flexible balance between generation quality and output domain similarity. Can the author elaborate more on this? Any experiments or evidence to support this claim?\n\n- Concern about the technical novelty. The proposed model consists of multiple techniques that have been introduced and commonly used in previous works. Although the performance looks promising by adding up all these modules together from the user study and the proposed method looks reasonable, there may not be some essential out-of-box ideas or findings that are very exciting.\n\n- More detailed ablation study would be helpful. It is appreciated that the author conducts the ablation study to evaluate the effects of different modules. However, it is desirable to have a more thorough ablation study besides Table 3, since it may be still not very clear which modules among the three components play the most important factor in the proposed method. For example, without human feedback, the performance of TCD and TSCD is quite close, especially with more steps. How about with human feedback to TCD and then compare it? Does DMD are added in both TCD and TSCD experiments to have a fair comparison?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind words about our writing, intuition and experiments. We would like to answer every question you have in the following.\n\nW1: \n\nFirstly, the randomness of t_end that broadened the boundary condition of the original consistency distillation in CM[1] (Theorem 1) has been proven to be valid and guaranteed to converge in TCD[2] (Theorem 4.1) or CTM[3] (Appendix B.1). And our time segments within each stage are fixed and identical, it just gets progressively smaller between stages. \n\nSecondly, comparing with CTM, our multi-stage design choice is actually breaking down the training objective into several pieces. During the early training stage, the model focuses on learning the consistency of nearby neighboring timesteps. While in the later stages, the model can easily handle those with small intervals and extend its capability to the consistency of further timesteps or larger intervals based on that. In other words, there is overlap in the training objectives between stages, even when there are fewer segments in the later stages, we still have a probability that the target timestep with a small interval will be stepped on. \n\nFrom the perspective of accumulation error, it comes from the difficulty of fitting the model from simple tasks (smaller intervals) to difficult tasks (larger intervals), and this is much smaller than the fitting error that comes from learning a mixed task (arbitrary intervals). Similar ideas could be evidenced by Progressive Distillation[4], where the promising distilled results indicate a smaller fitting error compared to the original non-distilled diffusion model. In our experiments, we also conduct extensive ablation and user studies to validate our superior performance against TCD across different settings and inference steps. For more ablation, we also conduct extra experiments in Table 1 of global rebuttal pdf which will be detailed later in W4 section.\n\nW2:\n\nExactly, we would like to highlight that one of our main contributions is considering human feedback learning to be an effective technique to reformulate better trajectories, thus output distribution could be biased toward human preference rather than distorted badly. Since human preference is subjective and hard to be expressed through quantitative metrics, our assessment is also quite straightforward, that is, through extensive user studies as we show in the manuscript. To elaborate more on this, we would like to raise your attention to Figure 1 in the global rebuttal pdf, where we demonstrate the effectiveness of human feedback learning by different mixing ratios for TSCD and RLHF LoRAs.\n\nW3:\n\nAs for technical contributions, we would like to highlight and summarize as follows:\n1. We're the first to split the acceleration objective of diffusion model into trajectory-preserving and trajectory-reformulating. While previous approaches all just focus on the specific aspect, we break the objective down into pieces and yield different technical solutions, bringing new insights and perspectives in this field.\n2.  As for trajectory-preserving, we're based on TCD[2] and hope to further detach the training objectives as neighboring timesteps and more distant timesteps to different training stages, which could reduce training difficulty and mitigate accumulation errors.\n3. As for trajectory-reformulating, we propose to consider human feedback learning as an effective technique to bias output distribution toward human preference. It's exciting to see that simple merging of TSCD and RLHF LoRAs could bring significant benefits instead of distortion or collapse, which further demonstrates the importance of breaking down training objectives into pieces.\n\nW4:\n\nWe fully understand your concerns about ablation studies. To prove the effectiveness of TSCD against TCD, we would like to raise your attention to Table 1 in the global rebuttal pdf, where we conduct experiments on TCD+RLHF and TCD+DMD. The results show that our TSCD demonstrates superior performance consistently over different training settings, which prove the robustness of TSCD with reduced training difficulty and less accumulation errors.\n\n> [1] Song, Yang, et al. \"Consistency Models.\" International Conference on Machine Learning. PMLR, 2023.\n> [2] Zheng, Jianbin, et al. \"Trajectory consistency distillation.\" arXiv preprint arXiv:2402.19159 (2024).\n> [3] Kim, Dongjun, et al. \"Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion.\" The Twelfth International Conference on Learning Representations.\n> [4] Salimans, Tim, and Jonathan Ho. \"Progressive Distillation for Fast Sampling of Diffusion Models.\" International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y34BdsCyEG", "reviewer": "Reviewer_JxBk", "summary": "This paper presents an approach for distilling a diffusion model into a multi-step generator. Previous distillation methods typically fall into two categories: those that preserve the ODE (Ordinary Differential Equation) trajectory and those that match the teacher model at the distribution level. This research offers a unified solution by integrating ideas from both categories. Specifically, it employs a multi-segment consistency distillation objective, akin to the ODE trajectory-preserving methods like Consistency Trajectory Model, and incorporates a combination of GAN (Generative Adversarial Network) and distribution matching distillation losses from the distribution-level matching methods. Additionally, an important contribution of this work is the incorporation of human or reward model feedback in the distillation process. The resulting method demonstrates high-quality text-to-image generation.", "questions": "My concerns are outlined in the weaknesses section. While the paper demonstrates strong results and well-executed experiments, the overall method lacks significant novelty or performance improvement. Therefore, I will give it a borderline accept as my initial rating.", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. The paper is well-written and easy to follow, providing a thorough introduction to the background and connections to previous works.\n\nS2. The final approach is effective, with solid ablation studies of various components. The evaluations are comprehensive and demonstrate strong results.\n\nS3. The utilization of reward optimization in diffusion distillation is relatively new, showcasing the potential of unifying distillation with other post-training techniques to enhance final performance in terms of aesthetics, alignment, and efficiency.", "weaknesses": "W1. The final method combines well-explored modules (consistency trajectory model, GAN, and DMD) into a unified framework. While this effective integration is a contribution, it is not particularly novel and doesn't introduce significantly new knowledge.\n\nW2.Regarding the evaluation, diversity comparisons are missing from the current paper. Although FID is not the best metric, the authors should still report it for a comprehensive understanding. Additionally, assessing per-prompt diversity is important. The authors are encouraged to generate a set of images with the same prompt and measure their variations as a proxy metric. I am particularly interested in the influence of reward optimization on generation diversity.\n\nW3. For one-step generation, the improvements appear small compared to vanilla DMD (Table 3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind words about our writing, experiments and reward optimization approaches. We would like to answer every question you have in the following.\n\nW1:\n\nAs for technical novelty, we would like to highlight and summarize as follows:\n1. We're the first to split the acceleration objective of diffusion model into trajectory-preserving and trajectory-reformulating. While previous approaches all just focus on the specific aspect, we break the objective down into pieces and yield different technical solutions, bringing new insights and perspectives in this field.\n2.  As for trajectory-preserving, we're based on TCD[2] and hope to further detach the training objectives as neighboring timesteps and more distant timesteps to different training stages, which could reduce training difficulty and mitigate accumulation errors.\n3. As for trajectory-reformulating, we propose to consider human feedback learning as an effective technique to bias output distribution toward human preference. It's exciting to see that simple merging of TSCD and RLHF LoRAs could bring significant benefits instead of distortion or collapse, which further demonstrates the importance of breaking down training objectives into pieces.\n\nW2:\n\nWe agree that assessing per-prompt diversity is quite important and would like to raise your attention to Table 2 in the global rebuttal pdf, where we have picked 100 prompts and generated 4 images per prompt for each method shown. We extract the CLIP image embeddings per image and report their pair-wise average similarities for quantitative evaluation. The results show that our diversity without RLHF is quite similar compared to other acceleration approach. While biasing the output distribution toward human preference would inevitably compromise the diversity, our reward optimization doesn't sacrifice significantly and achieve an excellent trade-off compared to SDXL base model with a slight similarity increase (+0.0128). In Figure 2 of global rebuttal pdf, we also illustrate several examples for qualitative comparison.\n\nW3:\n\nWe're sorry to bring you this confusion. The DMD result in Table 3 of manuscript is not vanilla but is with RLHF and without TSCD. To further validate the effectiveness of our proposed method in one-step generation, we have provided another ablation study in Table 1 of global rebuttal pdf. The results show that vanilla DMD doesn't yield competitive results compared to consistency distilled models in terms of CLIPScore, ImageReward and PickScore, while our TSCD performs to be better.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1ggSjv0YBT", "reviewer": "Reviewer_nBsQ", "summary": "The paper introduces a novel framework named Hyper-SD, which enhances diffusion models’ efficiency by combining consistency distillation, human feedback learning, and distribution matching. Hyper-SD performs consistency distillation in segments to preserve the original ODE trajectory, maintaining high-quality generation while reducing inference steps. Human feedback learning further boosts performance in low-step inference, mitigating performance loss during the distillation process. Experimental results demonstrate that Hyper-SD achieves state-of-the-art performance from 1 to 8 inference steps, significantly surpassing existing methods, particularly in aesthetic scores and CLIP scores.", "questions": "Please check Weaknesses.", "limitations": "the limitations are discussed", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The components of this work are clear: \na) Based on TCD, the author segments the ODE and then performs distillation on these segments. \nb) To enhance the model’s overall performance, the author utilizes RLHF techniques. \nc) The author further employs the DMD scheme to optimize the results obtained in a single step.\n\n2. The results are solid, clearly demonstrating the effectiveness of the proposed approach.", "weaknesses": "The primary issue is the insufficiency of the ablation experiments, the whole Hyper-SD sounds contain 2 modules, a TCSD LoRA, a RLHF LoRA, and TCSD LoRA for 1 step needs further DMD loss.   However, the ablation study does not demonstrate the effectiveness of each part. Here's my concerns:\n\n1. In TCSD part, the distance metric includes GAN and L2. However, in TCD, it only contains L2. Is this metric similar to CTM (L2+GAN) or different? And how's the contribution of GAN part?\n\n2. RLHF module needs another LoRA, is this the only different between Tab 3 w/ and w/ RLHF?\n\n3. May I know the RECALL of the module w/ (w/o) GAN, w/ (w/o) RLHF ?\n\n4. RLHF module is totally separately trained from the TCSD module?\n\n5. The one-step model is the same training pipeline with TCSD (which means [8,4,2,1] + RLHF) but with DMD loss based on some base models like SDXL? Or it is a continue trained model based on a TCSD LoRA?  \n\n6. For one-step generator for comparison, does it include RLHF LoRA?\n\n7. A minor question. Is $\\Delta t$  ( or we can say $t_n - t_{n-1}$ ) the same for different segs?  My feeling is it should be same, but in Fig.2, the length of ODE solver is obviously different.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind words about our method and main results. We would like to answer every question you have in the following.\n\nW0:\n\nWe fully understand your primary concerns about ablation studies. To further validate the effectiveness of our proposed method in multi-step or one-step generation, we would like to raise your attention to Table 1 in the global rebuttal pdf, where we conduct experiments on TCD+RLHF, and TCD+DMD. \n1. From the results, performance across different training settings when replacing the TSCD with TCD has decreased significantly. \n2. As for RLHF component, we can see that vanilla TCD demonstrates severe aesthetic dropping when no RLHF LoRA is applied. In contrast, TCD+RLHF performs better but still lags behind our full Hyper-SD thanks to the refined trajectory-preserving capability from TSCD. \n3. For one-step generation, the conclusions are similar that only after our TSCD performs better trajectory-preserving, the score estimation will be more accurate and the power of DMD to match teacher distribution will be stronger.\n\nW1:\n\nOur GAN implementation is different from CTM that adopts StyleGAN-XL[1] discriminator for unconditional or classifier-guidance image synthesis. Since the prompting in text-to-image diffusion is quite important, we turn to a more advanced discriminator from SDXL-Lightning[2] that uses U-Net for backbone and student model for initialization. Therefore, the inputs to the discriminator would be noisy latent and timesteps together and it could better capture the subtle variation between the prediction and targets across timesteps.\n\nTo highlight the importance of introducing GAN loss, we conduct extra ablation studies in Table 3 of global rebuttal pdf. The results show that even if we also use adversarial loss for TCD, its performance is not as good as TSCD, which verifies our leadership in trajectory-preserving.\n\nW2:\n\nYes it is. We apply RLHF through LoRA merging. It's exciting to see that simple merging of TSCD and RLHF LoRAs could bring significant benefits instead of distortion or collapse, which further demonstrates the importance of breaking down training objectives into trajectory-preserving and trajectory-reformulating pieces.\n\nW3:\n\nWe have provided a more detailed comparison w/ or w/o RLHF (Table 1), w/ or w/o GAN (Table 3) in global rebuttal pdf. The results fully demonstrate the effectiveness of our proposed modules. For one, RLHF can still be effective even on TCD, and the results are even better on TSCD with better trajectory-preserving. For another, our GAN loss performs better against L2 only on TCD, and our TSCD benefit more from adversarial loss.\n\nW4:\n\nNot exactly. The training process of RLHF LoRA is with frozen TSCD LoRA applied. And we merge them into one after RLHF for convenience.\n\nW5:\n\nIt is a continuously trained model based on the two-step TCSD LoRA since we set a progressive training strategy.\n\nW6:\n\nYes it is. The result for DMD in Table 3 of manuscript is with RLHF. We're sorry to bring you this confusion. We have provided a clearer comparison for one-step generation w/o RLHF in Table 1 of global rebuttal pdf.\n\nW7:\n\nIt's the same within a stage but different between stages. Like in the 4-segments stage, the Δt is 1000//16=62. While in the 2-segments stage, the Δt is 1000//8=125.\n\n> [1] Sauer, Axel, Katja Schwarz, and Andreas Geiger. \"Stylegan-xl: Scaling stylegan to large diverse datasets.\" ACM SIGGRAPH 2022 conference proceedings. 2022.\n> [2] Lin, Shanchuan, Anran Wang, and Xiao Yang. \"Sdxl-lightning: Progressive adversarial diffusion distillation.\" arXiv preprint arXiv:2402.13929 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bgZHQtqKOK", "reviewer": "Reviewer_5gi9", "summary": "This paper introduces Hyper-SD, a novel framework designed to mitigate the computational overhead associated with the multi-step inference process of Diffusion Models (DMs). The proposed framework addresses the limitations of existing distillation techniques, which focus on either ODE Trajectory Preservation or ODE Trajectory Reformulation and often suffer from performance degradation or domain shifts. Hyper-SD synergistically combines the advantages of both approaches while maintaining near-lossless performance during step compression. Key innovations include Trajectory Segmented Consistency Distillation for preserving the original ODE trajectory, human feedback learning to enhance low-step performance, and score distillation to improve low-step generation capabilities. The framework also uniquely leverages a unified LoRA for all inference steps. Extensive experiments and user studies demonstrate the Hyper-SD's supreme performance 1 to 8 inference steps for both SDXL and SD1.5, surpassing existing methods such as SDXL-Lightning in CLIP and Aes scores.", "questions": "refer to weakness", "limitations": "refer to weakness", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper comprehensively considers segmented distillation and the incorporation of human feedback scores, potentially outperforming the original diffusion models in certain cases. It even enables one-step image generation.\n2. The experiments are thorough and well-executed and the paper is well-writen.", "weaknesses": "1. The additional introduction of the human feedback model might result in excessive memory usage. Since the evaluation steps require decoding and an additional model, would the actual computational resource consumption be clarified?\n2. The inclusion of reward terms in the loss functions of LCM-based methods has been explored in related works, such as Reward Guided Latent Consistency Distillation. A comparison with these methods, if feasible, would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind words about our method, experiments and writing. We would like to answer every question you have in the following.\n\nW1:\n\nWe would like to clarify that our RLHF training is separated from TSCD and we apply RLHF through LoRA merging. So there would be no excessive memory usage in our distillation process. During RLHF, the convergence speed of RLHF LoRA is exceptionally rapid, requiring approximately 8000 iterations for around 96 A100 GPU hours. More importantly, the reward model is not involved in the evaluation process so no additional model needs to be decoded.\n\n\nW2:\n\nThanks for your suggestion. We have read the provided paper carefully and will include it in our related work and discussion. The most essential difference is that we view feedback learning as a technique for trajectory-reformulating parallel to trajectory-preserving, and thus the training process is also separated. While in Reward Guided Latent Consistency Distillation (RG-LCM), the authors view it as some kind of guidance towards biasing the distillation process through a unified training objective. \n\nOur advantage is that we do not need to modify the original model's scheduler (i.e. DDIM) and thus have greater generalizability, while in RG-LCM the model needs to directly predict x_0 and fit the LCM scheduler. Moreover, our approach supports one-step generation with better trajectory-preserving and trajectory-reformulating pieces.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear all,

For each question from different reviewers, we have responded individually with a targeted rebuttal under.

We put all the figures and tables into the pdf file submitted here.

Hope this address your concerns and we welcome more discussion at any time.

Regardless of the final decision, thank you all for your hard work.

Best,

Authors of Submission 2694.

Meta-review: This paper has initially received borderline reviews. Though after the discussion period, most reviewers' concerns were answered and they have all increased their score. 

One reviewer who most negatively judged the current submission did not participate in the discussion despite many reminders. In addition, his remarks come from a misunderstanding that has been answered by the authors. Therefore I decided to not take this review into account.

Other reviewers acknowledged the value and originality of the proposed work:
- "results are solid, clearly demonstrating the effectiveness of the proposed approach."
- "The final approach is effective, with solid ablation studies of various components. The evaluations are comprehensive and demonstrate strong results."
- "utilization of reward optimization in diffusion distillation is relatively new"
- "paper is well written and easy to follow."

The authors should update their paper according their discussion with the reviewers for the camera-ready version.
Decision: Accept (poster)