Phased Consistency Models
Authors: Fu-Yun Wang
Zhaoyang Huang
Alexander William Bergman
Dazhong Shen
Peng Gao
Michael Lingelbach
Keqiang Sun
Weikang Bian
Guanglu Song
Yu Liu
Xiaogang Wang
Hongsheng Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Consistency Models
Diffusion Models
Distillation
Primary area: generative_models
TL;DR: Consistency model analysis and training design for advanced performance on text-to-image generation and text-to-video generation

Abstract: Consistency Models (CMs) have made significant progress in accelerating the generation of diffusion models. However, their application to high-resolution, text-conditioned image generation in the latent space remains unsatisfactory. In this paper, we identify three key flaws in the current design of Latent Consistency Models~(LCMs). We investigate the reasons behind these limitations and propose Phased Consistency Models (PCMs), which generalize the design space and address the identified limitations. Our evaluations demonstrate that PCMs outperform LCMs across 1--16 step generation settings. While PCMs are specifically designed for multi-step refinement, they achieve comparable 1-step generation results to previously state-of-the-art specifically designed 1-step methods. Furthermore, we show the methodology of PCMs is versatile and applicable to video generation, enabling us to train the state-of-the-art few-step text-to-video generator. Our code is available at https://github.com/G-U-N/Phased-Consistency-Model.

Phased Consistency Models
Fu-Yun Wang' Zhaoyang Huang²
Alexander William Bergman 36
Dazhong Shen'
Peng Gao
Michael Lingelbach
Kegiang Sun!
Weikang Bian
Guanglu Song
Yu Liut
Xiaogang Wang!
Hongsheng Lj
MMLab, CUHK
Avolution AI Hedra
Shanghai Al Lab
Sensetime Research
Sttanford University "CPI under InnoHK
(fywang@link, xgvangdee, hsli@ee) cuhk, edu. hk
Abstract
Consistency Models (CMs) have made significant progress in accelerating the generation diffusion models. However, their application to high-resolution, text-conditioned image generation in the latent space remains unsatisfactory. In this paper, we identify three key flaws in the current design of Latent Consistency Models (LCMs). We investigate the reasons behind these limitations and propose Phased Consistency Models (PCMs), which generalize the design space and ad- dress the identified limitations. Our evaluations demonstrate that PCMs outperform LCMs across 1-16 step generation settings. While PCMs are specifically designed for multi-step refinement, they achieve comparable I-step generation results previously state-of-ther of-the-artt specifically designed 1-step methods. Furthermore, we show the methodology of PCMs is versatile and applicable to video generation, enabling us to train the state-oftte the-art art few- vixeoo generator. Our code is available htttps:: : git.gtt..b.......
U- htwp:uuu//hhaaed--oosssteeeeeeee Model
Text-lo-lmage
Text-to-Video
2-Step
1-Stop
2-Step
4-Step
PCM
8-Step
16- 16-Step
Figure 1: PCMs: Towards stable and fast image and video generation.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

POW CIG
LEM
LOME C2 codt Clutt Staie colaative lngelin pronge
LPG
LOW Koaa Stop 2
PCM loza
LOM boG: SNG 4 (10c mocil
POM loos: SKID
riginss
Figure 2: Summative motivation. We observe and summarize three crucial limitations for (latent) consistency models, and generalize the design space, well tackling all these limitations.
Introduction
Diffusion models 220 57 68 have emerged as the dominant methodology for image synthesis [45] 7 and video synthesis 22 53 52 63). These models have shown the ability to generate high- quality and diverse samples conditioned on varying signals. At their core, diffusion models rely on an iterative evaluation to generate new samples. This iterative evaluation trajectory models the probability flow ODE (PF-ODE) [60 57 that transforms an initial normal distribution to target real data distribution. However, the iterative nature of diffusion models makes the generation of new samples time-intensive and resource-consuue resource-consumingg To address this challenge, consistency 55 have emerged to reduce the number of iterative steps required to generate samples. These models work by training model that enforces the self-consistency [56] property: any point along the same PF-ODE trajectory shall be mapped to the same solution point. These models have been extended to high-resoluties
models [56]
eext-to-image synthesis with latent consistency models (LCMs) [35]. Despite the improvements in efficiency and the ability to generate samples in a few steps, the sample quality of such models is still limited.
We show that the current design of LCMs is flawed, causing inevitable drawbacks in controllability, consistency, and efficiency during image sampling. Fig. 2 iillustrates our observations of LCMs. The limitations are listed as follows:
(1) Consistency. Due to the specific consistency property, CMs can only use the purely stochastic multi-step sampling algorithm, which assumes that the accumulated noise variable in each generative step is independent and causes varying degrees of stochasticity for different inference- step settings. As result, can find inconsistency among the samples generated with the same seeds in different inference steps.
(2) Controllability. Even though diffusion models can adopt classifier-free guidance (CFG)
in a wide range of values (i.e. 2-15), equipped with weights of LCMs, they can only accept values of CFG within range of 1-2. Larger values of CFG would cause the exposure problem. This brings difficulty for the hyper-parameter selection. Additionally, we find that LCMs are insensitive the negative prompt. As shown the figure, LCMs still generate black dogs even when the negative prompt is set to "black dog". Both phenomenon reduce the controllability on generation. We show the reason behind this is the CFG- CFG-augmented ODE solver adopted in the consistency distillation stage.
(3) Efficiency. LCMs tend to generate much inferior samples at the few-step settings, especially in less than inference steps, which limits the sampling efficiency. We argue that the reason lies in the traditional L22 loss or the Huber loss used the LCMs procedure, which is insufficient for the fine-grate fine-grained supervision in few-step settings.
To this end, we propose Phased Consistency Models (PCMs), which can tackle discussed limitations of LCMs and are easy to train. Specifically, instead of mapping all points along the ODE trajectory to the same solution, PCMs phase the ODE trajectory into several sub-trajectories and only enforce the self-consistency property on each sub-trajectory. Therefore, PCMs can sample

---

Driusion Madiet
Danaerancty Modells
Cursestnncy Tnqedaay Madila
Ploaeed Corseeberayy Mucels
Data cxtuuiion
Tarring
Plonora PR-00S
sooc function
Fil CIE sourton
Fi arbiiaryy ODE va oetories es
phased OZE solutins
Dbeowtration
Stodheadicty
Trajectorios recundont
Thening iilenence mamaach
Figure 3: (Left) Illustrative comparison of diffusion models [15], consistency models (56), consistency trajectory models (21) and our phased consistency model. (Right) Simplified visualization of the forward SDE and reverse-time time PF-ODE trajectories.
samples along the solution points of different sub- -aaberrajectoiess in deterministic manner without error accumulation. As an example shown in Fig. 3 we train PCMs with two sub-trajectories, with three edge points including XT, Xo. and [2/2] selected. Thereby, we achieve 2-step deterministic sampling (i.e., Xr + | + Xo for generation. Moreover, the phased nature PCMs leads to an additional advantage. For distillation, we can choose use normal ODE solver without the CFG alternatively in the consistency distillation stage. which is not viable in LCMs. As result, PCMs can optionally use larger values of CFG for inference and be more responsive to the negative prompt. Additionally, to improve the sample quality for few-step settings, we propose an adversarial loss in the latent space for more fine- fnneggained supervision.
To conclude, we dive into the design of (latent) consistency models, analyze the reasons for their unsatisfactory generation characteristics, and propose effective strategies for tackling these limitations. We validate the effectiveness of PCMs on widely recognized image generation benchmarks with stable diffusion v1-5 (0.9 B) 45 and stable diffusion XL (3B) [4] and video generation benchmarks following AnimateLCM 63 Vast experimental results show the effectiveness of PCMs.
2 Preliminaries
Diffusion models define a forward conditional probability path, with a general representation of 04Xo + 0,6 for intermediate distribution P,(x xo) conditioned on ~ Po, which is equiv- alent to the stochastic differential equation dx, = fix,dt + giden, with uz denoting the stan- dard Winer process, fc dlogg and of do?
2 2álogg There exists determinis- tic flow for reversing the transition, which is represented by the probability flow ODE (PF- ODE) dx, = [ix - 96 1/2xx log [xx) dt. In standard diffusion training, a neural network is typically learned to estimate the score of marginal distribution (x,)), which is equivalent to (x, t) x Vx log (x
dt
Exp~P(xol (x) [x log Py (x]x))] Substitue the Vx log P.(x)) with so (z, t), and we get the empirical PF-ODE. Famous solvers including DDIM [54), DPM-solver [33, Euler and Heun (20) can be generally perceived the approximation of the PF-ODE with specific orders and forms of diffusion for sampling finite discrete inference steps. However, when the number of discrete steps is too small, they face inevitable discretization errors.
Consistency models [56), instead of estimating the score of marginal distributions, learn to di- rectly predict the solution point of ODE trajectory by enforcing the self seff-consistency property: all points at the same ODE trajectory map to the same solution point. To specific, given a ODE trajectory {xx)reje.2) the consistency models fo(..() learns to achieve fa ()
by enforcing fe(x,, t) fe (xr, for all t, E [e. 2T. A boundary condition fe(x,, = x, is set to guarantee the successful convergence of consistency training. During the sampling procedure, we first sample from the initial distribution Xy N/ I). Then, the final sample can be generated/refined by alternating denoising and noise injection steps, i.e.
Th fo Tk), X = x + 77 ~ ~~
where (Tk) K are selected time points in the ODE trajectory and тк = T. Actually, because only one solution point is used to conduct the consistency model, inevitable introduce stochastic error, i.c., 77, for multiple sampling procedures.

---

Smr1
Fn
ODE trajectory
9
Discr iminator
+
Noserd
Mac
Encender
Nossing
Nearg
T1F
TF
TF
TIF
T/R
ODE trajectory
9yn-1
Stepes
Frezzn
Treeratee
Figure 4: Training paradigm of PCMs. '?' means optional usage.
Recently, Consistency trajectory models (CTMs) 2 propose a more flexible framework. specif- ically, CTMs fo(,, 1, s) learns to achieve fx 1,8) f(z,, I, 8; Φ) by enforcing fo (, = fo (xxx,''', with S min(t, t') and t, E [c, T). However, the learning objectives of CTMs are redundant, including many trajectories that will never be applied for inference. More specifi- cally, if we split the continuous time trajectory into discrete points, diffusion models learn (NN) objectives (i.e., each point learns move to its adjacent point), consistency models learn O(N) objectives (i.e., each point learns to move to solution point), and consistency trajectory models learn ((N2) objectives (i.e., each point learns to move to all the other points in the trajectory). Hence, except for the current timestep embedding. CTMs should additionally learn a target timestep embedding, which is not comprised of the design space of diffusion models.
Different from the above approaches, PCMs can be optimized efficiently and support deterministic sampling without additional stochastic error. Overall, Fig. F Billustraaes the difference in training and inference processes among diffusion models, consistency models, consistency trajectory models, and phased consistency models.
3 Method
In this section, we introduce the technical details of PCMs, which overcome the limitations of LCMs in terms of consistency. controllability, and efficiency. Consistency: Specifically, we first introduce the main framework of PCMs, consisting of definition, parameterization, the distillation objective, and the sampling procedure in Sec. Sec.B.I In particular, by enforcing the self-consistency property in multiple ODE sub-trajectories respectively, PCMs can support deterministic sampling to preserve image consistency with varying inference steps. Controllability: Secondly, Sec. S.2. improve the controllability of text guidance, we revisit the potential drawback of the guided distillation adopted in LCMs, and propose to optionally remove the CFG for consistency distillation. Efficiency: Thirdly, in Sec. 3.3 to further improve inference efficiency, we introduce adversarial consistency loss to enforce the modeling of data distribution, which facilitates 1-step generation.
3.1 Main Framework
Definition. For solution trajectory of diffusion model (xIheee,T) following the PF-ODE, split the trajectory into multiple sub-trajectories with hyper-defined edge timesteps Sp, , SM where So and SAS = T. The M sub-trajectories
be represented {x4 fre (eemms with m = 0, 1,, 1. We treat each sub-trajectory as an independent CM and define the consistency function fr (x,,t) t) + X.m. t [Sm 8m+1] |]- We learn fo" estimate by enforcing the self-consistency property on each sub-trajectory that its outputs are consistent for arbitrary pairs on the same sub-trajectory. Namely, (x,,) that the consistency function is only defined on the sub-trajectory. However, for sampling, it is necessary define a transition from timestep T' (i.c., sM) to e (i.c., so). Thereby, we defined fm.mn fmm (... fm (mm-1(f") (x,; f), sm); sm-1) .... .'"))m))))))) that transforms any point x, on m-th sub-trajectory the solution point of m trajectory.
fr"(xx((') for all 1, E [&m 8m Note
Parameterization. Following the definition, the corresponding consistency function of each sub- trajectory should satisfy boundary condition f (Xee.. 8m) X which is crucial for guaranteeing

---

the successful training of consistency models. To satisfy the boundary condition, we typically need cm
to explicitly parameterize fø" (x. t) as (x,,,, Skip (t)x, + Cou (o)FF) (x,, S,n), where em gradually increases to 1 and cou (t) gradually decays to 0 from timestep 8m+11 to 8m.
Skip (t)
Another important thing is how to parameterize [x f, s), basically it should be able to indicate the target prediction at timestep given the input timestep Since we build upon the epsilon prediction models, we hope to maintain the iaailon-reeiction learning target.
For the above-discussed PF-ODE, there exists an exact solution
33 from timestep to
0g X + 0s /" O V log Pr (a)
(2)
where A = In and Êx is a inverse function with A, The equation shows that the solution of the ODE from t to di
is the scaling of x; and the weighted sum of scores. Given a epsilon prediction diffu- sion network EΦ (x; (), we can estimate the solution as x, 22 x, - 0ta fx Ex (A): Lx (A))dA. dee
However, note that the solution requires knowing the epsilon prediction at each timestep between t and s, but consistency modes need to predict the solution with only x available with single network evaluation. Thereby, we parameterize the Fe (x, i, 8) as following,
Os X (xx((x),)) f/ e cd
X
(3)
at
One can show that the parameterization has the same format with DDIM X, =a (x- (x=_o_c((xx (x
(x,,(x, t) (see Theorem 3). But here we clarify that the parameterization has intrinsic difference from DDIM. DDIM is the first-order approximation of solution ODE, which works because we assume the linearity of the score in small intervals. This causes the DDIM to degrade dramatically in few-step settings since the linearity is no longer satisfied. Instead, our parameterization is not approximation but exact solution learning. The learning target of é t) is no more the scaled score -o, Vx log (x(
(which epsilon-prediction diffusion models learn to estimate) but 12 e e="ealx (A)))) AAtually,
Sr. Add
we can define the parameterization in other formats, but find this format is simple and has a small gap between the original diffusion models. The parameterization of Fg also allows for better property that we can drop the introduced em and C in consistency models to ease the complexity of the framework. Note that following Eq, 3 we can get (e( 8m Sm) Oom X.,.. -0 x Therefore, the boundary condition is already satisfied. Hence, we can simply 0.m
define fø (x, f) Fo (x, t. Sm)). This parameterization also aligns with several previous diffusion distillation techniques [46] 74) utilizing DDIM format, building deep connection with previous distillation methods. The difference is that we provide more fundamental explanation of the meaning and learning objective of parameterizations.
Phased consistency distillation objective. Denote the pre-trained diffusion models as s f) (x(x.)) which induces an empirical PF-ODE. We firstly discretize the whole trajectory into N sub-intervals with N +1 1 discrete timesteps from [e, which we denote as In
< 4 (2 < < IN = T. Typically N should be sufficiently large to make sure the ODE solver approximates the ODE trajectory correctly. Then we sample M + 1 timesteps as edge timesteps 8o = lo < 81 < 82 < .... < SM = IN E (1,) N to split the ODE trajectory into M sub-trajectories. 11
Each sub-trajectory [st, s1+1] consists of the set of sub-intervals
Here we define & ((x tn+k. tm: Φ) as the k-step ODE solver that approximate x", from Xt on the same sub-trajectory following Equation 2 namely,
2 (X tntbs tni Φ).
Following CMs [56) we set ks to minimize the ODE solver cost. The training loss is defined as
(5) where P(m) = ((0, 1,..., M- 1}), P(n|m) = uniform (n + 1|b241 S smt1, 2
LPCM +) Ep(m).F(r/m))
(n.
[A(tn)d (X (n11), S0 (8f, bn))
sm )), P( |n, m) = Penti (x), and 0- u + (1 0.
5

---

Slabie Ditisioon
CTM
InstaFiow
SD-Turo
SEXu-igghiing
SDNL-Turoo
PCM
glrs queiity
1-5kp
2.Stpp
Loti resuind der
vamir gou
Les Enciieal Sret gwally clound brennen cwe amoud with
Figure 5: Qualitative Comparison. Our method achieves top-tier performance.
We show that when LPCM (0c ; Φ) 0 and local errors of ODE solvers uniformly bounded by ((At))t)), the solution estimation error within the arbitrary sub-trajectory fø (Xt,, tn)) bounded Additionally, the solution estimation error across any sets of sub-
by ((()))) in Theorem !
"X (Xt b)) is also bounded by trajectories (i.e., the error between fø fm.m (x),, and f fm.m" ((A))) in Theorem
Sampling. For given initial sample at timestep which belongs the sub-trajectory [8ms 8m 1
we can support deterministic sampling following the definition of fmm Previous work [21] 67
reveals that introducing certain degree of stochasticity might lead to better generation quality. We show that our sampling method can also introduce randomness through a simple modification, which we discuss at Sec. [V.
3.2 Guided Distillation
For convenience, we take the apsilon-predictiinn format with text conditions e for the following discussion. The consistency model is denoted as (x,,,, t, c), and the diffusion model is denoted as c t, c). commonly applied strategy for text-conditioned diffusion models classifier free guidance (CFG) [16, 51 [. At training, is randomly substituted with null text embedding 2. At each inference step, the model computes eø (Xe; actual prediction is the linear combination of them. Namely,
(xe, L, c) and E (x,, L, Creg) simultaneously, and the
(x(x,,t, C, Cneg; w) (x,, Cneg) + ur (r(E,(X,,,,, - (x;; L, Cueg))
where w controlling the strength and Caeg can be set to Ø or text embeddings of unwanted char- acteristies. A noticeable phenomenon is that diffusion models with this strategy can not generate content with good quality without using CFG. That is, the empirical ODE trajectory induced with pure eø (xe, L, c) deviates away from the ODE trajectory to real data distribution. Thereby, it is necessary to apply CFG-augmented prediction é (x,,,, 1, C, Cnegi w) for ODE solvers. Recall that we have shown that the consistency learning target of the consistency model (x is the weighted sum of epsilon prediction on the trajectory. Thereby. we have Eg (Xe, f, c) x (x I', 2; w), for all < On this basis, if we additionally apply the CFG for the consistency models, we can prove that
EØ (x(e, t, C, Caegi w') x ww (eu'(Es(xx) (t,, t),c)= E merge)) + Eg (x( Cueg)
merge = (1 (Xe, Cueg) + (x 2) and a (w-1) (See Theorem 4 This where Es equation indicates that applying CFG w' to the consistency models trained with CFG-augmented ODE solver confined with w, is equivalent to scaling the prediction of original diffusion models by w/ w'w, which explains the the exposure problem. We can also observe that the epsilon prediction with negative prompts is diluted by the prediction with null text embedding, which reveals that the impact of negative prompts is reduced.
ww"

---

Table 1: Comparison of FID-SD and FID-CLIP with Stable Diffusion v1-5 based methods under different steps.
FID-SD
FID-CLIP
METHODS
COCO-30K
CC12M-30K
3OK
4
5
16
2
8
16
2
4
8
16
InstaFlow
102.15
122.20 20
139.29
11.90
62.48
58
105.34
113.56
9.56
38.24
43.60
SDDTubbo
10.62
12.22
16.66
24.30
30.32
10.35
12.03
15.15
19.91
23.34
12.08
15.07
15.12
14.90
15.09
LCM 55
53.43
11.03
6.66
6.62
7.56
12.67
10.51
6.40
5,99
9.02
21.04
10.18
10.64
12.15
13.85
CTM 21
63.55
9.93
9.30
15.62
21.75
28.47
8.98
8.22
13.27
17.43
30.30
10.32
10.51
11.27
12.75
7.91
3
3.88
11.66
2.12
9.07
9.49
Ouns
7.46
5.42
5.78
4.99
5.01
5.13
8.85
8.33
8.09
We ask the question: Is it possible to conduct consistency distillation with diffussoo models trained for CFG usage without applying CFG-augmented ODE solver? Our finding that not applicable for original CMs but works well for PCMs especially when the number of sub-trajectory M is large. We empirically find that M 4 sufficient for successful training. As we discussed, the text-conditioned diffusion models trained for CFG usage fail at achieving good generation quality when removing CFG for inference. That is, target data distribution induced by PF-ODE with Eg (X,, t, c) has large distribution distance to real data distribution. Therefore, fitting the spoiled ODE trajectory is only to make the generation quality bad. In contrast, when phasing the whole ODE trajectory into several sub-trajectories. the negative influence is greatly alleviated. On one hand, the starting point x of sub- trajeectooiie is replaced by adding noise to the real data. On the other hand, the distribution distance between the distribution of solution points Xsm. of sub-trajectories and real data distribution pdata at the same timestep is much smaller proportional to the noise level introduced. To put sm
straightforwardly, even though the distribution gap between the real data and the samples generated from Es (x, f c) is large, adding noise to them reduce the gap. Thereby, we can optionally train a consistency model whether supporting larger values of CFG or not.
3.3 Adversarial Consistency Loss
We introduce an adversarial loss to enforce the distribution consistency, which greatly improves the generation quality in few-step settings. For convenience, introduce an additional symbol Ti which represents flow from P. to P,, Let T The be the transition mapping following the ODE trajectory of pre-trained diffusion and consistency models. Additionally, let T be the distribution transition following the forward process SDE (adding noise). The loss function is defined as the following
padv PCCM
( :0; -D(TT - To 1 #P t.++ T to 1 +t f.tt1)
(Tem--sTT
(8)
where # is the pushforward operator, and D is the distribution distance metric. To penalize the distribution distance, we apply the GAN-style training paradigm. To be specific, as shown in Fig. f
for the sampled Xt,+, and the xf, solved through the pre-trained diffusion model Φ,, first compute
their predicted solution point x = f (xt bntt) and &... = fo- (x,,, Then we randomly add noise to x.,. and & to obtain x, and &, with randomly sampled E [Sms sm+1]. We optimize the adversarial loss between x, and Xz. Specifically, the Lpéx can be re-written as
CPCM (0, 0 Φ m) ReLU(1 + D(K,, c)) ReLU(1 D(x,.
where ReLU(x) = x if O else ReLU(x) = 0. D is the discriminator, is the image condi- tions (e.g., text prompts), and the loss is updated in a min- max manner [12]. Therefore the eventual optimization objective is LPCM + ACrdy with À as a prree--ar controlling the trade-off of distribution consistency and instance consistency. We adopt À = 0.1 for all training settings. How- ever, we hope to clarify that the adversarial loss has an intrinsic difference from the GAN. The GAN training aims to align the training data distribution and the generation distribution of the model. That is, D(T D #P tx a |IIP a..)) In Theorem 5 we show that consistency property is enforced, our introduced adversarial loss will also coverage to zero. Yet in Theorem 6 we show that combining standard GAN with consistency distillation is a flawed design when considering the pre-trained data distribution and distillation data distribution mismatch. Its loss will be non-zero when the self-consistency property is achieved, thus corrupting the consistency distillation learning- Our experimental results also verify our statement (Fig.

---

Table 2: One-step generation comparison on Stable Diffusion v1-5.
COCO-30K
CC12M-306
METHODS
CONSISTENCY T
FID
FID-CLIP
IID-SD
CLIP SCORE
FID
FID-CLIP
FID.SD
CLIP SCORE
InstaHow
13.59
11.51
29.37
6.16
11.90
0.61
SD Turbo
49
16.56
12.08
10.62
31.21
17.17
6.18
12.48
26.30
0.71
CTM
21
67.55
30.39
63.55
23.98
56.39
28.47
56.34
18.81
0.65
1CM
5381
53.33
44.35
18.58
42.67
0.62
TCD
74
71.69
31,69
68.04
23.60
57.97
30.03
57.21
18.57
Ours
17.91
11.66
8.27
29.26
14.79
5,38
7.91
26.33
0.81
4 Experiments
4.1
Experimental Setup
For image generation, we train all models on the CC3M [ dataset. For video generation, we train the model on WebVid-2M (2). Evaluation dataset: For image generation,
Dataset. Training dataset:
we evaluate the performance on the COCO-2014 (28] following the 30K split of karpathy. We also
evaluate the performance on the CC12M with our randomly chosen 30K split. For video generation, we evaluate with the captions UCF-101 (58).
Backbones. We verify the text-to-image generation based on Stable Diffusion v1-5 45 and Stable Diffusion XL 41. We verify the text-to-video generation following the design of AnimateLCCM 63 with decoupled consistency distillation.
Evaluation metrics. Image:
We report the FID [1] and CLIP score 43 of the generated images and the validation 30K-sample splits. Following g 47). we also compute the FID with CLIP features (FID-CLIP). Note that, all baselines and our method focus on distilling the knowledge from the pre-trained diffusion models for acceleration. Therefore, we also compute the FID of all baselines and the generated images of original pre-trained diffusion models including Stable Diffusion v1-5 and Stable Diffusion XL (FID-SD). Video: For vidco generation, we evaluate the performance from three perspectives: the CLIP Score measure the text-video alignment, the CLIP Consistency to measure the framr-frame consistency of the generated videos, the Flow Magnitude to measure the motion magnitude of the generated videos with Raft-Large [59].
4.2 Comparison
Comparison methods. We compare PCM with Stable Diffusion v1-5 to methods including Stable
Diffusion v1-5 [45,, InstaFlow [31]. LCM (35) CTM (21), TCD (74] and SD-Turbo 49
We
compare PCM with Stable Diffusion XL to methods including Stable Diffusion XL 41.. CTM
21
SDXL-Lightning (27), SDXL-TTubo 49]. and LCM (35. We apply the 'Ours' and 'Ours*' to denote our methods trained with CFG-augmented ODE solver or not. We only report the performance of "Ours"' with more than 4 steps which aligns with our claim that is only possible when phasing the ODE trajectory into multiple sub-trajectories. For vidco generation, we compare with DDIM [54), DPM 333, and AnimateLCM [63].
Qualitative comparison. We evaluate our model and comparison methods with a diverse set of prompts in different inference steps. The results are listed in Fig. Our method shows clearly the top performance in both image visual quality and text-image alignment across 1-16 steps.
Quantitative comparison. One-step generation: We show the one-step generation results com- parison of methods based Stable Diffusion v1-5 and Stable Diffusion XL. in Table 2nd Table respectively. Notably, PCM consistently surpasses the consistency model-based methods including LCM and CTM by large margin. Additionally, it achieves comparable even superior to the statc-of-the-ar GAN-based (SD-Turbo, SDXL-Turbo SDXL-Lightning) or Rectified- Flooerba--obbbbeedd (In- staFlow) one-step generation methods. Note that InstaFlow applies the LIPIPS [72] loss for training and SDXL-Tuubo can only generate 512 x 512 resolution images, therefore it is easy for them to obtain higher scores. Multi-step generation: We report the FID changes of different methods on COCO-30K and CC12M-30K in Table [ and Table B 'Ours' and "Ours*' achieve the best or second-best performance in most cases. It is notably the gap of performance between our methods and other baselines becoming large as the timestep increases, which indicates the phased nature of our methods supports more powerful multi-step sampling ability. Video generation: We show the quantitative comparison of video generation in Table our model achieves consistent superior
8

---

Table 3: Comparison of FID-SD on Table 4: Quantitative comparison for video generation.
CC12M-30K with Stable Diffusion XL.
Mitboos
Score
Maguinde
Consisiasy 1
METROUS
1.5ap
1.8xp
4.kk
5-3ky
15.Sap
1-5xep
4.5cp
2.5kpp
1.5ap
4.5op
SUX1- Tano 22
SDXL-Ligent Ligenirg
3.50
0.53
1.39
5.36
3.29
13388
5.83
6:10
DDIM 14
DPM [3
11.21
4.44
17.05
7.00
23.05
26.57
138
1.47
01477
0,77
50X0
CTM
10
73.45
36.06
27.67
39.00
59
AsinsaL.MM
[3
25.41
29.50
30662
1.10
1.81
2.40
0.967
0.957
0.65
29,88
36.2
4.38
0.556
0.668
6.26
5.21
500
Table 5: One-step and IWO-step generation comparison on Stable Diffusion XL,
METHODS
COC0-30K (oup-step)
0C12M-30K (two-step)
CONSISTENCY
FTD
FID CLIP 1 FID SD 1
CLIP SCORE
[FID
FID- 1 [TD SD
CLIP SCORE
SDXL Turbo (512 x 512)
19
19.84
13.56
9.60
32.31
15.36
5.26
6.53
27.91
0.54
SDXL-Lihhning
19.73
13.33
9.11
30.S1
17.99
7.39
7.29
26.31
0.76
SDXL-LCM K51
74,65
31,63
74,46
27.29
25.88
10.36
19.64
25.84
0.66
SDXL-CTM
37.43
13,66
31.55
performance. The 1-step and 2-step generation results of DDIM and DPM are very noisy, therefore is meaningless to evaluate their Flow Magnitude and CLIP Consistency.
Human evaluation metrics. To more comprehensively reflect the performance of phased consistency models, we conduct a thorough evaluation using human aesthetic preference metrics, encompass- ing 1-16 steps. This assessment employs well-regarded metrics, including HPSv2 (HPS) [66], PickScore (PICKSCORE) [23], and Laion Aesthetic Score (AES) [50], to benchmark our method against all comparative baselines. As shown in Table and Table 7 across all evaluated settings, our method consistently achieves either superior or comparable resuTrs, with a marked performance advantage over the consistency model baseline LCM. demonstrating its robustness and appeal across diverse human-centrie evaluation criteria. We conduct a human preference ablation study on the pro- posed adversarial consistency loss, with the results presented in Table The inclusion of adversarial consistency loss consistently enhances human evaluation metrics across different inference steps.
Table 6: Aesthetic evaluation on SD v1-5.
Table 7: Aesthetic evaluation on SDXL.
Steps
Methods
HPS
AES
PICKSCORE
Steps
Methods
UIPS
AES
PICKSCORE
InstaHlow
0.267
5,010
0.207
1
SD TDrbur
CTM
0,276 (1)
0,240
5.445 (1)
5.155
0,223 (1)
0.195
IDXL-Lighinng
SDXL-CTM
SDXL-Turbo
0.279(()
0.278
0.239
5.65 (1)
4,86
5.40
0.228 (1)
0.201
0.223
0.251
5.178
0.201
SDXL-LCM
0.205
0.206
Ours
0,276 (1)
5,389 (2)
0.213
Ouns
0.280 (1)
5.62 (2)
0.225 (2)
InstaHlow
0.249
5,050
0.196
SDXL-Lighming
0.280
5.72
0.227 (1)
SD-Turbo
0,278
5,570 (1)
(1226
SDXL-TTubbo
0.281 (2)
5.46
0.226 (2)
2
CTM
0.267
5,117
0.208
SDXL-CTM
0.267
5.58
0.216
LCM
0.266
5.135
0.210
SDXL-LCM
0.265
0.217
Ours
0.275 (2)
5,370 (2)
0,217
Ours
0.282 (1)
5.688 (2)
0.225
InstaFloo
0.243
4.765
0.192
SDXL-Lighming
0.281
5.76
0.228
SD-Turbo
0.278 (2)
5.537 (1)
(1)
SDXL-Turo
0.284 (1)
5.49
0.224
CTM
0.274
5.189
0.213
SDXL-CTM
0.278
5.84 (1)
0.231
LCM
0.273
5.264
0.215
SDXL-LCM
0.274
5.48
0.223
Ours
0.279
(2)
(2217
Ours
0.284 (1)
5.645
0.228 (2)
InstaFlow
0.267
4.548
0.189
SDXL iightning
0.282
5.75
0.229 (1)
SD-Turbe
(2)
(2)
0.221 (1)
SDXL-Tubo
0.283 (2)
5.59
0.225
8
CTM
0.271
5.026
0.210
SDXI-CTM
0.276
5.88
0.218
0.274
5.366
SDXL-LCM
0,277
5.57
0.223
Ours
0,278 (1)
5.398 (1)
0,218
Ours
0.285 (I)
5.676
0.229 (2)
InstaFlow
0.237
4.437
0.187
SDXL HDXI-Lighhning
0.280
5.72 (2)
0.225 (2)
16
SD-Turbo
CTM
0,277 (1)
0.270
4,870
5.275
0,219 (1)
0.209
SDXL-
SDXL-CTM
SDXL-LCM
0.276
0.277
0.274
5.85
5.56
5.64
0.221
0.219
0.215
LCM
0.274
5.352 (2)
0.216
Ours
0.284 (1)
5.646
0.228
Ours
0.277 (1)
5.442 (1)
0,217 (2)
Table 8: Aesthetic ablation study on the adversarial consistency loss.
Methoda
Step
Stap 2
Stop
Sies
Step
10S
AES
MOCSCORE
I0S
AES
PXCKSOORE
IPF
ABS
PSCKSO
185
AlSS
PIC3300KE
109
ABS
PICSSCORE
PCM sul
0,250
5,620
0225
0.383
3,689
0.225
0.284
56445
0.225
0.285
5,676
0.229
0.254
5,646
0.218
PCM
0,251
0206
0.75
3,502
0.220
3.575
5/37
0.227
5,620
0.227
4.3 Ablation Study
Sensitivity to negative prompt. To show the comparison of sensitivity to negative prompt between model tuned without CFG-augmented ODE solver and LCM. We provide an example of prompt

---

Figure 6: Sensitivity to nega- tive prompt.
Figure 7: Effectiveness of ad- versarial consistency loss.
Figure 8: Randomness for sam- pling.
Replact Shotrninator with DRC
Paplace car acveraria buswit romra CAlilaua
Out
Dur
Figure 9: Ablation study on the adversarial consistency design. (Left) Replacing latent discrimi- nator with DINO causes detail loss. (Right) Replacing adversarial loss with normal GAN loss causes training conflicted objectives and instability.
and negative prompt to GPT-4o and ask it generate 100 pairs of prompts and their corresponding negative prompts. For each prompt, we generate 10 images. We first generate images without using the negative prompt show the positive prompt alignment comparison. Then generate images with positive and negative prompts. We compute the CLIP score generated images and the prompts only achieve better prompt alignment but much
and negative prompts. Fig. 6 shows that more sensitive to negative prompts.
Consistent generation ability. Consistent generation ability under different inference steps is valuable in practice for multistep refinement. We compute the average CLIP similarity between the 1-step generation and the 16-step generation for each method. As shown in the rightmost column of Table and T5 our method achieves significantly better consistent generation ability.
Adversarial consistency design and its effectiveness. We show the ablation study the adversarial consistency loss design and effectiveness. From the architecture level discriminator, we compare the latent discriminator shared from the teacher diffusion model and the pixel discriminator from pre-trained DINO 4]. Note that DINO is trained with 224 resolutions, therefore we should resize the generation results and feed them into DINO. We find this could make the generation results fail at details as shown the left of Fig. 9
From the adversarial loss, we compare our adversarial loss to the normal GAN loss. We find normal GAN loss causes the training to be unstable and corrupts the generation results, which aligns with our previous analysis. For its effectiveness, we compare the FID-CLIP and FID scores with the adversarial loss without the adversarial loss under different inference steps. Fig. shows that greatly improves the FID scores in the low-step regime and gradually coverage to similar performance of our model without using the adversarial loss the step increases.
Randomness for sampling. Fig. iillustrates the influence of the randomness introduced in sampling as Eq. [4 The figure shows that introducing a certain of randomness in sampling may help to alleviate unrealistic objects or shapes.
Limitations and Conclusions
Despite being able generate high-quality images and videos in few steps, we find that when the number of steps is very low, especially with only one step, the generation quality is unstable. The model may produce structural errors or blurry images. Fortunately, we discover that this phenomenon can be mitigated through multi-step refinement, In conclusion, in this paper, we observe the defects latent consistency models. We summarize these defects on three levels, analyze their causes, and generalize the design framework to address these defects.
10

---

6 Acknowledgements
This project funded in part by National Key R&D Program of China Project 20222D016 100, by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s ' InnoHK, by General Research Fund Hong Kong RGC Project 14204021. Hongsheng Li is Pl of CPII under the InnoHK.
References
[1] Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with cettuubbddatteention guidance. arXiv preprint wxxx::2403.17777 2024.
[2] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In International Conference on Computer Vision, pages 1728-1738, 2021.
[3] David Berthelot, Arnaud Autef, Jierui Lin, Dian Ang Yap. Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbott, and Eric Gu. Tract: Denoising diffusion models with transitive closure time-distillation. arXiv preprint arXiv:2303.042448 2023.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supe vision transformers. In International Conference on Computer Vision, 2021.
[5] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Conference Computer Vision and Pattern Recognition, pages 3558-3566 2021.
[6] Ricky TQ Chen and Yaron Lipman. Riemannian flow matching on general geometries, arXiv preprint 232.::2302.03660. 2023.
[7] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[8] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206, 2024.
[9] Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et Lumina-12x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arXiv:2405.0582 05945, 2024.
[10] Zhengyang Geng, Ashwini Pokle, and J Zico Kolter. One-step diffusion distillation via deep equilibrium models. Advances in Neural Information Processing Systems, 36, 2024.
[11] Zhengyang Geng, Ashwini Pokle, William Luo, Justin Lin, and Zico Kolter. Consistency models made easy. arXiv preprint arXiv:2406. 14548 2024.
[12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1):119)-4444)) 144, 2020.
[13] Jonathan Heek, Emiel Hoogeboom, and Tim Salimans. Multistep consistency models. arXiv preprint arXiv:2403.06807, 2024.
[14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. ddpm. Advances in Neural Information Processing Systems, 33:6840-68511 2020.

---

[16] Jonathan Ho and Tim Salimans. cfg. arXiv preprint arXiv:2207.12596 2022.
[17] Minghui Hu, Jianbin Zheng, Chuanxia Zheng, Chaoyue Wang, Dacheng Tao, and Tat-Jen Cham. One more step: A versatile plug-and-play module for rectifying diffusion schedule flaws and enhancing oowfrrquencyy controls. Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), pages 7331-7340, June 2024.
[18] Dongzii Jiang, Guanglu Song, Xiaoshi Wu, Renrui Zhang, Dazhong Shen, Zhuofan Zong, Yu Liu, and Hongsheng Li. Comat: Aligning text-to-image diffusion model with image-to-text concept matching, arXiv preprint arXiv:2404.03655. 2024.
[19] Minguk Kang, Richard Zhang, Connelly Barnes, Sylvain Paris, Suha Kwak, Jaesik Park, Eli Shechtman, Jun- Yan Zhu, and Tacsung Park. Distilling diffusion models into conditional gans. arXiv preprint mwXivv2405.05990 2024.
[20] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. edm. Advances in Neural Information Processing Systems, 55526565-2667777 2022.
[21] Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279 2023.
[22] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. On density estimation with diffusion models. In Advances in Neural Information Processing Systems, 2021.
[23] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36666 2023.
[24] Fei Kong, Jinhao Duan, Lichao Sun, Hao Cheng. Renjing Xu, Hengtao Shen, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. Act-diffusion: Efficient adversarial consistency training for one-step diffusion models, 2024.
[25] Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. Cllms: Consistency large language models, 2024.
[26] Sangyun Lee, Yilun Xu, Tomas Geffner, Giulia Fanti, Karsten Kreis, Arash Vahdat, and Weili Nie. Truncated consistency models. arXiv preprint arXiv:2410.14895. 2024.
[27] Shanchuan Lin, Anran Wang, and Xiao Yang. Sdxl-lightning: Progressive adversarial diffusion distillation. arXiv preprint arXiv:2402. 13929, 2024.
[28] Tsung-Yr Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In European Conference on Computer Vision, volume 8693, pages 740-755. Springer, 2014.
[29] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02774, 2022.
[30] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03000 2022.
[31] Xingchao Liu. Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality diffusion based text-to-image generation. In International Conference on Learning Representations, 2023.
[32] Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous time consistency models, arXiv preprint arXiv:2410.1 1081, 2024.
[33] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
12

---

[34] Guanxing Lu, Zifeng Gao. Tianxing Chen, Wenxun Dai, Ziwei Wang, and Yansong Tang. Manicm: Real-time 3d diffusion policy via consistency model for robotic manipulation. arXiv preprint arXiv:2406.01586 2024.
[35] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378. 2023.
[36] Bingqi Ma, Zhuofan Zong, Guanglu Song, Hongsheng Li, and Yu Liu. Exploring the role of large language models in prompt encoding for diffusion models, 2024.
[37] Xiaofeng Mao, Zhengkai Jiang, Fu-Yun Wang, Wenbing Zhu, Jiangning Zhangg Hao Chen, Mingmin Chi, and Yabiao Wang. Osv: One step is enough for high-quality image to video generation. arXiv preprint arXiv:2409.11 2024.
[38] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Conference Computer Vision and Pattern Recognition, pages 14297-14306, 2023.
[39] Zachary Novack, Ge Zhu, Jonah Casebeer, Julian McAuley, Taylor Berg-Kirkaatrick, and Nicholas Bryan. Presto! distilling steps and layers for accelerating music generation. arXiv preprint arXiv:2410.05167, 2024.
[40] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4195 4205, October 2023.
[41] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint urXiv:230..01952, 2023.
[42] Aaditya Prasad, Kevin Lin, Jimmy Wu, Linqi Zhou, and Jeannette Bohg. Consistency policy: Accelerated visuomotor policies via consistency distillation, 2024.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Learning Representations, pages 8748-8763, PMLR, 2021.
[44] Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao. Hyper-sd: Trajectory segmented consistency model for efficient image synthesis. arXiv preprint arXiv:2404. 13684 2024.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[46] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00511. 2022.
[47] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. Advances Neural Information Processing Systems, 34: 17492, 2021.
[48] Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. International Conference on Learning Representationss pages 30105-30118 18. PMLR, 2023.
[49] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint 2rXi::231 2023.
[50] Christoph Schuhmann. Laion-asstheties. http://wh...a.....a.ao/a/aaaa/aa
2022. Accessed: 2023-11-10.
13

---

[51] Dazhong Shen, Guanglu Song, Zeyue Xue. Yun Wang. and Yu Liu. Rethinking the spatial inconsistency classifier-free diffusion guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3370-9379, 2024.
[52] Xiaoyu Shi, Zhaoyang Huang. Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung. Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable
generation with explicit motion modeling. In ACM SIGGRAPH 2024 Conference Papers, pages 1-11, 2024.
[53] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
[54] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
[55] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310.141880 2023.
[56] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, arXiv preprint arXiv:2303.01 2023.
[57] Yang Song, Jascha Sohl- Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. bcore-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.133555 2020.
[58] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos the wild. CoRR, abs/1212.0402 2012.
[59] Zachary Teed and Jia Deng. Raft: Recurrent all pairs field transforms for optical flow, 2020.
[60] Cunzheng Wang, Ziyuan Guo, Yuxuan Duan, Huaxia Li, Nemo Chen, Xu Tang, and Yao Hu. Target-driven distillation: Consistency distillation with target timestep selection and decoupled guidance. arXiv preprint arXiv:2409.01344 2024.
[61] Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen-I-video: Multi-text to long video generation via temporal co-denoising. arXiv preprint arXiv:2305. 18264, 2023.
[62] Fu- Yun Wang, Zhengyang Geng, and Hongsheng Li. Stable consistency tuning: Understanding and improving consistency models. arXiv preprint arXiv:2410. 2024.
[63] Fu- Yun Wang, Zhaoyang Huang, Xiaoyu Shi, Weikang Bian, Guanglu Song. Yu Liu, and Hongsheng Li. Animatelem: Accelerating the animation of personalized diffusion models and adapters with decoupled consistency learning. arXiv preprint arXiv:2402.00766, 2024.
[64] Fu- Yun Wang, Xiaoshi Wu, Zhaoyang Huang, Xiaoyu Shi, Dazhong Shen, Guanglu Song, Yu Liu, and Hongsheng Li. ce-your-outpainter Mastering video outpainting through input- specific adaptation. In European Conference on Computer Vision, pages 153-168. Springer, 2025.
[65] Fu-Yun Wang, Ling Yang, Zhaoyang Huang, Mengdi Wang, and Hongsheng Li. Rectified diffusion: Straightness is not your need in rectified flow. arXiv preprint arXiv:2410.07303, 2024.
[66] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.099: 2023.
[67] Yilun Xu, Mingyang Deng, Xiang Cheng, Yonglong Tian, Ziming Liu, and Tommi Jaakkola. Restart sampling for improving generative processes, 2023.
[68] Yilun Xu, Shangyuan Tong, and Tommi Jaakkola. Stable target field for reduced variance score estimation in diffusion models. arXiv preprint arXiv:2302.006770 2023.
14

---

[69] Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to- generation via large mixture of diffusion paths. Advances in Neural Information Processing Systems, 36, 2024.
[70] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6613-6623, 2024.
[71] Yuanhao Zhai, Kevin Lin, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Chung-Ching Lin, David Doermann, Junsong Yuan, and Lijuan Wang. Motion consistency model: Accelerating video diffusion with disentangled motion-appearance distillation. arXiv preprint S4Ki::2400.06990 2024.
[72] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. In Conference on Computer Vision and Puttern Recognition, June 2018.
[73] Long Zhao, Sanghyun Woo, Ziyu Wan, Yandong Li, Han Zhang, Boqing Gong, Hartwig Adam, Xuhui Jia, and Ting Liu. epsilon-vae: Denoising visual decoding. arXiv preprint 1.Xiv22410.04081, 2024.
[74] Jianbin Zheng, Minghui Hu, Zhongyi Fan, Chaoyue Wang, Changxing Ding, Dacheng Tao, and Tat-Jen Cham. Trajectory consistency distillation. arXiv preprint arXiv:2402.1911 2024.
[75] Le Zhuo, Ruoyi Du, Han Xiao, Yangguang Li, Dongyang Liu, Rongjie Huang, Wenze Liu, Lirui Zhao, Yun Wang, Zhanyu Ma, et al. Lumina-next: Making lumina- (2x stronger and faster with next-dit. arXiv preprint arXiv:2406. 185.83 2024.
15

---

Appendix
Related Works
Diflusion Models
1.2
Consistency Models
Proofs
I1.1
Phased Consistency Distillation
11.2
Parameterization Equivalence)
I1.3
Guided Distillation
Distribution Consistency Convergence
Discussions
Numerical Issue of Parameterization
III.2
Why CTM Needs Target Timestep Embeddings"
11.3 Contributions
8
IV Sampling
[V Introducing Randomness for Sampling
IV.2 Improving Diversity for Generation
Societal Impact
V.I Positive Societal Impact
Negative Societal Impact
10
V.3 Safeguards
10
Implementation Details
10
VLI Training Details
10
VI.2 Pseudo Training Code
10
VL.3 Decoupled Consistency Distillation For Video Generation
11
VI.4 Discriminator Design of Image Generation and Video Generation
11
VIlMore Generation Results.
12
Related Works
I.1 Diffusion Models
have gradually become the dominant foundation models in image synthesis. Many works have greatly explored the nature of diffusion models 29
Diffusion models [I
57
20 73
57 22
and
generalize/improve the design space of diffusion models [54] 20 22]. Some works explore the model architecture for diffusion models 7 for text-conditioned synthesis real-world applications 45
75). Some works scale up the diffusion models
52 64
18
36
69
17
61
Some
works explore the sampling acceleration methods, including scheduler-level
20
33
or training-
level [38 56). The formal ones are basically to explore better approximation of the PF
PE
B3 54]

---

The latter are mainly distillation methods 38
46
30
65
70
10
or initializing diffusion weights for
GAN training [49 27 39
1.2 Consistency Models
The consistency model is a new family of generative models [1 56
supporting fast and high quality generation. It can be trained either by distillation direct training without teacher models. Improved techniques even allow consistency training excelling the performance of diffusion training [55]. Consistency trajectory model [21] proposes to learn the trajectory consistency, providing a more flexible framework. Some works combine [24] consistency training with GAN [24] for better training efficiency. Some works adopt continuous consistency models and achieve excelling performance I 62 32 Some works apply the idea consistency model to language model [25]
and policy learning 42 34 Some works extends the application scope of consistency models for
text-conditioned image generation 35
[4]
and cext-conditooned video generation [63 37 71 60).
We notice that recent work multistep consistency models [3] also proposes to splitting the ODE trajectory into multi-parts for consistency learning, and here we hope credit their work for the valuable exploration. However, our work is principally different from multistep consistency models. Since they not open-source their code and weights, we clarify the difference between our work and multistep consistency models based on the details of their technical report. Firstly, they didn 't provide explicit model definitions and boundary conditions nor theoretically show the error bound to prove the soundness of their work. Yet in our work, we believe we have given an explicit definition of important components and theoretically shown the soundness of important techniques applied. For example, they claimed using DDIM for training and inference, while in our work, have shown that although we can also optionally parameterize as the DDIM format, there's 's an intrinsic difference between that parameterization and DDIM (i.e., the difference between exact solution learning and first-order approximation). The aDDIM and invDDIM as highlighted their pseudo code have no relation to PCM. Secondly, they are mainly for the unconditional or class-conditional generation, yet we aim for text-conditional generation large models and dive into the influence of classifier-fres guidance in consistency distillation. Besides, introduce an adversarial loss that aligns well with consistency learning and improves the generation results in few-step settings. We recommend readers read their work for better comparison.
II Proofs
II.1 Phased Consistency Distillation
The following is an extension of the original proof of consistency distillation for phased consistency distillation.
Theorem 1. For arbitrary sub-trajectory [9ms Sm+1l. 1. let At := maxt,, du- 1 Sm +ta+11
by l}, and (-.; be the target phased consistency function induced by the pre-trained diffusiio model (empirical PF-ODE). Assume the f is L-Lipschitz and the ODE solver has local error uniformly bounded by O((t t..) 1+1 with p 2 1. Then, if Lr (0, 0; Φ)) O, we have
sup
11f0 tx) ) 1r (x, t. (((tt))))
Proof. From the condition LPCM (0, 0: for any Xt,, and by bnt1 E [Sms 8m+1), we have
f (xents.tat1) f (2 tn)
(10)
Denote e := f (x1,,tn) - fr (xi,, tni Φ), we have
e
1 (Xi,11: (ocs11:(44) bn+1) - (xt in
S (x kn) - -5 (Xt,,s Kttmyho bni
(11)
SX : bm)) -0 (Xt,s (n)) + fo (Xt,, (n) bni
-0" (x.. bn) - -8 (xt,, bn) + em

---

Considering that fø is L-Lipschitz, we have
le
222 = lle + S0 (x (n) S0 (xe.: 1n)l2
< |1e% 12 15 (x (n) S0 (Xe, bnlll
< l1e 1| + L|| x 1/
(12)
| la + L. (((n++1 )
S ||e ll L(tn+1 tn) (((Atm) )))
Besides, due to the boundary condition, we have
Se (Xmm= 8m) 1 (Xom. 8mi ②))
8m
X
X
(13)
Hence, we have
ee
S
m mm
1l2 + + L- (((I)))))))
E
4+1 - 4
fmlm. [1]
(14)
=0+ 1. ((AA))) (9m+1 8m)
= =O((Atm))))
Theorem 2. For arbitrary set of sub-trajectories (si, 81+1 i=m Let Alm
max,, an 1f(mm 12 [f (f Atm == m mi) (.tt). y. and fm (1, : Φ) be the targer phased consistency function induced by the pre-trained diffusion model (empirical PF-ODE), Assume
the fø fm is L-piscchitz and the ODE solver has local error uniformly bounded by ((tt++1-tn)) -n with 1. Then, if LOCM(0, 0; cb) O, have
sup
mm (x, (x) 1I
fm. (x, tn)l O((Atm,me)))
Proof. From the condition (P 0: Φ)
for any Xt,, and tn, tn+1 E [sms sm+1), we have
fg (xtm111ft111) f (+ (n)
(15)
Denote e
emm'm
== fo
pm
(xt,, tn) - pm.m' (xe,, tii Φ), we have
e e+11
m.m'
f mm
(Xin41: tn+1) fm,m (Xint1s tn+is Φ))
f m'
(x. tn) pm.m' (xl, tni x))
(16)
f mm
(x ba) fon mm (xt,, ba) + f pm. (x,,, bn) fm.. (xt,, bai Φ))
mm (& (n) pm (xt,, bn)
en m.m
Considering that f is L-Lisschitz we have
|| 1e
12 = || mem
+ f mm
(x tn)) f pm.m' (xt,, tn
< || Te m.m'
||2 + mm.
(xt, (n) S mm. (xe,, bn ll1
m,m'
1|2 + Lm +1 (x) tn) sm m' +1 (xt, kn)lla
|| mmm
||2 + L |f
mmm' +2 (x)) tn) fm. +2 (x tn)ll2
(17)
I|em..m
||2 + 1 mmm [1 (x1, 1, tn.) f0h (X... tn)ll2
S Ile m,m'
||2 + I, m. m0 1|x,, x l|2
mm
||2 + L m- ma-mm'+ +1 O 1
- | mmm
||2 + L mm +1 (tn+1 - (xm O((Atm))))
3

---

Hence, we have
le lent1
mm
12 I mmm
1/2 mm-m' m'm (((At)))
Σ
t,41 t
4,,44 8m+m))
le mm
112 L (((At))) (5m+1 8m)
(18)
m.m
112 +(((tt)))
Smc
m
1,m ||2 + (((Atm)))
mm
Where the last equation fmi-1.mi 1,mm (ft" (Xem sm), 9m)
due to the boundary condition form (X sm)
lo
1.m (Xam 5m).
Thereby, we have
le
mmm n+11
12
17
6 pm
1,m mm
1|2 + +(((At
W1
m 2,m'
+ (+(((())))) O((Amm-1)))
8m--
O((A))))
8m/41
(19)
ml (((At))))
i=m
I (m - m' + ())O(AAt
((Atm.. )
II.2 Parameterization Equivalence
We show that the parameterization of Equation is equal to the DDIM inference format
33 54
Theorem 3. Define Fo t, 2xx; axéo(xi, t) SN e~>d then the parameterization has the same format of DDIM X, = 0s xx-oxxx(xx_) + (xee(x(,t).
Proof.
a,eo(xi,)) t) (rr -dA
ag _ X1
Fe (x;x,t,,))
0g
a.Eo(x,, t) (e - )d)))
X1
0s
axEo(xe, t) 01 _00)
X1
(20)
at
at as
as
a,0t
co 1) + (x(o(x,,1)
Xt
at
az
X; oxē0(x;;1) + e,E0(xx,t)
02.
an
IL.3 Guided Distillation
We show the relationship epsilon prediction of consistency models trained with guided distillation and diffusion models when considering the classifier-fres free guidance.

---

Theorem 4. Assume the consistency model is trained by consistency distillation with the teacher diffusion model co. and the ODE solver augmented with CFG value w and null text embedding 3.. Let and Curg be the prompt and negative prompt applied for the inference of consistency model. Then, if the ODE solver is perfect and the CrCM(0,0 0) = 0, we have
co co(x, b, C, Cnegi w') wwl (x(xx,,)) - ((1 www
) (x. t, Creg)
ww co (x 2011
(x, Cueg)
Proof. If the ODE solver is perfect, that means the empirical PF-ODE is exactly the PF-ODE of the training data. Then considering Theorem I and Theorem [ it is apparent that the consistency model will fit the PF-ODE. To show that, considering the case Theorem
| we have
e fo (Xtn+1: -f (Xt=.1+ int1: ())
fō (x), tn) - (Xt,, tni Φ)
fō (x), (m) .0 (xt,, (n) + S0 (tt, tn) (Xt, Φ)
fō (xe tn) - -f0 (Xt, tn) + e
f8 (Xt,: tn) f0 (x1,: tn) en
(21)
en
egm.
O,
where (i) is because the ODE solver perfect. Considering our parameterization Equation then it should be equal to the exact solution in Equation
2 That is,
as ax,=0,,0(x1,)) az
/// e Ad
ag 0s
X1 +
- (2) log PLN
0t
dr
to(
e c-d =
C (x)( (A), (A))dA
(22)
)r
Ax
co(xI,t) SAt e (A),
SK. e
Then we have the epsilon prediction is weighted integral of diffusion- epsilon prediction on the trajectory. Therefore, it is apparent that, for any 1' l and l' the same sub-trajectory with have
é t) cx(xx,t') {')
(23)
When considering the text-conditioned generation and the ODE solver being augmented with the CFG value w, we have
co(X,, t, e) xx (', 2)+ + (xca(xe, t', c) c 2))
(24)
If we additionally apply the classifier-free guidance the consistency models with negative prompt embedding Cacy and CFG value w', we have
Co(x,, c, w')
= Eo(xx,, f, Cneg) + w/ (Éo(Xe, t, c) éx(x,, t, Cnsg))
x (x Ø0) + (x 11, c) - e (x(xx)'' 2))
+ w'(lea(xe 1' 2) + (X
(xy, 1' c) (x,, 22)]
- (x t', Ø + (x c) (xv, 0))]]
(x(xe, 11, 2 + uo( Cneg) (x 11, Ø)) + ww ww'les (xy, (. - (x L', Goeg))
(ew'(ess 11, e) - ((1 (xx, 1', Creg) + aee (xe, 1', 2))) + (x(xx, 1' Coey)
(25)
5

---

11.4 Distribution Consistency Convergence
We firstly show that when LPCM = is achieved, our distribution consistency loss will also converge to zero. Then, we additionally show that, when considering the pre-train data distribution and distillation data distribution mismatch, combining GAN a flawed design. The loss still non-zero when the self-consistency is achieved, thus corrupting the training.
Theorem 5. Denote the data distribution applied for consistency distillation phased is Po. And considering that the forward conditional probability path is defined by 0xX) + TtE, we further define the intermediate distribution Px(x) (Lo (,) .,)) oO). Similarly, denote the data distribution applied for pretraining the diffusion model is IPpretruin (x) and the intermediate distribution following forward process are P (x) (PPtretaii (x) (Peretraing (2)) = (O,,0.) This is reasonable since current large diffusion models are typically trained with much more resources
much larger datasets compared to those of consistency distillation. And, we denote the flow Tems. The and T correspond to our consistency model, pre-trained diffusion model, and the PF-ODE of the data distribution used for consistency distillation, respectively. Additionally, let Tg be the distribution transition following the forward process SDE (adding noise). Then, if the LPCM 0, for arbitrary sub-trajectory [Sm; 11, we have,
CPCM(0, (2 = =D D Tea
Te +s Tea
#IP (utt1 O.
tut1
Proof. Firstly, considering that the forward process Te is equivalent scaling the original variables and then performing convolution operations with (10, ,, I). Therefore, as long as
Th D T +s #I
#I
-mm Ten WP..WPenn (ePeens)) 0,
(26)
then we have
D -o To Tn #IPt.-1 T -0 To mm Thti
To tte # (.Pttut) 0.
(27)
From the condition LreM (0, =0, -0, for any Xt, E Pe, and X E Plaee and tm: tnt1 E (8m, [8m,8m+11 I.. we have
f (x,.))2,011)=== S S0 bn),
(28)
which induces that
T Th ssm #Pr 1att Te tsm
T, I - # bnts
(29)
Therefore, we show that if LPMM (0,0; = 0, then
Cody (0, 0: =0
(30)
Theorem 6. Denote the data distribution applied for consistency distillation phased is Poo And considering that the forward conditional probability path is defined by 0,No + a,C, we further define the intermediate distribution Pe(x) ( 2,) * (0, ot). Similarly, we denote the data distribution applied for pretraining the diffusion model is 1P Ippremmiin (x) and the intermediate distribution following forward process are parettrain (x) = (Ippretrain (x) (pgretrain
( ...) (0,,.)). This is reasonable since current large diffusion models are typically trained with much more resources on
much larger datasets compared to those of consistency distillation. And, we denote the flow T
The and The correspond to consistency model, pre-trained diffusion model, and the PF-ODE of the data distribution used for consistency distillation, respectively. Then, if the LPCM = 0, for arbitrary sub-trajectory [Sms sm+1), we have,
Epdy 0,,, 0; db, m.) +e (hh #Prm.1
Ppm 2 O.

---

Proof. From the condition LPCM(0, 00 = 0, for any Xt, E Pt, and Xin+1 E Plaes and En tx+1 E [sm, Sm I. we have
So (Xtn411441))
f0 (2 tn)
(31)
which induces that
T +1 7m
#1 tn41
En
2m41
(32)
Besides, since Ti corresponds to the PF-ODE of the data distribution used for consistency distilla- tion, we can rewrite
P. Psm. T - 1mmm
#IPt.. tu+1
(33)
Therefore, have
D (T (7 #1 #I [ (7
(Tri0
#P #Pttut
tn+
1
#1Ptw+1)
(34)
Since we know that Po paretrain # Fo. Therefore, there exists m and n achieving the strict inequality. Specifically, if we set 8m = and tn+1 = T, we have
D TT=e#PP #P,, I (PttHPr) D
(P
pprretran Po) 0.
(35)
III Discussions
III. Numerical Issue of Parameterization
Even though our above-discussed parameterization is theoretically sound. it poses a numerical issue when applied to the resilon-prediction based models, especially for the one-step generation. To be specific, for the one-step generation, we are required to predict the solution point Xy with x, from the self-consistency property of consistency models. With epsilon- epsilon-prediction models, the formula can represented as the following
- oico((x,
Xo
(36)
at
However, when inference, should choose = T since it is the noisy timestep that is closest the normal distribution. For the DDPM framework, there should be or & and aT z to make the noisy timestep 7' as close to the normal distribution possible. For instance, the noise schedulers applied by Stable Diffusion v1-5 and Stable Diffusion XL have the or = 0.068265. Therefore, we have
Xy (xi,)) R 14.64(xT - oxTo(xx,,)
X0
(37)
This indicates that we will multiply over 10x to the model prediction. In our experiments, we find the SDXL. more influenced by this issue, tending to produce artifact points or lines.
Generally speaking, using the x0-prediction and v-prediction can well solve these issues. It is obvious for the x0-prediction. For the v-prediction, we have
X = axx; (x,,
(38)
transforming them into the other prediction formats takes additional computation resources and might harm the performance due to the relatively large gaps among different prediction formats.
However, since the diffusion models are trained with the
We solve this issue through a simple way that we set a clip boundary to the value of a, Specifically, we apply
x, 0 (x,,))
Xg
(39)
max 0.5}

---

I11.2 Why CTM Needs Target Timestep Embeddings?
Consistency trajectory model (CTM), as we discussed, proposes to learn how 'move' from arbitrary points on the ODE trajectory to arbitrary other points. Thereby, the model should be aware of "where they are' and 'where they target' for precise prediction.
Basically, they can still follow our parameterization with an additional target timestep embedding.
as x; -x--a. -=x,o(xx; //v ediaa
X, a4
(40)
In this design, we have
Sr: E - (A)
ex( 1,5)
(41)
e
Here, show that dropping the target timestep embedding is flawed design.
Assume we hope to predict the results at timestep s' and timestep s" from timestep and sample x where 5' < s"
If hope to learn both X and x correctly, without the target timestep embedding, we must have
(x1,)) SAL' e (A), SA e-d
(42)
(x JNL" (Epl(aNN) (A))dA
(43)
PA." e-
This will lead to the conclusion that all the segments [4, s] the ODE trajectory have the same weighted epsilon prediction, which violates the truth, ignoring the dynamic variations and dependen- cies specific to each segments.
I11.3
Contributions
Here we re-emphasize the key components of PCM and summarize the contributions of our work.
The motivation of our work is to accelerate the sampling of high-resolution text-to-image and text- to-video generation with consistency models training paradigm. Previous work, latent consistency model (LCM), tried to replicate the power of the consistency model in this challenging setting but did not achieve satisfactory results. We observe and analyze the limitations of LCM from three perspectives and propose PCM, generalizing the design space and tackling all these limitations.
At the heart of PCM is to phase the whole ODE trajectory into multiple phases. Each phase corresponding to a sub-trajectory is treated as an independent consistency model learning objective. We provide standard definition of PCM and show that the optimal error bounds between the prediction of trained PCM and PF-ODE are upper-bounded by (( The phasing technique allows for deterministic multi-step sampling, ensuring consistent generation results under different inference steps.
Additionally, provide deep analysis of the parameterization when converting pre-trained diffusion models into consistency models. From the exact solution format of PF-ODE, we propose a simple yet effective parameterization and show that it has the same formula first-order ODE solver DDIM. However, we show that the parameterization has natural difference from the DDIM ODE solver. That is the difference between exact solution learning and score estimation. Besides, point out that even though the parameterization is theoretically sound, poses numerical issues when building the one-step generator. We propose simple yet effective threshold clip strategy for the parameterization.
We introduce an innovative adversarial loss, which greatly boosts the generation quality a low-step regime. We implement the loss in GAN style. However, show that method has intrinsic

---

difference from traditional GAN. We show that our introduced adversarial loss will converge to zero when the celf-consistency property is achieved. However, the GAN loss will still be non-zero when considering the distribution distance between the pre-trained dataset for diffusion training and the distillation dataset for consistency distillation. We investigate the structures of discriminators and compare the latent-based discriminator and pixel-based discriminator. Our finding is that applying the latent-based visual backbone of pre-traines diffusion U-Net makes the discriminator design simple and can produce better visual results compared to pixel-based discriminators. We also implement similar discriminator structure for the text-to-vided to-eo generation with a temporal inflated U-Net.
For the setting of text-to-imags generation and text-to-video generation, classifier-fre free guidance (CFG) has become an important technique for achieving better controllability and generation quality. We investigate the influence of CFG on consistency distillation. And, to our knowledge, we, for the first time, point out the relations of consistency model prediction and diffusion model prediction when considering the CFG-augmented ODE solver (guided distillation). We show that this technique causes the trained consistency models unable to use large CFG values and be less sensitive to the negative prompt.
We achieve state-of-the-art few-step text-to-image generation and text-to-video generation with only 8 A 800 GPUs, indicating the advancements of our method.
IV
Sampling
IV.1 Introducing Randomness for Sampling
For given initial sample at timestep t belonging to the sub-trajectory [sm: sm+1], we can support deterministic sampling according to the definition of fm...
However, previous work [21 67 reveals that introducing certain degree of stochasticity can lead to better generation results. We also observe a similar trade-off phenomenon in our practice. Therefore, we reparameterize the (x, as
X - (xE0(xx)) 0g5
o. (VTE((X)), + v11 ()-r))), 6 ~~,(
(44)
02
By controlling the value o E [0, we can determine the stochasticity for generation. With r = 1, is pure deterministic sampling. With r = 0, it degrades to pure stochastic sampling. Note that, introducing the random will make the generation results be away from the target induced by the diffusion models. However, since Eg and all follows the normal distribution, we have (xea(xx,1) () + vl1 rre ~ (N(,,)) and thereby the predicted x, should still follow the same distribution P. approximately.
IV.2 Improving Diversity for Generation
Our another observation is the generation diversity with guided distillation is limited compared to original diffusion models. This is due to that the consistency models are distilled with relatively large CFG value for guided distillation. The large CFG value, though known for enhancing the generation quality and text-image alignment, will degrade the generation diversity. We explore simple yet effective strategy by adjusting the CFG values. w (x, t, c) + (1- w we)co t, 2), where w E (0.5, 1]. The epsilon prediction is the convex combination of the conditional prediction and the unconditional prediction.
Societal Impact
V.1 Positive Societal Impact
We firmly believe that our work has profound positive impact on society. While diffusion techniques excel in producing high-quality images and videos, their iterative inference process incurs significant computational and power costs. Our approach accelerates the inference of general diffusion models by up to 20 times more, thus substantially reducing computational and power consumption. Moreover, fosters enthusiasm among creators engaged AI Al-generated content creation and lowers entry barriers.

---

V.2 Negative Societal Impact
Addressing potential negative consequences, given the generative nature of the model, there's 's a risk of generating false or harmful content.
V.3 Safeguards
To mitigate this, we conduct harmful information detection on user-provided text inputs. If harmful prompts detected, the generation process halted. Additionally, our method, fortunately, builds upon the open-source Stable Diffusion model, which includes an internal safety checker for detecting harmful content. This feature significantly enhances our ability prevent the generation of harmful content.
VI Implementation Details
VI.1 Training Details
For the comparison baselines, the training code for InstaFlow BI, SDXL- Turbo 49), SD- Turbo 49), TCD (74), and SDXL-Lighting [27 not open-sourced yet, only compare their open-source weights.
For LCM 35
CTM 21 and our method, they are all implemented by us and trained with the same
configuration.
Specifically, for multi-step models, we trained LoRA with a rank of 64. For models based on SD v1-5, we used a learning rate of 5e-6, batch size of 160, and trained for 5k iterations. For models based on SDXL, we used learning rate of 5e-6, a batch size of 80, and trained for 10k iterations. We did not use EMA for LoRA training.
For single-step models, we followed the approach of SD-Turbo and SDXL-LLihhtnig, training all parameters. For models based SD) v1-5, used learning rate of 5e-6, batch size of 160, and trained for 10k iterations. For models based on SDXL,, we used a learning rate of 1e-6, a batch size of 16, and trained for 50k iterations. We used EMA=0099 for training.
For CTM, we additionally learned a timestep embedding to indicate the target timestep.
For all training settings, we uniformly sample 50 timesteps from the 1000 timesteps of StableDiffusion and apply DDIM the ODE solver.
VI.2 Pseudo Training Code
Algorithm Phased Consistency Distillation with CFG-augmented ODE solver (PCD)
Input: dataset D, initial model parameter 0, learning y,. ODE solver ((s.))))) )) distance metric d(,, :). EMA rate μ. noise schedule ax a.. guidance scale [Wmin Wmas]. number of ODE step ks discretized timesteps = < ta ... ty T, edge timesteps 8u to split the ODE trajectory into M sub-trajectories.
ty (t
Training data: Dx [(x )
0 <-0
repeat
Sample (z, D., U(0 N - kjj and w [uminn Whaxx]
Sample Xt,+t (a 2
Determine (Sm, 8m+1] given
x (1 w) (xe,,, lntks Ln, c) - wVV(x,, |x butts
_.,, = (x(,,x (inee,,nnte bnte, and š.,, So (x. tn,c) c)
Obtain X. and & through adding noise to X and X.m.
C(O, = d(Xem. & &em) A(ReI.U(1 x.) + ReLU(1 xx))
0 Vo.
< stopgrad(110
(1 411 14)
until convergence
10

---

Algorithm 2 Phased Consistency Distillation with normal ODE solver (PCD*)
Input: dataset D, initial model parameter 0, learning rate 1j. ODE solver V( ) distance metric d(.;)) :), EMA rate noise schedule at; o, drop ratio number of ODE step k. discretized timesteps to = = < 4 <
< ... ix T. edge timesteps 8a to S2 SM iN E {t. J2 to split the ODE trajectory into M sub-trajectories.
Training data : Dxx = {(x.
0- -
repeat
Sample (2,c) ~ D,, n U10 N k]
Sample Xt,+k ~ N (ar 2; oi
Determine [8mi, 8m given
~ -0,
if < r then
c Ø
end if
xf t- (xi butke b,.c)
x.m = fe (x butee c) and 8s,, fa- (xf,,
t2, c)
Obtain x. and X. through adding noise to X and Xam.
C(O, 0
d(x &am) ((ReLU(1 + x.) + ReLU((1 - x..)
θ - 0- Vo 0
0 < s0o
+ (1 (1))00
until convergence
VI.3 Decoupled Consistency Distillation For Video Generation
Our video generation model follows the design space of most current text-to-video generation models, viewing videos as temporal stacks of images and inserting temporal blocks to pre-trained text-to- image generation U-Net accommodate 3D features of noisy video inputs. We term this process temporal inflation.
Video generation models typically much more resource-consumingg than image generation models. Also, the overall caption and visual quality of video datasets are generally inferior to those of image datasets. Therefore, we apply the decoupled consistency learning ease the training burden of text-to-video generation, which was first proposed by the previous work AnimateLCM. To be specific, we first conduct phased consistency distillation on stable diffusion v1-5 to obtain the one-step text-to- image generation models. Then we apply the temporal inflation to adapt the text-to-image generation model for video generation. Eventually, we conduct phased consistency distillation on the video dataset. We observe an obvious training speed-up with this decoupled strategy. This allows us to train the state-of-the-ae fast video generation models with only 8 A 800 GPUs.
VL.4 Discriminator Design of Image Generation and Video Generation
VL.4.1 Image Generation
The overall discriminator design was greatly inspired by previous work StyleGAN-T [48], which showed that pre-trained visual backbone can work as great diseriminator. For training the discriminator, we freeze the original weight of pre-trained visual backbones and insert light-weight Convolution- based discriminator heads for training.
Discriminator backbones. We consider two types of visual backbones as the discriminator backbone: pixel-based and latent-based.
We apply DINO 4 as the pixel-based backbone. To be specific, once obtaining the denoised latent code, we decode it through the VAE decoder to obtain the generated image. Then, we resize the generated image into the resolution that DINO trained with (c.g.. 224 x 224) and feed it as the inputs of the DINO backbone. We extract the hidden features of the DINO backbone of different layers and feed them the corresponding discriminator heads. Since DINO is trained in self- supervised manner and no texts are used for training. Therefore, it is necessary to incorporate the text embeddings the discriminator heads for text-to-image generation. To achieve that, we use the embedding of [CLS] token in the last layer of the pre-trained CLIP text encoder, linearly map it to the same dimension of discriminator output, and conduct affine transformation to the discriminator output. However, the pixel-based backbones have inevitable drawbacks. Firstly, it requires mapping
11

---

the latent code to a very -iehndiiensooall image through the VAE decoder, which is much more costly compared to traditional diffusion training. Secondly, can only accept the inputs of clean images, which poses challenges to PCM training, since we require the discrimination of inputs of intermediate noisy inputs. Thirdly, it requires resizing the images into the relatively low resolution it trained with. When applying this backbone design with high-dimensional image generation with XL, we observe unsatisfactory results.
We apply the U-Net of the pre-trained diffusion model as the latent-based backbone. It has several advantages compared the pixel-based backbones. Firstly, trained on the latent space, the U- Net possesses rich knowledge about latent space and can directly work as the discriminator at the latent space, avoiding costly decoding processes, Besides, it can accept the inputs of latent code at different timesteps, which aligns well with the design of PCM and can be applied for regularizing the consistency of all the intermediate distributions instead (i.e., (x)) of distribution at the edge points (i.e., P.. (x)). Additionally, since the text information is already encoded in the U-Net backbone, we do not need to incorporate the text information in the discriminator head, which simplifies the discriminator design. We insert several randomly initialized lightweight simple discriminator heads after each block of the pre-trained U-Net. Each discriminator head consists of two lightweight convolution blocks connected with residuals. Each convolution block is composed of a convolution 2D layer, Group Normalization, and GeLU non-linear activation function. Then we apply a 2D point-wiss convolution layer and set the output channel to 1, thus mapping the input latent feature to a 2D scalar value output map with the same spatial size.
We train our one-step text-to-image model on Stable Diffusion XI. using these two choices of discriminator respectively, while keeping the other settings unchanged. Our experiments reveal that the ditent-based discriminator not only reduces the training cost but also provides more visually compelling generation results. Additionally, note that the teacher diffusion model applied for phased consistency distillation is actually pre-trained U-Net. And we freeze the parameters of U-Net for the training discriminator. Therefore, we can apply the same U-Net, which simultaneously works as the teacher diffusion model for numerical ODE solver computation and discriminator visual backbone.
V1.4.2 Video Generation
For the video discriminator, we mainly follow our design on the image discriminator. We use the same U-Net with temporal inflation the visual backbone for video latent code as well as the teacher video diffusion model for phased consistency distillation. We still apply the 2D convolution layers as discriminator heads for each frame of hidden features extracted from the temporal inflated U-Net since we do not observe performance gain when using 3D convolutions. Note that the temporal relationships are already processed by the pre-traines temporal inflated U-Net, therefore using simple 2D convolution layers as discriminator heads is enough for supervision of the distribution of video inputs.
VII
More Generation Results.
12

---

Prompt: "a happy white man in black suit, sky, red tie, river, mountain, colourful, clouds. best view, fall"
1-step
2-step
4-step
Prompt: "a astronaut walking on the moon"
1-step
2-step
p-step
Figure 10: PCM video generation results with Stable- Dolle-Diffusio v1-5 under inference steps.
13

---

Prompt: "Photo of a dramatic cliffside lighthouse in a storm, waves crashing, symbol of guidance and resilience"
1-step
2-step
4-step
Prompt: "RAW photo, face portrait photo of beautiful 26 y.o woman, cute face, wearing black dress, happy face, hard shadows, cinematic shot, dramatic lighting"
1-step
2-step
p-epep
Figure 11: PCM video generation results with Stable-Diffusiae v1-5 under inference steps.
14

---

Prompt: "a car running on the snowy road"
1-step
2-step
4-step
Prompt: "a monkey eating apple"
1-step
2-step
s-step
Figure 12: PCM video generation results with Sible-Diffusion v1-5 under inference steps.
15

---

Prompt: "Vincent vangogh style, painting, a boy, clouds in the sky"
1-step
2-step
4-step
Prompt: "bonfire, wind, snow land"
1-step
2-step
p-step
Figure 13: PCM video generation results with v1-5 under ~ inference steps.
16

---

Prompt: "a lion on the grass land"
1-step
2-step
4-step
PCM
1-step
2-step
4-step
AnimateLCM
Figure 14: PCM video generation results comparison with AnimateLCM under inference steps.
17

---

Prompt: "river reflecting mountain"
1-step
2-step
4-step
PCM
1-step
2-step
4-step
AnimateLCM
Figure 15: PCM video generation results comparison with AnimateLCM under 1 inference steps.
18

---

1-step Prompt: " monkey living on the tree"
2-step
4-step
8-step
16-step
Prompt: "a lion"
Prompt: "heart-like pink cloud in the sky"
Prompt: "a cat"
Prompti "a red car"
Figure 16: PCM generation results with Diffu-Diffusion v1-5 under different inference steps.
19

---

1-step Prompt: " boy walking by the sea"
2-step
4-step
8-step
16-step
Prompt: "a dog"
Prompt: "a swimming pool"
Prompt: " bike"
Prompti "bedroom"
Figure 17: PCM generation results with Stable- Diffe--iifuuion v1-5 under different inference steps.
20

---

1-step Prompt: a blue bed in the bottle, surrounded by clouds"
2-step
4-et
8-step
16-step
Prompt: "a cat near the sea"
Prompt: "a girl with white dress"
Prompt: "an ice made lion"
Prompt: "Son Goku made of marble"
Figure 18: PCM generation results with Stable- Diffussinf XL under different inference steps.
21

---

1-step Prompt: a pink dog wearing blue sunglasses"
2-step
4-step
8-step
16-step
Prompt: "a dream island, surrounded by sea, with bird flying on the sky"
Prompt: "photography of a man kissing woman, vangogh style"
Prompt: "a pikachu made of wood"
Prompt: "firework in the night, best quality, modern city, river reflecting"
Figure 19; PCM generation results with
XL under different inference steps.
22

---

Figure 20: PCM generation 1-step results with Stable- Diflle-Diffuion v1-5.
23

---

H
Figure 21: PCM generation 2-step results with Sabbe-Diffuusio v1-5.
24

---

Figure 22: PCM generation 4-step results with Stable-Diffusiae Diffusion v1-5.
25

---

Figure 23: PCM generation 8-step results with Sibbe-Diffusion v1-5.
26

---

Figure 24: PCM generation 16-step results with Sibbl--iffuusin v1-5,
27

---

Figure 25: PCM generation 1-step results with Dibbl--iffusion XL.
28

---

Figure 26: PCM generation 2-step results with Sibbe-Diffussin XL.
29

---

Figure 27: PCM generation 4-step results with Stable- Soole-Diffusion XL.
30

---

A
Figure 28: PCM generation 8-step results with Stable- Sofle-Diffusion XL.
31

---

PCM
PCM CCM
Figure 29: PCM generation 16-step results with Stable- Diile-Diffusion XL.
32

---

Figure 30: PCM generation 16-step results with XL.
33

---

Figure 31: Visual examples of ablation study on the proposed distribution consistency loss. Left: Results generated without the distribution consistency loss. Right: Results generated with the distribution consistency loss.
34

---

r=1
0=
0.8
0.7
0.6
r= 0.5
r= 0.4
r= 0.3
0.2
0.1
0.0
0.9
0.8
0.7
= 0.6
7= 0.5
r= 0.4
r= 0.3
r=
0.1
0=0
Figure 32: Visual examples of ablation study the proposed way for stochastic sampling. Pure deterministic sampling algorithms sometimes bring artifacts the generated results. Adding stochas- ticity to a certain degree can alleviate those artifacts.
35

---

103
CFG value 1.0. The entropy of color: 1.75
222
DR
CFG value = 0.6. The entropy of color: 2.16
Figure 33: Visual examples of ablation study on the proposed strategy for promoting diversity. Upper: Batch samples generated with prompt "a car" with normal CFG=1.0 value in 4-step. Lower: Batch sample generated with prompt "a car" with our proposed strategy with CFG=0.6.
36

---

2W
CFG value 1.0. The entropy of color: 1.30
110
CFG value 0.6. The entropy of color: 1.75
Figure 34: Visual examples of ablation study on the proposed strategy for promoting diversity. Upper: Batch samples generated with prompt "a car" with normal CFG=1.0 value in 4-step. Lower: Batch sample generated with prompt "a car" with our proposed strategy with CFG=0.6.
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have claimed in the abstract and introduction that, in this paper. we identify three key flaws in the current design of LCM. We investigate the reasons behind these limitations and propose the Phased Consistency Model (PCM), which generalizes the design space and addresses all identified limitations. Accordingly, Lines 32 to 42 in the introduction of the main paper, we reveal the limitations of LCM, and in Sec. 3 we propose PCM.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Sec. 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
38

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We present complete proof in Sec.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have carefully introduced the framework architecture in Sec. 3.1 the training strategy in Sec. 3.2 and Sec. 3.3 Also, we enclose the pseudo training code, the detailed framework design the Appendix. Eventually, We will release the code and model of this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
39

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code
is
available at https://github. Uttp:///
com/G-U-N
Phased- Consistency-
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co co/
public//guides/fooeettbnmssss
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurlPS code and data submission guidelines (https
//iis. ctappsccc/uubicc/uuii httpa//gguiieee/tttuubmiisssss
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all training and test details in Sec, VI including the training and testing details in Sec. VI.1 the pseudo training scripts in Sec. VI.2 and additional details in
Sec. [V.33and and Sec. VI.4
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars not reported. because it would be too computationally expensive. However, we indeed present plenty qualitative results to demonstrate the statistical significance of the experiments.
Guidelines:
40

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have clarified in the Abstract and Sec. ||| that our results are achieved by only A800 GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
41

---

Justification:
The positive societal impacts are discussed in Sec. V.1 and the negative societal impacts are discussed Sec. V.2
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Yes, we are the image generators, which naturally have a risk of generating harmful content. Therefore, in Sec. V.3 introduce our solution avoid harmful content. We will include the safeguards in our pipeline when releasing the code and model.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the assets used in this paper are public datasets, allowing for research use. Specifically, the used datasets include:
CC3M custom terms reading: The dataset may be freely used for any purpose.
WebVid-2M custom terms reading: Non-Commereial research for 12 months,
42

---

COCO-2014 Creative Commons Attribution 4.0 License.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssii thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not release new asserts in the submission. But we will release the code and model shortly. We will prepare documentation alongside the assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
43

---

Answer:
[NA]
Justification:
We do not involve crowdsourcing, nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
44


Reviews:
[{"review_id": "PjHVe3BMN0", "reviewer": "Reviewer_sXTg", "summary": "The paper investigates three issues of Consistency Models on Latent space, thereby making a proposal named Phased Consistency Model that handles these weaknesses, supported by theoretical proofs and derivations. To assess the efficacy of their proposed solution, the authors conduct extensive experiments on two main scenarios: text-to-image and text-to-video generation. Additionally, details ablation studies and analysis on negative prompting, the self-consistency property and adversarial design to provide a strong motivation for the proposed approach.", "questions": "See the weakness above", "limitations": "The authors analyse the limitations of their method in case of 1-step generation in a few lines.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The method separates consistency objective by splitting the ODE trajectory into sub-trajectories, therefore it has better ability on preserving image consistency when varying inference steps in deterministic sampling. Theoretical proofs provide good bounds for these objectives.\n2. Reasonable arguments come from theoretical perspectives to address the CFG scale problem of Consistency Models.\n3. The proposed Adversarial Consistency Loss for distribution consistency distillation makes sense in design of discriminator’s architecture.", "weaknesses": "1. Pre-defined number of sub-trajectories seems to fix the number of inference steps, which reduces the model’s flexibility.\n2. Lack of explanation for the design of formula of Adversarial Consistency Loss, which employs ReLU function.\n3. Need an ablation study to investigate impact of Adversarial Consistency Loss, for example what is model’s behaviour and result without this loss?\n4. Detailed hyper-parameters used in settings, for example preconditioning weight, Adversarial Consistency Loss weight.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition in our work and for providing constructive feedback on our paper.\n\nQ1: Pre-defined number of sub-trajectories seems to fix the number of inference steps.\n\n1. Note that inherently, each sub-trajectory of PCM can be perceived as a normal CM. Therefore, within each sub-trajectory, we can still support more step refinement just like normal CM. Specifically,  our one-step model has the same flexibility as normal CM (**CM is a subset of PCM**). But for the best efficiency, we only apply one step for each sub-trajectory. \n2. We show in the paper that except for one-step generation, all the other step models can be tuned through a small LoRA adapter with only 4% storage burden of original diffusion models. Therefore, users can switch the inference steps easily.\n\nQ2: ReLU Function\n\nThis is a typo here, and it should be $\\textrm{ReLU}(1 + D (\\tilde{x}_s, c, s)) + \\textrm{ReLU}(1 - D(\\hat{x}_s, c, s))$, where $D$ is the discriminator, $c$ is the text conditions and $s$ is the timestep conditions. The $\\textrm {ReLU}$ function is equivalent to normally applied hinge loss. We apologize if it caused any confusion.\n\nQ3: Ablation study on the adversarial loss.\n\nPlease refer to Fig. 7 , Fig. 9, and Fig. 31 (In the supplementary) in our submission.  In Fig. 7, we show that the adversarial loss greatly improves the FID at low-step regime and coverages the similar performance of PCM without applying the adversarial loss when the number of phases grows. In Fig.9, we show the visual ablation study on the discriminator choice and the comparison of normal GAN loss and our proposed adversarial loss. In Fig. 31, we show the side-by-side 2-step generation results with and without applying the adversarial loss. In Fig.31, we can observe a clear visual quality improvement. We also conduct a human preference metric comparison in the following table. The introduced adversarial loss consistency improves the human evaluation metrics consistently across different inference steps.\n\n| Step | Methods     | HPS   | AES   | PICKSCORE |\n| ---- | ----------- | ----- | ----- | --------- |\n| 1    | PCM w/ adv  | 0.280 | 5.620 | 0.225     |\n|      | PCM w/o adv | 0.251 | 4.994 | 0.206     |\n| 2    | PCM w/ adv  | 0.282 | 5.688 | 0.225     |\n|      | PCM w/o adv | 0.275 | 5.502 | 0.220     |\n| 4    | PCM w/ adv  | 0.284 | 5.645 | 0.228     |\n|      | PCM w/o adv | 0.281 | 5.576 | 0.225     |\n| 8    | PCM w/ adv  | 0.285 | 5.676 | 0.229     |\n|      | PCM w/o adv | 0.283 | 5.637 | 0.227     |\n| 16   | PCM w/ adv  | 0.284 | 5.646 | 0.228     |\n|      | PCM w/o adv | 0.283 | 5.620 | 0.227     |\n\nQ4: Hyper-parameters\n\nFor all training settings, we apply the loss as $\\mathcal L_{\\textrm{PCM}} + 0.1 \\mathcal L_{\\textrm {PCM}}^{adv} $. That is the $\\lambda$ is set to $0.1$ in all training settings. For the $\\lambda (t)$ in $\\mathcal L_{\\textrm{PCM}}$ (Equation 5), we use 1 for simplicity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2UUfEdC44H", "reviewer": "Reviewer_tuze", "summary": "The paper titled \"Phased Consistency Model\" (PCM) introduces a novel model designed to address the limitations of Latent Consistency Models (LCMs) in high-resolution, text-conditioned image generation. The authors identify three primary flaws in LCMs: inconsistency, controllability, and efficiency. PCM is proposed to generalize the design space of LCMs and overcome these limitations by segmenting the ODE trajectory into multiple sub-trajectories and enforcing self-consistency within each sub-trajectory. This approach allows for more stable and deterministic sampling, better control over image generation with larger CFG values, and improved quality in low-step settings through an adversarial loss in the latent space. Extensive experiments demonstrate PCM’s superior performance in both image and video generation tasks compared to existing methods.", "questions": "- Could the author provide some other human related evaluation metrics, e.g., ImageReward, Aesthetic scores, etc, to compare the generation quality within different efficient diffusion models?\n- Given the introduction of an adversarial loss, how stable is the training process of PCM? Are there any observed issues with training stability or convergence, and how are they addressed?", "limitations": "The authors have demonstrated the limitations, e.g., inability in 1-step generation, in their paper.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. **Technical contribution**: PCM introduces a novel approach of segmenting the ODE trajectory, which is a significant advancement over traditional LCMs. This phasing approach effectively tackles the identified limitations of LCMs.\n   \n2. **Comprehensive Evaluation**: The paper provides extensive experimental results on widely recognized benchmarks, showing PCM's superiority in both image and video generation tasks across various settings.\n\n3. **Versatility**: PCM’s methodology is versatile, supporting multi-step refinement and enabling state-of-the-art few-step text-to-video generation.\n\n4. **Clear Illustrations and Comparisons**: Figures and tables in the paper clearly illustrate the performance improvements and qualitative differences between PCM and other models, aiding in understanding the advantages of PCM.", "weaknesses": "No specific weakness to mention.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your very positive review, and constructive suggestions.\n\nQ1: In terms of human evaluation metrics, we have re-evaluated the generation results of our method alongside all the comparative baselines mentioned in the paper. This re-evaluation was conducted over varying steps: 1, 2, 4, 8, and 16, using a range of well-established human evaluation metrics such as HPS-v2 [1], PickScore [2], and Laion Aesthetic Score [3]. Our method consistently achieved the best or at least comparable performance across all tested settings, notably outperforming the consistency model baseline LCM.\n\nExperiments on SD v1-5\n\n| Steps | Methods           | HPS       | AES       | PICKSCORE |\n| ----- | ----------------- | --------- | --------- | --------- |\n| 50    | SD v1-5 (Teacher) | 0.277     | 5.389     | 0.218     |\n| 1     | INSTAFLOW         | 0.267     | 5.010     | 0.207     |\n| 1     | SD TURBO          | 0.276 (1) | 5.445 (1) | 0.223 (1) |\n| 1     | CTM               | 0.240     | 5.155     | 0.195     |\n| 1     | LCM               | 0.251     | 5.178     | 0.201     |\n| 1     | PCM               | 0.276 (1) | 5.389 (2) | 0.213 (2) |\n| 2     | INSTAFLOW         | 0.249     | 5.050     | 0.196     |\n| 2     | SD TURBO          | 0.278 (1) | 5.570 (1) | 0.226 (1) |\n| 2     | CTM               | 0.267     | 5.117     | 0.208     |\n| 2     | LCM               | 0.266     | 5.135     | 0.210     |\n| 2     | PCM               | 0.275 (2) | 5.370 (2) | 0.217 (2) |\n| 4     | INSTAFLOW         | 0.243     | 4.765     | 0.192     |\n| 4     | SD TURBO          | 0.278 (2) | 5.537 (1) | 0.224 (1) |\n| 4     | CTM               | 0.274     | 5.189     | 0.213     |\n| 4     | LCM               | 0.273     | 5.264     | 0.215     |\n| 4     | PCM               | 0.279 (1) | 5.412 (2) | 0.217 (2) |\n| 8     | INSTAFLOW         | 0.267     | 4.548     | 0.189     |\n| 8     | SD TURBO          | 0.276 (2) | 5.390 (2) | 0.221 (1) |\n| 8     | CTM               | 0.271     | 5.026     | 0.210     |\n| 8     | LCM               | 0.274     | 5.366     | 0.216     |\n| 8     | PCM               | 0.278 (1) | 5.398 (1) | 0.218 (2) |\n| 16    | INSTAFLOW         | 0.237     | 4.437     | 0.187     |\n| 16    | SD TURBO          | 0.277 (1) | 5.275     | 0.219 (1) |\n| 16    | CTM               | 0.270     | 4.870     | 0.209     |\n| 16    | LCM               | 0.274     | 5.352 (2) | 0.216     |\n| 16    | PCM               | 0.277 (1) | 5.442 (1) | 0.217 (2) |\n\nExperiments on SD XL\n\n| Steps   | Methods        | HPS       | AES       | PICKSCORE |\n| ------- | -------------- | --------- | --------- | --------- |\n| 1-step  | SDXL Lightning | 0.278     | 5.65 (1)  | 0.223     |\n| 1-step  | SDXL TURBO     | 0.279 (1) | 5.40      | 0.228 (1) |\n| 1-step  | CTM            | 0.239     | 4.86      | 0.201     |\n| 1-step  | LCM            | 0.205     | 5.04      | 0.2006    |\n| 1-step  | PCM            | 0.28 (1)  | 5.62 (2)  | 0.225 (2) |\n| 2-step  | SDXL Lightning | 0.28      | 5.72 (1)  | 0.227 (1) |\n| 2-step  | SDXL TURBO     | 0.281 (2) | 5.46      | 0.226 (2) |\n| 2-step  | CTM            | 0.267     | 5.58      | 0.216     |\n| 2-step  | LCM            | 0.265     | 5.40      | 0.217     |\n| 2-step  | PCM            | 0.282 (1) | 5.688 (2) | 0.225     |\n| 4-step  | SDXL Lightning | 0.281     | 5.76 (2)  | 0.228 (1) |\n| 4-step  | SDXL TURBO     | 0.284 (1) | 5.49      | 0.224     |\n| 4-step  | CTM            | 0.278     | 5.84 (1)  | 0.221     |\n| 4-step  | LCM            | 0.274     | 5.48      | 0.223     |\n| 4-step  | PCM            | 0.284 (1) | 5.645     | 0.228 (2) |\n| 8-step  | SDXL Lightning | 0.282     | 5.75 (2)  | 0.229 (1) |\n| 8-step  | SDXL TURBO     | 0.283 (2) | 5.59      | 0.225     |\n| 8-step  | CTM            | 0.276     | 5.88 (1)  | 0.218     |\n| 8-step  | LCM            | 0.277     | 5.57      | 0.223     |\n| 8-step  | PCM            | 0.285 (1) | 5.676     | 0.229 (2) |\n| 16-step | SDXL Lightning | 0.28 (2)  | 5.72 (2)  | 0.225 (2) |\n| 16-step | SDXL TURBO     | 0.277     | 5.56      | 0.219     |\n| 16-step | CTM            | 0.274     | 5.85 (1)  | 0.215     |\n| 16-step | LCM            | 0.276     | 5.64      | 0.221     |\n| 16-step | PCM            | 0.284 (1) | 5.646     | 0.228 (1) |\n\nQ2:  Regarding the training stability,  we did not observe any training instability, even when using a very small batch size. For example, we trained the SDXL version of PCM with a batch size of 16 using fp16 mixed precision and achieved comparable performance to SDXL-Lightning one-step, which uses a batch size of 512 (32 times larger). We are willing to submit our training code for SD v1-5, SDXL, and SD3 if needed. All models showed clear improvements within 2,000 iterations and consistently improved with longer training times. We believe this stability is due to our specially designed adversarial loss rather than the standard GAN loss.  However, as shown in our ablation study in Fig. 7, when replacing our proposed adversarial loss with normal GAN loss, we observe unstable training. We analyze the phenomenon from the perspective of distribution mismatch between pretraining datasets and distillation datasets in our paper (Theorem 5 & 6.).\n\n[1] Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.\n\n[2] Pick-a-pic: An open dataset of user preferences for text-to-image generation. NeurIPS 2023.\n\n[3] Laion-aesthetics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7WdDyCdh5s", "reviewer": "Reviewer_ukQK", "summary": "The paper introduces the Phased Consistency Model (PCM), which enhances stability and speed in high-resolution image and video generation by improving the design of consistency (trajectory) models. \nThe main improvement of PCM is a new Parameterization of consistency function and the phase trajectory.  There is also a new method to improve the utilization of CFG. \nExperimental results show that PCM performs better across 1 to 16-step generation settings.", "questions": "please see Weaknesses", "limitations": "The author has already discussed limitations.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The architecture of the method is straight-forward and clear. \n\n2. The insights gained from observing the multi-step process are commendable.\n\n3. The results are solid and demonstrate the effectiveness of the proposed approach.\n\n4. Detailed proofs have been provided for all the hypotheses.", "weaknesses": "Despite the strong experimental results, I still have many reservations about this work.\nThe primary concern stems from the inaccurate description and the incomplete experiments.\n\nHere's my MAJOR concerns:\n\n1. The author also asserts that the findings are derived based on CTM. However, the author does not provide any comparisons with the official CTM, e.g., ImageNet or CIFAR. \n\n2. Another concern is that the author claims to have compared the results with SD15 CTM and SDXL CTM. However, I could not find any related descriptions or implementations in the official CTM publications or GitHub repository.  The authors claimed such models are implemented by theirs but there's no detail. Specifically, the author stated they *additionally learned a timestep embedding to indicate the target timestep* but without any details and reasons.\n\n3. A further question about implement detail: 1-step PCM and multi-step PCM are trained separately? If so, only the range of training parameters different? Are there any further differences?\n\n4. The author appears to misunderstand the concept of the negative prompt, e.g. Controllability in the LCM drawbacks. Furthermore, the provided results do not demonstrate that PCM can mitigate the impact of the negative prompt. \n\n5. The author does not clearly explain the differences between CTM and PCM, especially without a clear description of implementation of CTM in SD15 and SDXL. \nBased on CTM, the novelty of this work on my side is 4-fold: a re-parameterization method, an improved GAN structure, an edged timestep and a CFG guidance method. \n\n\nIn conclusion,\nthe main problem is the authors **did not** conduct an objective comparison and the work lacks sufficient quantitative ablation analysis with official CTM on ImageNet or CIFAR. From the existing data, it is difficult for me to determine whether the proposed solutions by the authors are truly effective.\n\n\nHere's some minor questions:\n\n1. The author relied heavily on FID and CLIP metrics for comparison, but these have known limitations, particularly in large-scale text-to-image applications. More contemporary metrics, such as HPS or PICK-SCORE, should be used to demonstrate the effectiveness.\n\n2. Additionally, I suggest author to use recall to demonstrate the diversity of the model. The introduction of additional GAN modules can lead to a lack of diversity in the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the in-depth review. We appreciate the chance to more comprehensively compare PCM with CTM.\n\nQ1: \n\n**CIFAR-10**\n\nWe additionally implement the core idea \"phasing\" technique into CIFAR (unconditional generation) and ImageNet (conditional generation). For CIFAR, we train our models with a batch-size 256 for 200, 000 iterations (800, 000 iterations used in official CM), without any additional GAN training. We listed the FID and inception score comparison in the following table.\n\n| Methods       | NFE  | FID  | IS |\n| - | - | - | - |\n| EDM           | 35   | 2.04 | 9.84            |\n| Score SDE     | 2000 | 2.20 | 9.89            |\n| CD            | 1    | 3.55 | 9.48            |\n| CD            | 2    | 2.93 | 9.75            |\n| CTM w/ GAN ++ | 1    | 1.98 | -               |\n| CTM w/ GAN ++ | 2    | 1.87 | -               |\n| CTM w/ GAN    | 1    | 2.28 | -               |\n| CTM w/ GAN    | 18   | 2.23 | -               |\n| CTM w/o GAN   | 1    | 5.19 | -               |\n| CTM w/o GAN   | 18   | 3.00 | -               |\n| Ours w/o GAN  | 1    | 2.95 | 9.72            |\n| Ours w/o GAN  | 2    | 2.14 | 9.76            |\n| Ours w/o GAN  | 4    | 2.04 | 9.90            |\n| Ours w/o GAN  | 8    | 1.95 | 9.92            |\n\nIn the above table, \"CTM w/ GAN\" and \"CTM w/o GAN\" are obtained from Table 3 of supplementary of CTM paper, tagged with \"**fair comparison**\" in the title. \"CTM w/ GAN ++\" is obtained from Table 1 of the paper, with more advanced techniques including classifier rejection sampling. We did not apply these techniques to our method. We can observe that the success of CTM greatly relies on GAN training. Yet, we show that our method, without any GAN training, can achieve comparable performance through multistep refinement.\n\n**ImageNet**\n\n| Methods                              | NFE  | FID  | Recall |\n| -| - | - | - |\n| EDM                                  | 79   | 2.44 | 0.67   |\n| ADM                                  | 250  | 2.07 | 0.64   |\n| BigGAN-deep                          | 1    | 4.06 | 0.48   |\n| StyleGAN-XL                          | 1    | 2.09 | 0.52   |\n| CD (L2)                         | 1    | 12.5 | -      |\n| CD (L2)                         | 2    | 6.8  | -      |\n| CD (L2)                         | 4    | 5.2  | -      |\n| CD (LPIPS)                           | 1    | 6.2  | 0.63   |\n| CD (LPIPS)                           | 2    | 4.7  | 0.64   |\n| CD (LPIPS)                           | 4    | 4.3  | 0.64   |\n| CTM w/ GAN ++                        | 1    | 1.92 | 0.57   |\n| CTM w/ GAN ++                        | 2    | 1.73 | 0.57   |\n| Ours w/o GAN (3.5 % Training budget) | 4    | 3.7  | 0.66   |\n\nFor ImageNet, due to the limited time and GPU resources, we can only afford to run a tentative experiment. Specifically, we run a 4-phased PCM by adapting the official repo of CM with a normal L2 loss. For the last phase, we apply LPIPS loss. We first train the model for 100,000 iterations with a batch size of 64 and further train the model for 50, 000 iterations with a batch size of 1024. Note that the official CM trains the model with batch size 2048 for 800, 000 iterations. Therefore the current training budget for our method is 3.5% of the official CM. But importantly, we achieve even superior performance than the official CD. We believe this is clear evidence of the effectiveness of the \"Phasing\" ODE trajectory.\n\nQ2 & Q5:  The implementation details of CTM are the same as LCM and PCM, with the only difference on the loss design. We use the following table to show the difference.\n\n| Methods | timestep sampled                                      | previous timestep | target timestep                                    |\n| - | - | -| - |\n| CM      | $t_{n+1}\\sim\\rm{Uniform}(\\{t_1, t_2, \\dots\\})$     | $t_{n}$            | $s:=t_1:=0$                                         |\n| CTM     | $t_{n+1}\\sim\\rm{Uniform}(\\{t_1, t_2, \\dots\\})$     | $t_{n}$            | $s\\sim\\rm{Uniform}(\\{t_1, t_2, \\dots, t_{n}\\})$ |\n| PCM     | $t_{n+1}\\sim\\rm{Uniform}(\\{s_m, \\dots, s_{m+1}\\})$ | $t_{n}$            | $s := s_{m}$                                        |\n\nwhere $s_m, m=0,1,\\dots$  are pre-defined edge timesteps. Note that for both LCM and PCM, the target timesteps are fixed. While in CTM, the target timesteps are randomly chosen. Therefore it requires target timesteps embeddings. Besides, in the official CTM code base, the model can also optionally receive the target timestep embeddings. We also provide more theoretical proof in our submission, please see \"III.2 Why CTM Needs Target Timestep Embeddings?\".  To put it straightforwardly, CTM does not require a target timestep embedding if and only if the ODE trajectory is first-order ( i.e., it is straight), which violates the truth since ODE trajectories are typically higher orders.\n\nQ3: Training. No further difference. The LoRA has limited capacity, and therefore it is not enough for stable one-step generation.\n\nQ4: Negative prompt. Please refer to Ln 289-296 ablation study of our submission and Figure 6. We show that our model has a higher CLIP score on positive prompts lower CLIP  score on negative prompts compared with LCM.\n\nQ6: Human metrics. Please refer to our rebuttal to Reviewer tuze.\n\nQ7: Regarding recall, we have the following interesting findings. When computing the recall with coco, we find the introduced adversarial loss reduces the recall slightly. But when computing the recall with generation results of the teacher diffusion models, we find the adversarial loss increases the recall at a low-step regime, which we attributed to the quality improvements.\n\n| Dataset |    | 1step | 2step | 4step | 8step | 16step |\n|--|-|-|-|-|-|-|\n| COCO    | adv         | 0.535  | 0.679 | 0.626 | 0.611 | 0.609   |\n|    | w/o adv | 0.585  | 0.690 | 0.686 | 0.647 | 0.637   |\n| SD      | adv         | 0.819 | 0.824 | 0.813  | 0.821  | 0.825   |\n| | w/o adv | 0.608 | 0.707 | 0.793  | 0.849  | 0.861   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their valuable comments and suggestions. We are sincerely grateful to the reviewers for dedicating their time and effort to review our work.  We are delighted to see reviewers commenting on our paper with "detailed proofs", "solid and comprehensive experimental results", "clear and straight-forward illustrations", and "versatile approach". 

---



In this rebuttal, we try our best to solve the concerns of reviewers. We summarize the important and common concerns in the following:

**Human Evaluation Metrics. (Reviewer tuze, Reviewer ukQK)**: We re-test the generation results of our method and all compared baselines listed in the paper across 1, 2, 4, 8, and 16 steps with widely applied human evaluation metrics including HPS-v2 [1],  PickScore [2], Laion Aesthetic Score [3]. Our method consistently achieves the best or comparable performance with compared methods in all compared settings, significantly surpassing the consistency model baseline LCM [4]. 

**Experiments on CIFAR [5] and ImageNet [6] for more comprehensive comparison. (Reviewer ukQK)** 

We additionally implement the core idea "phasing" technique into CIFAR-10 (unconditional generation) and ImageNet (conditional generation). We try our best to run as many experiments as possible during the limited rebuttal period.

For CIFAR-10, we train our models with a batch-size 256 (512 used in official CM [7]) for 200, 000 iterations (800, 000 iterations used in official CM), without applying any additional GAN training. Our two-step results achieve FID 2.14 with an inception score of 9.76, significantly surpassing the CM (FID 2.93). Our four-step results achieve FID 2.04 with an inception score of 9.90, achieving comparable performance with 35-step results of EDM [8] (FID 2.04, inception score of 9.88).

For ImageNet, due to the limited time and GPU resources, we can only afford to run a tentative experiment. Specifically, we run a 4-phased PCM by adapting the official repo of CM with a normal L2 loss. For the last phase, we apply LPIPS loss. We first train the model for 100,000 iterations with a batch-size 64 and further train the model for 50, 000 iterations with a batch-size 1024. Note that the official CM train the model with batch-size 2048 for 800, 000 iterations. Therefore the current training budget for our method is 3.5% of the offical CM. However, our 4-step model achieves FID 3.7,  surpassing the 4-step results of official CM with LIPIPS loss (FID 4.3) and L2 loss (FID 5.2). We believe this is clear evidence of the effectiveness of the "Phasing" ODE trajectory.

We list a table in the following rebuttal for a more comprehensive and detailed comparison with CM, CTM [9], and other baselines. 

**Training Stability. (Reviewer tuze)** We didn't observe any training instability even training our approach with a very small bach-size. For example, we train the SDXL version PCM with a batch-size 16 using fp16 mixed precisions but achieve comparable performance with SDXL-Lightning [10] one-step which uses batch size 512 (32 times larger). We are willing to submit our training code on SD v1-5, SDXL, and SD3 if needed. All models can observe clear improvements within 2,000 iterations and consistently improve as the training time is longer. We believe the stability can be attributed to our specially designed adversarial loss instead of normal GAN loss.

**Impact of Adversarial Loss. (Reviewer sXTg)**

Please refer to Fig. 7 , Fig. 9, and Fig. 31 (In the supplementary) in our submission.  In Fig. 7, we show that the adversarial loss greatly improves the FID at low-step regime and coverages the similar performance of PCM without applying the adversarial loss when the number of phases grows. In Fig. 9, we show the visual ablation study on the discriminator choice and the comparison of normal GAN loss and our proposed adversarial loss. In Fig. 31, we show the side-by-side 2-step generation results with and without applying the adversarial loss. In Fig. 31, we can observe a clear visual quality improvement. Our further quantitative human evaluation metrics in the following rebuttal also aligns with these visual example ablations. 

 

Please refer to the following rebuttals for other specific concerns and more details. We looking forward to your further reply and discussion. 

Sincerely,

Authors of submission 2513

---



[1] Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. 

[2] Pick-a-pic: An open dataset of user preferences for text-to-image generation. NeurIPS 2023.

[3] Laion-aesthetics. 

[4] Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference. 

[5] Learning multiple layers of features from tiny images.

[6] Imagenet: A large-scale hierarchical image database. CVPR 2009.

[7] Consistency Models. ICML 2023.

[8] Elucidating the Design Space of Diffusion-Based Generative Models. NeurIPS 2022.

[9] Consistency Trajctory Models: Learning Probability Flow ODE. ICLR 2024.

[10] SDXL-Lightning: Progressive Adversarial Diffusion Distillation

Meta-review: One reviewer recommended rejection and also flagged a potential plagiarism concern. Upon further investigation the AC did not find evidence of plagiarism.

Amongst the other reviewers, one recommended strong accept and the other recommended weak accept. As mentioned by these reviewers, the paper introduces a novel method of generating multiple self consistent sub-trajectories and achieves strong distillation performance on image generation while also showing applicability on video generation tasks. The AC agrees with these evaluations and recommends acceptance.
Decision: Accept (poster)