Phased Consistency Models

Authors: fu-yun  wang
zhaoyang  huang
alexander william bergman
dazhong  shen
peng  gao
michael  lingelbach
keqiang  sun
weikang  bian
guanglu  song
yu  liu
xiaogang  wang
hongsheng  li
  mmlab
avolution  ai
  hedra
gelu  conv
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 15:03:06

Abstract: Consistency Models (CMs) have made significant progress in accelerating the generation of diffusion models. However, their application to high-resolution, text-conditioned image generation in the latent space remains unsatisfactory. In this paper, we identify three key flaws in the current design of Latent Consistency Models~(LCMs). We investigate the reasons behind these limitations and propose Phased Consistency Models (PCMs), which generalize the design space and address the identified limitations. Our evaluations demonstrate that PCMs outperform LCMs across 1--16 step generation settings. While PCMs are specifically designed for multi-step refinement, they achieve comparable 1-step generation results to previously state-of-the-art specifically designed 1-step methods. Furthermore, we show the methodology of PCMs is versatile and applicable to video generation, enabling us to train the state-of-the-art few-step text-to-video generator. Our code is available at https://github.com/G-U-N/Phased-Consistency-Model.

Main paper: 
 introduction: 
 Diffusion models [20, 15, 57, 68] have emerged as the dominant methodology for image synthesis [45, 41, 7] and video synthesis [22, 53, 52, 63] . These models have shown the ability to generate highquality and diverse samples conditioned on varying signals. At their core, diffusion models rely on an iterative evaluation to generate new samples. This iterative evaluation trajectory models the probability flow ODE (PF-ODE) [56, 57] that transforms an initial normal distribution to a target real data distribution. However, the iterative nature of diffusion models makes the generation of new samples time-intensive and resource-consuming. To address this challenge, consistency models [56, 11, 63, 55] have emerged to reduce the number of iterative steps required to generate samples. These models work by training a model that enforces the self-consistency [56] property: any point along the same PF-ODE trajectory shall be mapped to the same solution point. These models have been extended to high-resolution text-to-image synthesis with latent consistency models (LCMs) [35] . Despite the improvements in efficiency and the ability to generate samples in a few steps, the sample quality of such models is still limited.We show that the current design of LCMs is flawed, causing inevitable drawbacks in controllability, consistency, and efficiency during image sampling. Fig. 2 illustrates our observations of LCMs. The limitations are listed as follows:(1) Consistency. Due to the specific consistency property, CMs can only use the purely stochastic multi-step sampling algorithm, which assumes that the accumulated noise variable in each generative step is independent and causes varying degrees of stochasticity for different inferencestep settings. As a result, we can find inconsistency among the samples generated with the same seeds in different inference steps. (2) Controllability. Even though diffusion models can adopt classifier-free guidance (CFG) [16] in a wide range of values (i.e. 2-15), equipped with weights of LCMs, they can only accept values of CFG within range of 1-2. Larger values of CFG would cause the exposure problem. This brings difficulty for the hyper-parameter selection. Additionally, we find that LCMs are insensitive to the negative prompt. As shown in the figure, LCMs still generate black dogs even when the negative prompt is set to "black dog". Both phenomenon reduce the controllability on generation. We show the reason behind this is the CFG-augmented ODE solver adopted in the consistency distillation stage. (3) Efficiency. LCMs tend to generate much inferior samples at the few-step settings, especially in less than 4 inference steps, which limits the sampling efficiency. We argue that the reason lies in the traditional L2 loss or the Huber loss used in the LCMs procedure, which is insufficient for the fine-grained supervision in few-step settings.To this end, we propose Phased Consistency Models (PCMs), which can tackle the discussed limitations of LCMs and are easy to train. Specifically, instead of mapping all points along the ODE trajectory to the same solution, PCMs phase the ODE trajectory into several sub-trajectories and only enforce the self-consistency property on each sub-trajectory. Therefore, PCMs can sample [15] , consistency models [56] , consistency trajectory models [21] , and our phased consistency model. (Right) Simplified visualization of the forward SDE and reverse-time PF-ODE trajectories.samples along the solution points of different sub-trajectories in a deterministic manner without error accumulation. As an example shown in Fig. 3 , we train PCMs with two sub-trajectories, with three edge points including x T , x 0 , and x ⌊T /2⌋ selected. Thereby, we can achieve 2-step deterministic sampling (i.e., x T → x ⌊T /2⌋ → x 0 ) for generation. Moreover, the phased nature of PCMs leads to an additional advantage. For distillation, we can choose to use a normal ODE solver without the CFG alternatively in the consistency distillation stage, which is not viable in LCMs. As a result, PCMs can optionally use larger values of CFG for inference and be more responsive to the negative prompt. Additionally, to improve the sample quality for few-step settings, we propose an adversarial loss in the latent space for more fine-grained supervision.To conclude, we dive into the design of (latent) consistency models, analyze the reasons for their unsatisfactory generation characteristics, and propose effective strategies for tackling these limitations.We validate the effectiveness of PCMs on widely recognized image generation benchmarks with stable diffusion v1-5 (0.9 B) [45] and stable diffusion XL (3B) [41] and video generation benchmarks following AnimateLCM [63] . Vast experimental results show the effectiveness of PCMs.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We have claimed in the abstract and introduction that, in this paper, we identify three key flaws in the current design of LCM. We investigate the reasons behind these limitations and propose the Phased Consistency Model (PCM), which generalizes the design space and addresses all identified limitations. Accordingly, in Lines 32 to 42 in the introduction of the main paper, we reveal the limitations of LCM, and in Sec. 3 we propose PCM.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
preliminaries: 
 Diffusion models define a forward conditional probability path, with a general representation of α t x 0 + σ t ϵ for intermediate distribution P t (x|x 0 ) conditioned on x 0 ∼ P 0 , which is equivalent to the stochastic differential equation dx t = f t x t dt + g t dw t with w t denoting the standard Winer process, f t = d log αt dt and g 2 t = dσ 2 t dt -2 d log αt dt σ 2 t . There exists a deterministic flow for reversing the transition, which is represented by the probability flow ODE (PF-ODE) dx = f t -g 2 t /2∇ x log P t (x) dt. In standard diffusion training, a neural network is typically learned to estimate the score of marginal distribution P t (x t ), which is equivalent tos ϕ (x, t) ≈ ∇ x log P t (x) = E x0∼P(x0|x) [∇ x log P t (x|x 0 )].Substitue the ∇ x log P t (x) with s ϕ (x, t), and we get the empirical PF-ODE. Famous solvers including DDIM [54] , DPM-solver [33] , Euler and Heun [20] can be generally perceived as the approximation of the PF-ODE with specific orders and forms of diffusion for sampling in finite discrete inference steps. However, when the number of discrete steps is too small, they face inevitable discretization errors.Consistency models [56] , instead of estimating the score of marginal distributions, learn to directly predict the solution point of ODE trajectory by enforcing the self-consistency property: all points at the same ODE trajectory map to the same solution point. To be specific, given a ODE trajectory {x t } t∈[ϵ,T ] , the consistency models f θ (•, t) learns to achieve f θ (x t , t) = x ϵ by enforcingf θ (x t , t) = f θ (x t ′ , t ′ ) for all t, t ′ ∈ [ϵ, T ]. A boundary condition f θ (x ϵ , ϵ) = x ϵis set to guarantee the successful convergence of consistency training. During the sampling procedure, we first sample from the initial distribution x T ∼ N (0, I). Then, the final sample can be generated/refined by alternating denoising and noise injection steps, i.e,EQUATIONwhere {τ k } K k=1 are selected time points in the ODE trajectory and τ K = T . Actually, because only one solution point is used to conduct the consistency model, it is inevitable to introduce stochastic error, i.e., η, for multiple sampling procedures. Recently, Consistency trajectory models (CTMs) [21] propose a more flexible framework. specifically, CTMs f θ (•, t, s) learns to achieve f θ (x t , t, s) = f (x t , t, s; ϕ) by enforcing f θ (x t , t, s) = f θ (x t ′ , t ′ , s) with s ≤ min(t, t ′ ) and t, t ′ ∈ [ϵ, T ]. However, the learning objectives of CTMs are redundant, including many trajectories that will never be applied for inference. More specifically, if we split the continuous time trajectory into N discrete points, diffusion models learn O(N ) objectives (i.e., each point learns to move to its adjacent point), consistency models learn O(N ) objectives (i.e., each point learns to move to solution point), and consistency trajectory models learn O(N 2 ) objectives (i.e., each point learns to move to all the other points in the trajectory). Hence, except for the current timestep embedding, CTMs should additionally learn a target timestep embedding, which is not comprised of the design space of diffusion models.Different from the above approaches, PCMs can be optimized efficiently and support deterministic sampling without additional stochastic error. Overall, Fig. 3 illustrates the difference in training and inference processes among diffusion models, consistency models, consistency trajectory models, and phased consistency models.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of work in Sec. 5 Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory Assumptions and Proofs some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
method: 
 In this section, we introduce the technical details of PCMs, which overcome the limitations of LCMs in terms of consistency, controllability, and efficiency. Consistency: Specifically, we first introduce the main framework of PCMs, consisting of definition, parameterization, the distillation objective, and the sampling procedure in Sec. 3.1. In particular, by enforcing the self-consistency property in multiple ODE sub-trajectories respectively, PCMs can support deterministic sampling to preserve image consistency with varying inference steps. Controllability: Secondly, in Sec. 3.2, to improve the controllability of text guidance, we revisit the potential drawback of the guided distillation adopted in LCMs, and propose to optionally remove the CFG for consistency distillation. Efficiency: Thirdly, in Sec. 3.3, to further improve inference efficiency, we introduce an adversarial consistency loss to enforce the modeling of data distribution, which facilitates 1-step generation.Definition. For a solution trajectory of a diffusion model {x t } t∈[ϵ,T ] following the PF-ODE, we split the trajectory into multiple sub-trajectories with hyper-defined edge timesteps s 0 , s 1 , . . . , s M , where s 0 = ϵ and s M = T . The M sub-trajectories can be represented as {x t } t∈[sm,sm+1] with m = 0, 1, . . . , M -1. We treat each sub-trajectory as an independent CM and define the consistency function asf m : (x t , t) → x sm , t ∈ [s m , s m+1 ].We learn f m θ to estimate f by enforcing the self-consistency property on each sub-trajectory that its outputs are consistent for arbitrary pairs on the same sub-trajectory. Namely, f m (x t , t) = f m (x t ′ , t ′ ) for all t, t ′ ∈ [s m , s m+1 ]. Note that the consistency function is only defined on the sub-trajectory. However, for sampling, it is necessary to define a transition from timestep T (i.e., s M ) to ϵ (i.e., s 0 ). Thereby, we defined f m,m ′ = f m ′ • • • f m-2 f m-1 (f m (x t , t) , s m ), s m-1 • • • , s m ′ thatf m θ (x, t) as f m θ (x t , t) = c m skip (t)x r + c m out (t)F θ (x t , t, s m ), where c m skip (t) gradually increases to 1 and c m out (t) gradually decays to 0 from timestep s m+1 to s m . Another important thing is how to parameterize F θ (x t , t, s), basically it should be able to indicate the target prediction at timestep s given the input x at timestep t. Since we build upon the epsilon prediction models, we hope to maintain the epsilon-prediction learning target.For the above-discussed PF-ODE, there exists an exact solution [33] from timestep t to sEQUATIONwhere λ t = ln αt σt and t λ is a inverse function with λ t . The equation shows that the solution of the ODE from t to s is the scaling of x t and the weighted sum of scores. Given a epsilon prediction diffusion network ϵ ϕ (x, t), we can estimate the solution as x s = αs αt x t -α s λs λt e -λ ϵ ϕ (x t λ (λ) , t λ (λ))dλ. However, note that the solution requires knowing the epsilon prediction at each timestep between t and s, but consistency modes need to predict the solution with only x t available with single network evaluation. Thereby, we parameterize the F θ (x, t, s) as following,x s = α s α t x t -α s εθ (x t , t)λs λt e -λ dλ .One can show that the parameterization has the same format with DDIM Theorem 3) . But here we clarify that the parameterization has an intrinsic difference from DDIM. DDIM is the first-order approximation of solution ODE, which works because we assume the linearity of the score in small intervals. This causes the DDIM to degrade dramatically in few-step settings since the linearity is no longer satisfied. Instead, our parameterization is not approximation but exact solution learning. The learning target of εθ (x, t) is no more the scaled score -σ t ∇ x log P t (x) (which epsilon-prediction diffusion models learn to estimate) butx s = α s ( xt-σtϵ θ (xt,t) αt ) + σ s ϵ θ (x t , t) (seeλs λ t e -λ ϵ ϕ (x t λ (λ) ,t λ (λ))dλ λs λ t e -λ dλ. Actually, we can define the parameterization in other formats, but we find this format is simple and has a small gap between the original diffusion models. The parameterization of F θ also allows for a better property that we can drop the introduced c m skip and c m out in consistency models to ease the complexity of the framework. Note that following Eq. 3, we can get F θ (x sm , s m , s m ) = αs m αs mx sm -0 = x sm . Therefore, the boundary condition is already satisfied. Hence, we can simply define f m θ (x, t) = F θ (x, t, s m ). This parameterization also aligns with several previous diffusion distillation techniques [46, 3, 74] utilizing DDIM format, building a deep connection with previous distillation methods. The difference is that we provide a more fundamental explanation of the meaning and learning objective of parameterizations.Phased consistency distillation objective. Denote the pre-trained diffusion models as s ϕ (x, t) = -ϵ ϕ (x,t)σt , which induces an empirical PF-ODE. We firstly discretize the whole trajectory into N sub-intervals with N + 1 discrete timesteps from [ϵ, T ], which we denote ast 0 = ϵ < t 1 < t 2 < • • • < t N = T .Typically N should be sufficiently large to make sure the ODE solver approximates the ODE trajectory correctly. Then we sample M + 1 timesteps as edge timestepss 0 = t 0 < s 1 < s 2 < • • • < s M = t N ∈ {t i } N i=0 to split the ODE trajectory into M sub-trajectories. Each sub-trajectory [s i , s i+1 ] consists of the set of sub-intervals {[t j , t j+1 ]} tj ≥si,tj+1≤si+1 .Here we define Φ(x t n+k , t n+k , t n ; ϕ) as the k-step ODE solver that approximate x ϕ tn from x t n+k on the same sub-trajectory following Equation 2, namely,EQUATIONFollowing CMs [56] , we set k = 1 to minimize the ODE solver cost. The training loss is defined asEQUATIONwhere P(m) := uniform({0, 1, . . . , M -1}), P(n|m) := uniform({n + 1|t n+1 ≤ s m+1 , t n ≥ s m }), P(x tn+1 |n, m) = P tn+1 (x), and θ -= µθ -+ (1 -µ)θ. We show that when L PCM (θ, θ -; ϕ) = 0 and local errors of ODE solvers uniformly bounded by O((∆t) p+1 ), the solution estimation error within the arbitrary sub-trajectory f m θ (x tn , t n ) is bounded by O((∆t) p ) in Theorem 1. Additionally, the solution estimation error across any sets of subtrajectories (i.e., the error between f m,m ′ θ (x tn , t n ) and f m,m ′ (x tn+1,tn+1 ; ϕ)) is also bounded by O((∆t) p ) in Theorem 2.Sampling. For a given initial sample at timestep t which belongs to the sub-trajectory [s m , s m+1 ], we can support deterministic sampling following the definition of f m,0 . Previous work [21, 67] reveals that introducing a certain degree of stochasticity might lead to better generation quality. We show that our sampling method can also introduce randomness through a simple modification, which we discuss at Sec. IV.1For convenience, we take the epsilon-prediction format with text conditions c for the following discussion. The consistency model is denoted as ϵ θ (x t , t, c), and the diffusion model is denoted as ϵ ϕ (x t , t, c). A commonly applied strategy for text-conditioned diffusion models is classifier-free guidance (CFG) [16, 51, 1] . At training, c is randomly substituted with null text embedding ∅. At each inference step, the model computes ϵ ϕ (x t , t, c) and ϵ ϕ (x t , t, c neg ) simultaneously, and the actual prediction is the linear combination of them. Namely,EQUATIONwhere w controlling the strength and c neg can be set to ∅ or text embeddings of unwanted characteristics. A noticeable phenomenon is that diffusion models with this strategy can not generate content with good quality without using CFG. That is, the empirical ODE trajectory induced with pure ϵ ϕ (x t , t, c) deviates away from the ODE trajectory to real data distribution. Thereby, it is necessary to apply CFG-augmented prediction ϵ ϕ (x t , t, c, c neg ; w) for ODE solvers. Recall that we have shown that the consistency learning target of the consistency model ϵ θ (x t , t, c) is the weighted sum of epsilon prediction on the trajectory. Thereby, we have ϵ θ (x t , t, c) ∝ ϵ ϕ (x t ′ , t ′ , c, ∅; w), for all t ′ ≤ t. On this basis, if we additionally apply the CFG for the consistency models, we can prove thatEQUATIONwhereϵ merge ϕ = (1 -α)ϵ ϕ (x t ′ , t ′ , c neg ) + αϵ ϕ (x t ′ , t ′ , ∅) and α = (w-1)ww ′ (See Theorem 4). This equation indicates that applying CFG w ′ to the consistency models trained with CFG-augmented ODE solver confined with w, is equivalent to scaling the prediction of original diffusion models by w ′ w, which explains the the exposure problem. We can also observe that the epsilon prediction with negative prompts is diluted by the prediction with null text embedding, which reveals that the impact of negative prompts is reduced. We ask the question: Is it possible to conduct consistency distillation with diffusion models trained for CFG usage without applying CFG-augmented ODE solver? Our finding is that it is not applicable for original CMs but works well for PCMs especially when the number of sub-trajectory M is large. We empirically find that M = 4 is sufficient for successful training. As we discussed, the text-conditioned diffusion models trained for CFG usage fail at achieving good generation quality when removing CFG for inference. That is, target data distribution induced by PF-ODE with ϵ ϕ (x t , t, c) has a large distribution distance to real data distribution. Therefore, fitting the spoiled ODE trajectory is only to make the generation quality bad. In contrast, when phasing the whole ODE trajectory into several sub-trajectories, the negative influence is greatly alleviated. On one hand, the starting point x sm+1 of sub-trajectories is replaced by adding noise to the real data. On the other hand, the distribution distance between the distribution of solution points x sm of sub-trajectories and real data distribution P data sm at the same timestep is much smaller proportional to the noise level introduced. To put it straightforwardly, even though the distribution gap between the real data and the samples generated from ϵ ϕ (x t , t, c) is large, adding noise to them reduce the gap. Thereby, we can optionally train a consistency model whether supporting larger values of CFG or not.We introduce an adversarial loss to enforce the distribution consistency, which greatly improves the generation quality in few-step settings. For convenience, we introduce an additional symbol T t→s which represents a flow from P t to P s . Let T ϕ t→s , T θ t→s be the transition mapping following the ODE trajectory of pre-trained diffusion and our consistency models. Additionally, let T - s→t be the distribution transition following the forward process SDE (adding noise). The loss function is defined as the followingEQUATION)where # is the pushforward operator, and D is the distribution distance metric. To penalize the distribution distance, we apply the GAN-style training paradigm. To be specific, as shown in Fig. 4 , for the sampled x t n+k and the x ϕ tn solved through the pre-trained diffusion model ϕ, we first compute their predicted solution point xsm = f m θ (x t n+k , t n+k ) and xsm = f θ -(x ϕ tn , t n ). Then we randomly add noise to xsm and xsm to obtain xs and xs with randomly sampled s ∈ [s m , s m+1 ]. We optimize the adversarial loss between xs and xs . Specifically, the L adv PCM can be re-written asEQUATIONwhere ReLU(x) = x if x > 0 else ReLU(x) = 0, D is the discriminator, c is the image conditions (e.g., text prompts), and the loss is updated in a min-max manner [12] . Therefore the eventual optimization objective is L PCM + λL adv PCM with λ as a hyper-parameter controlling the trade-off of distribution consistency and instance consistency. We adopt λ = 0.1 for all training settings. However, we hope to clarify that the adversarial loss has an intrinsic difference from the GAN. The GAN training aims to align the training data distribution and the generation distribution of the model. That is, D(T θ t n+k →sm #P t n+k ∥P sm ) In Theorem 5, we show that consistency property is enforced, our introduced adversarial loss will also coverage to zero. Yet in Theorem 6, we show that combining standard GAN with consistency distillation is a flawed design when considering the pre-trained data distribution and distillation data distribution mismatch. Its loss will be non-zero when the self-consistency property is achieved, thus corrupting the consistency distillation learning. Our experimental results also verify our statement (Fig. 7 ). For image generation, we train all models on the CC3M [5] dataset. For video generation, we train the model on WebVid-2M [2] . Evaluation dataset: For image generation, we evaluate the performance on the COCO-2014 [28] following the 30K split of karpathy. We also evaluate the performance on the CC12M with our randomly chosen 30K split. For video generation, we evaluate with the captions of UCF-101 [58] .Backbones. We verify the text-to-image generation based on Stable Diffusion v1-5 [45] and Stable Diffusion XL [41] . We verify the text-to-video generation following the design of AnimateLCM [63] with decoupled consistency distillation.Evaluation metrics. Image: We report the FID [14] and CLIP score [43] of the generated images and the validation 30K-sample splits. Following [8, 47] , we also compute the FID with CLIP features (FID-CLIP). Note that, all baselines and our method focus on distilling the knowledge from the pre-trained diffusion models for acceleration. Therefore, we also compute the FID of all baselines and the generated images of original pre-trained diffusion models including Stable Diffusion v1-5 and Stable Diffusion XL (FID-SD). Video: For video generation, we evaluate the performance from three perspectives: the CLIP Score to measure the text-video alignment, the CLIP Consistency to measure the inter-frame consistency of the generated videos, the Flow Magnitude to measure the motion magnitude of the generated videos with Raft-Large [59] . 
comparison: 
 Comparison methods. We compare PCM with Stable Diffusion v1-5 to methods including Stable Diffusion v1-5 [45] , InstaFlow [31] , LCM [35] , CTM [21] , TCD [74] and SD-Turbo [49] . We compare PCM with Stable Diffusion XL to methods including Stable Diffusion XL [41] , CTM [21] , SDXL-Lightning [27] , SDXL-Turbo [49] , and LCM [35] . We apply the 'Ours' and 'Ours*' to denote our methods trained with CFG-augmented ODE solver or not. We only report the performance of 'Ours*' with more than 4 steps which aligns with our claim that it is only possible when phasing the ODE trajectory into multiple sub-trajectories. For video generation, we compare with DDIM [54] , DPM [33] , and AnimateLCM [63] .Qualitative comparison. We evaluate our model and comparison methods with a diverse set of prompts in different inference steps. The results are listed in Fig. Human evaluation metrics. To more comprehensively reflect the performance of phased consistency models, we conduct a thorough evaluation using human aesthetic preference metrics, encompassing 1-16 steps. This assessment employs well-regarded metrics, including HPSv2 (HPS) [66] , PickScore (PICKSCORE) [23] , and Laion Aesthetic Score (AES) [50] , to benchmark our method against all comparative baselines. As shown in Table 6 and Table 7 , across all evaluated settings, our method consistently achieves either superior or comparable results, with a marked performance advantage over the consistency model baseline LCM, demonstrating its robustness and appeal across diverse human-centric evaluation criteria. We conduct a human preference ablation study on the proposed adversarial consistency loss, with the results presented in Table 8 . The inclusion of adversarial consistency loss consistently enhances human evaluation metrics across different inference steps. Sensitivity to negative prompt. To show the comparison of sensitivity to negative prompt between our model tuned without CFG-augmented ODE solver and LCM. We provide an example of a prompt and negative prompt to GPT-4o and ask it to generate 100 pairs of prompts and their corresponding negative prompts. For each prompt, we generate 10 images. We first generate images without using the negative prompt to show the positive prompt alignment comparison. Then we generate images with positive and negative prompts. We compute the CLIP score of generated images and the prompts and negative prompts. Fig. 6 shows that we not only achieve better prompt alignment but are much more sensitive to negative prompts.Consistent generation ability. Consistent generation ability under different inference steps is valuable in practice for multistep refinement. We compute the average CLIP similarity between the 1-step generation and the 16-step generation for each method. As shown in the rightmost column of Table 2 and Table 5 , our method achieves significantly better consistent generation ability.Adversarial consistency design and its effectiveness. We show the ablation study on the adversarial consistency loss design and its effectiveness. From the architecture level of discriminator, we compare the latent discriminator shared from the teacher diffusion model and the pixel discriminator from pre-trained DINO [4] . Note that DINO is trained with 224 resolutions, therefore we should resize the generation results and feed them into DINO. We find this could make the generation results fail at details as shown in the left of Fig. 9 . From the adversarial loss, we compare our adversarial loss to the normal GAN loss. We find normal GAN loss causes the training to be unstable and corrupts the generation results, which aligns with our previous analysis. For its effectiveness, we compare the FID-CLIP and FID scores with the adversarial loss or without the adversarial loss under different inference steps. Fig. 7 shows that it greatly improves the FID scores in the low-step regime and gradually coverage to similar performance of our model without using the adversarial loss as the step increases.Randomness for sampling. Fig. 8 illustrates the influence of the randomness introduced in sampling as Eq. 44. The figure shows that introducing a certain of randomness in sampling may help to alleviate unrealistic objects or shapes. 
limitations and conclusions: 
 Despite being able to generate high-quality images and videos in a few steps, we find that when the number of steps is very low, especially with only one step, the generation quality is unstable. The model may produce structural errors or blurry images. Fortunately, we discover that this phenomenon can be mitigated through multi-step refinement. In conclusion, in this paper, we observe the defects in latent consistency models. We summarize these defects on three levels, analyze their causes, and generalize the design framework to address these defects.The latter are mainly distillation methods [38, 46, 30, 65, 70, 10] or initializing diffusion weights for GAN training [49, 27, 39, 19] .Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The positive societal impacts are discussed in Sec. V.1, and the negative societal impacts are discussed in Sec. V.2.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: Yes, we are the image generators, which naturally have a risk of generating harmful content. Therefore, in Sec. V.3, we introduce our solution to avoid harmful content. We will include the safeguards in our pipeline when releasing the code and model.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All the assets used in this paper are public datasets, allowing for research use. Specifically, the used datasets include:• CC3M -custom terms reading: The dataset may be freely used for any purpose. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The consistency model is a new family of generative models [11, 56, 55, 62, 26, 32] supporting fast and high quality generation. It can be trained either by distillation or direct training without teacher models. Improved techniques even allow consistency training excelling the performance of diffusion training [55] . Consistency trajectory model [21] proposes to learn the trajectory consistency, providing a more flexible framework. Some works combine [24] consistency training with GAN [24] for better training efficiency. Some works adopt continuous consistency models and achieve excelling performance [11, 62, 32] . Some works apply the idea of consistency model to language model [25] and policy learning [42, 34] . Some works extends the application scope of consistency models for text-conditioned image generation [35, 44, 74 ] and text-conditioned video generation [63, 37, 71, 60] .We notice that a recent work multistep consistency models [13] also proposes to splitting the ODE trajectory into multi-parts for consistency learning, and here we hope to credit their work for the valuable exploration. However, our work is principally different from multistep consistency models. Since they do not open-source their code and weights, we clarify the difference between our work and multistep consistency models based on the details of their technical report. Firstly, they didn't provide explicit model definitions and boundary conditions nor theoretically show the error bound to prove the soundness of their work. Yet in our work, we believe we have given an explicit definition of important components and theoretically shown the soundness of important techniques applied. For example, they claimed using DDIM for training and inference, while in our work, we have shown that although we can also optionally parameterize as the DDIM format, there's an intrinsic difference between that parameterization and DDIM (i.e., the difference between exact solution learning and first-order approximation). The aDDIM and invDDIM as highlighted in their pseudo code have no relation to PCM. Secondly, they are mainly for the unconditional or class-conditional generation, yet we aim for text-conditional generation in large models and dive into the influence of classifier-free guidance in consistency distillation. Besides, we introduce an adversarial loss that aligns well with consistency learning and improves the generation results in few-step settings. We recommend readers to read their work for better comparison.The following is an extension of the original proof of consistency distillation for phased consistency distillation. ((t n+1 -t n ) p+1 ) with p ≥ 1. Then, if L PCM (θ, θ; ϕ) = 0, we have sup tn,tn+1∈[sm,sm+1],x ∥f m θ (x, t n ) -f m (x, t n )∥ = O((∆t m ) p ).Proof. From the condition L PCM (θ, θ; ϕ) = 0, for any x tn and t n , t n+1 ∈ [s m , s m+1 ], we haveEQUATIONConsidering that f m θ is L-Lipschitz, we haveEQUATIONBesides, due to the boundary condition, we haveEQUATION)Hence, we haveEQUATIONTheorem 2. For arbitrary set of sub-trajectories {[s i , s i+1 ]} m ′ i=m . Let ∆t m := max tn,tn+1∈[sm,sm+1] {|t n+1 -t n |}, ∆t m,m ′ := max i∈[m ′ ,m] {∆t i },EQUATIONHence, we haveEQUATIONWhere the last equation is due to the boundary conditionf m,m ′ θ (x sm , s m ) = f m-1,m ′ θ (f m θ (x sm , s m ), s m ) = f m-1,m ′ θ (x sm , s m ).Thereby, we haveEQUATIONWe show that the parameterization of Equation 3 is equal to the DDIM inference format [33, 54] .Theorem 3. Define F θ (x t , t, s) = αs αt x t -α s εθ (x t , t)λs λt e -λ dλ, then the parameterization has the same format of DDIM x s = α s xt-σt εθ (xt,t) αt + σ s εθ (x t , t).Proof.EQUATIONWe show the relationship of epsilon prediction of consistency models trained with guided distillation and diffusion models when considering the classifier-free guidance.Theorem 4. Assume the consistency model ϵ θ is trained by consistency distillation with the teacher diffusion model ϵ ϕ , and the ODE solver is augmented with CFG value w and null text embedding ∅. Let c and c neg be the prompt and negative prompt applied for the inference of consistency model. Then, if the ODE solver is perfect and the L PCM (θ, θ) = 0, we haveϵ θ (x, t, c, c neg ; w ′ ) ∝ ww ′ ϵ ϕ (x, t, c) -((1 - w -1 ww ′ )ϵ ϕ (x, t, c neg ) + w -1 ww ′ ϵ ϕ (x, t, ∅)) + ϵ ϕ (x, t, c neg )Proof. If the ODE solver is perfect, that means the empirical PF-ODE is exactly the PF-ODE of the training data. Then considering Theorem 1 and Theorem 2, it is apparent that the consistency model will fit the PF-ODE. To show that, considering the case in Theorem 1, we haveEQUATIONwhere (i) is because the ODE solver is perfect. Considering our parameterization in Equation 3, then it should be equal to the exact solution in Equation 2. That is, α s α t x t -α s εθ (x t ,Then we have the epsilon prediction is weighted integral of diffusion-based epsilon prediction on the trajectory. Therefore, it is apparent that, for any t ′ ≤ t and t ′ on the same sub-trajectory with t, we have εθ (x t , t) ∝ ϵ ϕ (x t ′ , t ′ ) (23) When considering the text-conditioned generation and the ODE solver being augmented with the CFG value w, we haveEQUATION) If we additionally apply the classifier-free guidance to the consistency models with negative prompt embedding c neg and CFG value w ′ , we haveEQUATIONWe firstly show that when L PCM = 0 is achieved, our distribution consistency loss will also converge to zero. Then, we additionally show that, when considering the pre-train data distribution and distillation data distribution mismatch, combining GAN is a flawed design. The loss is still non-zero even when the self-consistency is achieved, thus corrupting the training.Theorem 5. Denote the data distribution applied for consistency distillation phased is P 0 . And considering that the forward conditional probability path is defined by α t x 0 + σ t ϵ, we further define the intermediate distribution P t (x) = (P 0 ( x αt ) • 1 αt ) * N (0, σ t ). Similarly, we denote the data distribution applied for pretraining the diffusion model is P pretrain 0 (x) and the intermediate distribution following forward process are P pretrain t (x) = (P pretrain t (x) = (P pretrain 0( x αt ) • 1 αt ) * N (0, σ t )). This is reasonable since current large diffusion models are typically trained with much more resources on much larger datasets compared to those of consistency distillation. And, we denote the flow T ϕ t→s , T θ t→s , and T ϕ ′ t→s correspond to our consistency model, pre-trained diffusion model, and the PF-ODE of the data distribution used for consistency distillation, respectively. Additionally, let T - s→t be the distribution transition following the forward process SDE (adding noise). Then, if the L PCM = 0, for arbitrary sub-trajectory [s m , s m+1 ], we have,L adv PCM (θ, θ; ϕ, m) = D T - sm→s T θ tn+1→sm #P tn+1 T - sm→s T θ tn→sm T ϕ tn+1→tn #P tn+1 = 0.Proof. Firstly, considering that the forward process T - s→t is equivalent to scaling the original variables and then performing convolution operations with N (0, σ 2 s→t I). Therefore, as long asEQUATIONthen we haveEQUATION)From the condition L PCM (θ, θ; ϕ) = 0, for any x tn ∈ P tn and x tn+1 ∈ P tn+1 and t n , t n+1 ∈ [s m , s m+1 ], we haveEQUATIONwhich induces thatEQUATIONTherefore, we show that if L P CM (θ, θ; ϕ) = 0, thenEQUATION)Theorem 6. Denote the data distribution applied for consistency distillation phased is P 0 . And considering that the forward conditional probability path is defined by α t x 0 + σ t ϵ, we further define the intermediate distribution P t (x) = (P 0 ( x αt ) • 1 αt ) * N (0, σ t ). Similarly, we denote the data distribution applied for pretraining the diffusion model is P pretrain 0 (x) and the intermediate distribution following forward process are P pretrain t (x) = (P pretrain t (x) = (P pretrain 0( x αt ) • 1 αt ) * N (0, σ t )). This is reasonable since current large diffusion models are typically trained with much more resources on much larger datasets compared to those of consistency distillation. And, we denote the flow T ϕ t→s , T θ t→s , and T ϕ ′ t→s correspond to our consistency model, pre-trained diffusion model, and the PF-ODE of the data distribution used for consistency distillation, respectively. Then, if the L PCM = 0, for arbitrary sub-trajectory [s m , s m+1 ], we have,L adv PCM (θ, θ; ϕ, m) = D T θ tn+1→sm #P tn+1 P sm ≥ 0.Proof. From the condition L PCM (θ, θ; ϕ) = 0, for any x tn ∈ P tn and x tn+1 ∈ P tn+1 and t n , t n+1 ∈ [s m , s m+1 ], we haveEQUATIONwhich induces thatEQUATIONBesides, since T ϕ ′ t→s corresponds to the PF-ODE of the data distribution used for consistency distillation, we can rewriteEQUATIONTherefore, we haveEQUATION)Since we know that P pretrain 0 ̸ = P 0 . Therefore, there exists m and n achieving the strict inequality. Specifically, if we set s m = ϵ and t n+1 = T , we haveEQUATION)Even though our above-discussed parameterization is theoretically sound, it poses a numerical issue when applied to the epsilon-prediction-based models, especially for the one-step generation. To be specific, for the one-step generation, we are required to predict the solution point x 0 with x t from the self-consistency property of consistency models. With epsilon-prediction models, the formula can represented as the followingEQUATION)However, when inference, we should choose t = T since it is the noisy timestep that is closest to the normal distribution. For the DDPM framework, there should be σ T ≈ 1 and α T ≈ 0 to make the noisy timestep T as close to the normal distribution as possible. For instance, the noise schedulers applied by Stable Diffusion v1-5 and Stable Diffusion XL all have the α T = 0.068265. Therefore, we haveEQUATION)This indicates that we will multiply over 10× to the model prediction. In our experiments, we find the SDXL is more influenced by this issue, tending to produce artifact points or lines.Generally speaking, using the x0-prediction and v-prediction can well solve these issues. It is obvious for the x0-prediction. For the v-prediction, we haveEQUATION)However, since the diffusion models are trained with the epsilon-prediction, transforming them into the other prediction formats takes additional computation resources and might harm the performance due to the relatively large gaps among different prediction formats.We solve this issue through a simple way that we set a clip boundary to the value of α t . Specifically, we applyEQUATION)Consistency trajectory model (CTM), as we discussed, proposes to learn how 'move' from arbitrary points on the ODE trajectory to arbitrary other points. Thereby, the model should be aware of 'where they are' and 'where they target' for precise prediction.Basically, they can still follow our parameterization with an additional target timestep embedding.EQUATIONIn this design, we haveEQUATION)Here, we show that dropping the target timestep embedding s is a flawed design.Assume we hope to predict the results at timestep s ′ and timestep s ′′ from timestep t and sample x t , where s ′ < s ′′ .If we hope to learn both x s ′ and x s ′′ correctly, without the target timestep embedding, we must haveEQUATION)This will lead to the conclusion that all the segments [t, s] on the ODE trajectory have the same weighted epsilon prediction, which violates the truth, ignoring the dynamic variations and dependencies specific to each segments.Here we re-emphasize the key components of PCM and summarize the contributions of our work.The motivation of our work is to accelerate the sampling of high-resolution text-to-image and textto-video generation with consistency models training paradigm. Previous work, latent consistency model (LCM), tried to replicate the power of the consistency model in this challenging setting but did not achieve satisfactory results. We observe and analyze the limitations of LCM from three perspectives and propose PCM, generalizing the design space and tackling all these limitations.At the heart of PCM is to phase the whole ODE trajectory into multiple phases. Each phase corresponding to a sub-trajectory is treated as an independent consistency model learning objective. We provide a standard definition of PCM and show that the optimal error bounds between the prediction of trained PCM and PF-ODE are upper-bounded by O((∆t) p ). The phasing technique allows for deterministic multi-step sampling, ensuring consistent generation results under different inference steps.Additionally, we provide a deep analysis of the parameterization when converting pre-trained diffusion models into consistency models. From the exact solution format of PF-ODE, we propose a simple yet effective parameterization and show that it has the same formula as first-order ODE solver DDIM. However, we show that the parameterization has a natural difference from the DDIM ODE solver. That is the difference between exact solution learning and score estimation. Besides, we point out that even though the parameterization is theoretically sound, it poses numerical issues when building the one-step generator. We propose a simple yet effective threshold clip strategy for the parameterization.We introduce an innovative adversarial loss, which greatly boosts the generation quality at a low-step regime. We implement the loss in a GAN style. However, we show that our method has an intrinsic difference from traditional GAN. We show that our introduced adversarial loss will converge to zero when the self-consistency property is achieved. However, the GAN loss will still be non-zero when considering the distribution distance between the pre-trained dataset for diffusion training and the distillation dataset for consistency distillation. We investigate the structures of discriminators and compare the latent-based discriminator and pixel-based discriminator. Our finding is that applying the latent-based visual backbone of pre-trained diffusion U-Net makes the discriminator design simple and can produce better visual results compared to pixel-based discriminators. We also implement a similar discriminator structure for the text-to-video generation with a temporal inflated U-Net.For the setting of text-to-image generation and text-to-video generation, classifier-free guidance (CFG) has become an important technique for achieving better controllability and generation quality. We investigate the influence of CFG on consistency distillation. And, to our knowledge, we, for the first time, point out the relations of consistency model prediction and diffusion model prediction when considering the CFG-augmented ODE solver (guided distillation). We show that this technique causes the trained consistency models unable to use large CFG values and be less sensitive to the negative prompt.We achieve state-of-the-art few-step text-to-image generation and text-to-video generation with only 8 A 800 GPUs, indicating the advancements of our method.For a given initial sample at timestep t belonging to the sub-trajectory [s m , s m+1 ], we can support deterministic sampling according to the definition of f m,0 .However, previous work [21, 67] reveals that introducing a certain degree of stochasticity can lead to better generation results. We also observe a similar trade-off phenomenon in our practice. Therefore, we reparameterize the F θ (x, t, s) asEQUATIONBy controlling the value of r ∈ [0, 1], we can determine the stochasticity for generation. With r = 1, it is pure deterministic sampling. With r = 0, it degrades to pure stochastic sampling. Note that, introducing the random ϵ will make the generation results be away from the target induced by the diffusion models. However, since ϵ θ and ϵ all follows the normal distribution, we have √ rϵ θ (x t , t) + (1 -r)ϵ ∼ N (0, I) and thereby the predicted x s should still follow the same distribution P s approximately.Our another observation is the generation diversity with guided distillation is limited compared to original diffusion models. This is due to that the consistency models are distilled with a relatively large CFG value for guided distillation. The large CFG value, though known for enhancing the generation quality and text-image alignment, will degrade the generation diversity. We explore a simple yet effective strategy by adjusting the CFG values. w ′ ϵ θ (x, t, c) + (1 -w ′ )ϵ θ (x, t, ∅), where w ′ ∈ (0.5, 1]. The epsilon prediction is the convex combination of the conditional prediction and the unconditional prediction.We firmly believe that our work has a profound positive impact on society. While diffusion techniques excel in producing high-quality images and videos, their iterative inference process incurs significant computational and power costs. Our approach accelerates the inference of general diffusion models by up to 20 times or more, thus substantially reducing computational and power consumption. Moreover, it fosters enthusiasm among creators engaged in AI-generated content creation and lowers entry barriers.Addressing potential negative consequences, given the generative nature of the model, there's a risk of generating false or harmful content.To mitigate this, we conduct harmful information detection on user-provided text inputs. If harmful prompts are detected, the generation process is halted. Additionally, our method, fortunately, builds upon the open-source Stable Diffusion model, which includes an internal safety checker for detecting harmful content. This feature significantly enhances our ability to prevent the generation of harmful content.For the comparison of baselines, the training code for InstaFlow [31] , SDXL-Turbo [49] , SD-Turbo [49] , TCD [74] , and SDXL-Lightning [27] are not open-sourced yet, so we only compare their open-source weights.For LCM [35] , CTM [21] , and our method, they are all implemented by us and trained with the same configuration.Specifically, for multi-step models, we trained LoRA with a rank of 64. For models based on SD v1-5, we used a learning rate of 5e-6, a batch size of 160, and trained for 5k iterations. For models based on SDXL, we used a learning rate of 5e-6, a batch size of 80, and trained for 10k iterations. We did not use EMA for LoRA training.For single-step models, we followed the approach of SD-Turbo and SDXL-Lightning, training all parameters. For models based on SD v1-5, we used a learning rate of 5e-6, a batch size of 160, and trained for 10k iterations. For models based on SDXL, we used a learning rate of 1e-6, a batch size of 16, and trained for 50k iterations. We used EMA=0.99 for training. + xs) + ReLU(1 -xs)) θ ← θ -η∇ θ L(θ, θ -) θ -← stopgrad(µθ -+ (1 -µ)θ) until convergencethe latent code to a very high-dimensional image through the VAE decoder, which is much more costly compared to traditional diffusion training. Secondly, it can only accept the inputs of clean images, which poses challenges to PCM training, since we require the discrimination of inputs of intermediate noisy inputs. Thirdly, it requires resizing the images into the relatively low resolution it trained with. When applying this backbone design with high-dimensional image generation with Stable-Diffusion XL, we observe unsatisfactory results.We apply the U-Net of the pre-trained diffusion model as the latent-based backbone. It has several advantages compared to the pixel-based backbones. Firstly, trained on the latent space, the U-Net possesses rich knowledge about latent space and can directly work as the discriminator at the latent space, avoiding costly decoding processes. Besides, it can accept the inputs of latent code at different timesteps, which aligns well with the design of PCM and can be applied for regularizing the consistency of all the intermediate distributions instead (i.e., P t (x)) of distribution at the edge points (i.e., P sm (x)). Additionally, since the text information is already encoded in the U-Net backbone, we do not need to incorporate the text information in the discriminator head, which simplifies the discriminator design. We insert several randomly initialized lightweight simple discriminator heads after each block of the pre-trained U-Net. Each discriminator head consists of two lightweight convolution blocks connected with residuals. Each convolution block is composed of a convolution 2D layer, Group Normalization, and GeLU non-linear activation function. Then we apply a 2D point-wise convolution layer and set the output channel to 1, thus mapping the input latent feature to a 2D scalar value output map with the same spatial size.We train our one-step text-to-image model on Stable Diffusion XL using these two choices of discriminator respectively, while keeping the other settings unchanged. Our experiments reveal that the latent-based discriminator not only reduces the training cost but also provides more visually compelling generation results. Additionally, note that the teacher diffusion model applied for phased consistency distillation is actually a pre-trained U-Net. And we freeze the parameters of U-Net for the training discriminator. Therefore, we can apply the same U-Net, which simultaneously works as the teacher diffusion model for numerical ODE solver computation and discriminator visual backbone.For the video discriminator, we mainly follow our design on the image discriminator. We use the same U-Net with temporal inflation as the visual backbone for video latent code as well as the teacher video diffusion model for phased consistency distillation. We still apply the 2D convolution layers as discriminator heads for each frame of hidden features extracted from the temporal inflated U-Net since we do not observe performance gain when using 3D convolutions. Note that the temporal relationships are already processed by the pre-trained temporal inflated U-Net, therefore using simple 2D convolution layers as discriminator heads is enough for supervision of the distribution of video inputs.VII More Generation Results.Prompt: "a happy white man in black suit, sky, red tie, river, mountain, colourful, clouds, best view, fall"  


Decision: One reviewer recommended rejection and also flagged a potential plagiarism concern. Upon further investigation the AC did not find evidence of plagiarism.

Amongst the other reviewers, one recommended strong accept and the other recommended weak accept. As mentioned by these reviewers, the paper introduces a novel method of generating multiple self consistent sub-trajectories and achieves strong distillation performance on image generation while also showing applicability on video generation tasks. The AC agrees with these evaluations and recommends acceptance.

Reviews: review: paper_summary: paper_summary: The paper investigates three issues of Consistency Models on Latent space, thereby making a proposal named Phased Consistency Model that handles these weaknesses, supported by theoretical proofs and derivations. To assess the efficacy of their proposed solution, the authors conduct extensive experiments on two main scenarios: text-to-image and text-to-video generation. Additionally, details ablation studies and analysis on negative prompting, the self-consistency property and adversarial design to provide a strong motivation for the proposed approach.
strength_weakness: strength_weakness: 1. Pre-defined number of sub-trajectories seems to fix the number of inference steps, which reduces the model’s flexibility.
2. Lack of explanation for the design of formula of Adversarial Consistency Loss, which employs ReLU function.
3. Need an ablation study to investigate impact of Adversarial Consistency Loss, for example what is model’s behaviour and result without this loss?
4. Detailed hyper-parameters used in settings, for example preconditioning weight, Adversarial Consistency Loss weight.
questions: questions: See the weakness above
limitations: limitations: The authors analyse the limitations of their method in case of 1-step generation in a few lines.
paper_summary: paper_summary: The paper titled "Phased Consistency Model" (PCM) introduces a novel model designed to address the limitations of Latent Consistency Models (LCMs) in high-resolution, text-conditioned image generation. The authors identify three primary flaws in LCMs: inconsistency, controllability, and efficiency. PCM is proposed to generalize the design space of LCMs and overcome these limitations by segmenting the ODE trajectory into multiple sub-trajectories and enforcing self-consistency within each sub-trajectory. This approach allows for more stable and deterministic sampling, better control over image generation with larger CFG values, and improved quality in low-step settings through an adversarial loss in the latent space. Extensive experiments demonstrate PCM’s superior performance in both image and video generation tasks compared to existing methods.
strength_weakness: strength_weakness: No specific weakness to mention.
questions: questions: - Could the author provide some other human related evaluation metrics, e.g., ImageReward, Aesthetic scores, etc, to compare the generation quality within different efficient diffusion models?
- Given the introduction of an adversarial loss, how stable is the training process of PCM? Are there any observed issues with training stability or convergence, and how are they addressed?
limitations: limitations: The authors have demonstrated the limitations, e.g., inability in 1-step generation, in their paper.
paper_summary: paper_summary: The paper introduces the Phased Consistency Model (PCM), which enhances stability and speed in high-resolution image and video generation by improving the design of consistency (trajectory) models. 
The main improvement of PCM is a new Parameterization of consistency function and the phase trajectory.  There is also a new method to improve the utilization of CFG. 
Experimental results show that PCM performs better across 1 to 16-step generation settings.
strength_weakness: strength_weakness: Despite the strong experimental results, I still have many reservations about this work.
The primary concern stems from the inaccurate description and the incomplete experiments.

Here's my MAJOR concerns:

1. The author also asserts that the findings are derived based on CTM. However, the author does not provide any comparisons with the official CTM, e.g., ImageNet or CIFAR. 

2. Another concern is that the author claims to have compared the results with SD15 CTM and SDXL CTM. However, I could not find any related descriptions or implementations in the official CTM publications or GitHub repository.  The authors claimed such models are implemented by theirs but there's no detail. Specifically, the author stated they *additionally learned a timestep embedding to indicate the target timestep* but without any details and reasons.

3. A further question about implement detail: 1-step PCM and multi-step PCM are trained separately? If so, only the range of training parameters different? Are there any further differences?

4. The author appears to misunderstand the concept of the negative prompt, e.g. Controllability in the LCM drawbacks. Furthermore, the provided results do not demonstrate that PCM can mitigate the impact of the negative prompt. 

5. The author does not clearly explain the differences between CTM and PCM, especially without a clear description of implementation of CTM in SD15 and SDXL. 
Based on CTM, the novelty of this work on my side is 4-fold: a re-parameterization method, an improved GAN structure, an edged timestep and a CFG guidance method. 


In conclusion,
the main problem is the authors **did not** conduct an objective comparison and the work lacks sufficient quantitative ablation analysis with official CTM on ImageNet or CIFAR. From the existing data, it is difficult for me to determine whether the proposed solutions by the authors are truly effective.


Here's some minor questions:

1. The author relied heavily on FID and CLIP metrics for comparison, but these have known limitations, particularly in large-scale text-to-image applications. More contemporary metrics, such as HPS or PICK-SCORE, should be used to demonstrate the effectiveness.

2. Additionally, I suggest author to use recall to demonstrate the diversity of the model. The introduction of additional GAN modules can lead to a lack of diversity in the model.
questions: questions: please see Weaknesses
limitations: limitations: The author has already discussed limitations.

Comments: title: Sincerely thank you for raising the score!
Regarding  the accusation of Reviewer ukQK
Sincerely thank you for the very positive review!
Thank you for your further response!
A suspicion of potential plagiarism.
We are willing for further discussion! We validate the design of PCM on much more baselines than all compared papers!
We are willing for further discussion! CTM metrics needs to generate much more samples than compared methods, which is unfair to compare directly.
We are willing for further discussion!
Thank you for your response!
comment: We are grateful for your decision to raise the score. Your recognition and feedback mean a lot to us.

Please feel free to reach out if there’s anything else we can assist with.

Best regards,

The Authors
Thank you for addressing my concern. I decide to raise score to 6.
**Parameterization**

Our final parameterization is finally parameterized in the same format as DDIM for simplicity (as stated in the paper).

For distillation-based acceleration, using the format of DDIM for distillation is a common practice. We list works using the format for large-scale text-to-image distillation in the following:

1. Progressive Distillation for Fast Sampling of Diffusion Models. ICLR 2022
2. On Distillation of Guided Diffusion Models. CVPR 2023
3. BOOT : Data-free Distillation of Denoising Diffusion Models with Bootstrapping. arXiv 2023

We list the works using the format of DDIM for distillation in unconditional and class-conditional  in the following

1. TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillation. arXiv 2023
2. BOOT : Data-free Distillation of Denoising Diffusion Models with Bootstrapping. arXiv 2023

Therefore, we believe It is not TCD (arXiv 2024) that first applied the parameterization for either large-scale text-to-image distillation or unconditional image generation and class-conditional generation.  And we would say it's not strange to potentially have a similar parameterization. The important thing is the discussion in the paper on how to understand and explain the parameterization, which we believe we have an entirely different discussion with TCD (Ln 151 -- 157).

**Boundary condition**

The DPM-Solver has shown that the denoising from $t$ to $s$ is to add an integral of weighted epsilon from $t$ to $s$. So it is obvious that the integral from $s$ to $s$ (i.e., the integral length is 0, $\int_{s}^s f = 0$, where $f$ can be any function) will equal 0. We believe this is an evident and natural conclusion and can be easily observed with minor exposure to calculus.  We admit we did not notice relevant discussion in TCD. As stated in our previous reply, we are willing to add citations to it.

Sincerely,
I disagree with the author's current response.

Given that the author was aware of [1], deliberately using the same reparameterization scheme without proper citation clearly constitutes plagiarism. 

I am **not questioning the innovation or contribution of this work**. I am simply questioning why [1] was not mentioned at all in this manuscript submitted by authors, despite the striking similarity of both submitted manuscript and [1]. This seems to go beyond just a lack of citation.

Also I totally agree about `The important conclusion of DPM-Solver (NeurIPS 2022) shall not be considered as contributions of TCD (arXiv 2024).` But I also do believe TCD as the first work to apply a reparameterization method [2] for large-scale text-to-image distillation does exhibit a degree of innovation and contribution. (If using [2] for distillation doesn't qualify as contribution, then this reparameterization should have been placed in the *related work* section rather than the *method* section of the submitted manuscript.)

Furthermore, the author's explanation is intriguing. `From Line 158 to 161, we do find some similar discussion on the boundary condition in the TCD paper. But please note that it is a natural conclusion from the exact solution formula proposed in DPM-Solver.`  I'm wondering if the author was aware of the TCD paper, does this not suggest the possibility of plagiarism, as I previously mentioned:  Uncited Paraphrasing and Inadequate Paraphrase? 

Lastly, I find it somewhat ironic that the author seems to suggest that if [1] cites [2], and their submitted work also cites [2], then their work can avoid citing [1], even though the author(s) know their work and [1] share substantial similarities in terms of tasks, comparisons, and experiments, while [2] is from an entirely different field compared to [1] and their work.

[1] Trajectory consistency distillation

[2] Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps
3. In the following table, we summarize more differences between TCD and PCM.

| Method | Learning Paradigm      | Adversarial Loss | 1-Step Generation | Video Generation | Sampling Method | CFG Analysis | Consistent Generation | Num  Traing Objects | Prediction  | FID / NFE                               |
| ------ | ---------------------- | ---------------- | ----------------- | ---------------- | --------------- | ------------ | --------------------- | ------------------- | ----------- | --------------------------------------- |
| TCD    | Trajectory Consistency | No               | No                | No               | Gamma-Sampling  | No           | No                    | $\mathcal O(N^2)$   | 1-N mapping | Worse when NFE > 4                      |
| PCM    | Multiphase Consistency | yes              | yes               | yes              | Deterministic   | Yes          | Yes                   | $\mathcal O(N)$     | 1-1 mapping | Continue to be better as the NFE larger |



In addition to the above accusation of plagiarism, we find the reviewer also made judgments on our method that we believed to be unfair.

4. The model of CTM requires a target timestep embedding. But the reviewer asked us the question `Why does CTM need target timestep embeddings`. It is clearly stated in the paper of CTM and also shown in the official code of CTM. Besides, we also set up an independent section to discuss why CTM needs target timestep embeddings (Ln 602 - 615, titled exactly "Why CTM Needs Target Timestep Embeddings?"). The comment might suggest that the reviewer may `have a gap` in understanding the related works and might not have fully grasped the details of our paper.

5. The motivation of our paper is to analyze the limitations of previous LCMs and generalize the design space to solve these limitations (as stated in the abstract, introduction, teaser figure). The LCM is only for high-resolution `text-conditioned` image and video generation. Therefore, we believe we have efficiently validate the effectiveness of PCM.

   For the reviewers' request to run CIFAR and ImageNet which are low-resolution unconditional generation and class-conditional generation tasks and have no direct relations to LCM, we have tried our best to run as many experiments as possible and provide clear evidence of the superiority of PCM compared to CM and CTM in the fair comparison setting. We have validated our methods on  `five baselines` including CIFAR, ImageNet, SD v1-5, SDXL, and Text-to-Video. It is noting that all compared methods only validate their methods for `at most two baselines`, but the reviewer still claims that our method lacks validation.

We believe the contribution of our paper is underestimated. Please kindly take the time to look at our detailed rebuttal.



Thanks, 

Paper 2513 Authors

[1] Consistency Models. ICML 2023
[2] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. NeurIPS 2022
2. Reviewer ukQK thinks we have the same training objective as TCD. Instead, our design is **totally and inherently different** from the training objective of TCD and CTM, which we have explained in our submission and the rebuttals. 

Firstly, from a high-level perspective, we enforce the model to follow self-consistency within the pre-defined sub-phases (i.e. Our ODE is firstly split into multiphases along the time axis, and then we learn CM within the sub-trajectory).  Therefore, our model enforces multiphase consistency. Instead, TCD just  follows CTM and learns to follow trajectory consistency. That is, they enforce their model to learn how to move from arbitrary initial timesteps to arbitrary other timesteps.

 We provide a not strict but very straightforward example to better illustrate the difference. Assume, the model has 1000 timesteps (0, 1, ..., 999) in total and we hope to train a 4-step model of PCM. Then we have

   The learning objectives of PCM:  

   $f(x_{999}, 999) =  x_{749}$

   $f(x_{749}, 749) = x_{499}$

   $f(x_{499}, 499) = x_{249}$ 

   $f(x_{249}, 249) = x_0$ 

   The learning objectives of TCD: 

   $ f(x_{999}, 999, 749) = x_{749} $

   $ f(x_{999}, 999, 499) = x_{499} $ 

   $ f(x_{999}, 999, 249) = x_{249} $ 

   $ f(x_{999}, 999, 0) = x_0 $ 

   $ f(x_{749}, 749, 499) = x_{499} $ 

   $ f(x_{749}, 749, 249) = x_{249} $ 

   $ f(x_{749}, 749, 0) = x_0 $ 

   $ f(x_{499}, 499, 249) = x_{249} $ 

   $ f(x_{499}, 499, 0) = x_0 $ 

   $ f(x_{249}, 249, 0) = x_0 $

 In the above example, PCM only maps one timestep to a predefined corresponding middle timestep (e.g., $999 \rightarrow 749$ ), which is pre-defined. That is, PCM is a one-to-one mapping. But for TCD, it has to accept an additional target timestep to indicate which timestep prediction it hopes to predict, and learn to predict all the other timesteps. That is, TCD is a one-to-N mapping. Therefore, PCM only has $O(N)$  training objectives but TCD has $\mathcal O(N + (N-1) + \cdots + 1) = \mathcal O(N^2)$.  This is also the direct reason why it requires a target timestep embedding. 

We list a more straightforward illustration of the differences of the loss functions,

| Methods | timesteps sampled                                            | adjacent timesteps | target timesteps                                             |
| ------- | ------------------------------------------------------------ | ------------------ | ------------------------------------------------------------ |
| CM      | $t_{n+1}\sim\textrm{Uniform}(\{t_1, t_2, \dots\})$           | $t_{n}$            | $s:=t_1:=0$ (`Fixed`)                                        |
| CTM     | $t_{n+1}\sim\textrm{Uniform}(\{t_1, t_2, \dots\})$           | $t_{n}$            | $s\sim\textrm{Uniform}(\{t_1, t_2, \dots, t_{n}\})$ (`Random`) |
| TCD     | $t_{n+1}\sim\textrm{Uniform}(\{t_1, t_2, \dots\})$ (Sampled from all possible timesteps) | $t_{n}$            | $s\sim\textrm{Uniform}(\{t_1, t_2, \dots, t_{n}\})$ (`Random`) |
| PCM     | $t_{n+1}\sim\textrm{Uniform}(\{s_m, \dots, s_{m+1}\})$ (Sampled within pre-defined phases) | $t_{n}$            | $s := s_m$ (`fixed`)                                                     |

For the accusation of Equation 29 in TCD, please note that the loss of TCD accepts an additional timestep  `$t_m$`  for training, which is equivalent to the `$s$`  in the third row of the above table.  That is, TCD needs to `randomly choose` the target timestep. While for PCM, we fixed the target timestep (i.e., $s:=s_{m}$) before training and `set them to pre-defined`. `This is exactly the core difference between PCM and TCD, CTM`.

Therefore, `we argue that the training design of PCM and TCD, CTM are inherently different and the reviewer ukQK might misunderstand key aspects of our method`.
Dear AC and Reviewers,

We are writing to **strongly** defend the plagiarism accusation of Reviewer ukQK. A plagiarism accusation is very serious and shall not be made based on the incomplete understanding. We defend Reviewer ukQK's plagiarism accusation on our paper as follows:

1. Reviewer ukQK thinks our Equations 2, 3 are copied from TCD. However, `our Equation 2 is a direct conclusion formula from the previous paper DPM-Solver [2]`, which we have cited. In our submission, we write the sentence as the following: 

   > For the above-discussed PF-ODE, there exists an exact solution [Citation to DPM-Solver [2]] from timestep t and s,
   >
   > [Equation 2]

   In addition, our equation 3 is also a direct adaption of first-order approximation, which is equivalent to DDIM in implementation format. This is also a clear conclusion by DPM-Solver. 

Both TCD and PCM clearly cited DPM-Solver [2].  The important conclusion of DPM-Solver (NeurIPS 2022) shall not be considered as contributions of TCD (arXiv 2024).

For the accusation from Line 134 to Line 161.

- From Line 134 to 139, it follows the standard definition of Consistency Models [1], which is not proposed by TCD.

- From Line 143 to 150, it follows the standard conclusion of DPM-Solver [2], which is not proposed by TCD.

- From Line 151 to 157, it is our clarification on the epsilon prediction meaning of consistency models. We do not find any relevant discussion in the TCD paper.

- From Line 158 to 161, we do find some similar discussion on the boundary condition in the TCD paper. But please note that it is a natural conclusion from the exact solution formula proposed in DPM-Solver.

Thereby, we believe the plagiarism accusation on our paper copying TCD's formulas is groundless.
Thank you for maintaining the very positive score. We sincerely appreciate the time and effort you invested in reviewing our paper. Your feedback has been exceptionally valuable, and we are deeply grateful for your support.

We are always here should you have any further questions or need additional information.

Best regards,  
The Authors
Thank you for the rebuttal. I will maintain my original score.
Thank you for your reply!

We understand your concern on the paper: Trajectory Consistency Distillation: Improved Latent Consistency
Distillation by Semi-Linear Consistency Function with Trajectory Mapping (TCD) [1]. 



**Parameterization**

The parameterization of PCM was induced from the analysis form of the exact solution provided in the paper DPM-Solver [2].  All the characters used in our paper and first-order approximation proof follow DPM-Solver (including Eq. 2 - 3).  We carefully checked the paper TCD and found it provided a similar discussion from DPM-Solver. It appeared on the arXiv earlier. We are willing to add citations of TCD in the relevant discussion.  Both PCM and TCD cite DPM-Solver. 

Additionally,  we hope to clarify that's just a initially basic part for our further discussion. Our further discussion on the parameterization including the analysis of $\boldsymbol \epsilon_{\theta} (\mathbf x_t, t) = \frac{\int_{\lambda_t}^{\lambda_s}e^{-\lambda} \boldsymbol \epsilon_{\boldsymbol \phi}(\mathbf x_{t_{\lambda}(\lambda)}, t_{\lambda}(\lambda))\mathrm d \lambda}{\int_{\lambda_t}^{\lambda_s} e^{-\lambda} \mathrm d\lambda} $ and relevant proofs including, `error bound`, `CFG`, `adversarial loss`, have no intersections with TCD.

**Training objective**

The objective design of PCM is `totally different` from TCD. 

For equation 29 in TCD, please note that the loss of TCD accepts an additional timestep  `$t_m$`  for training, which is equivalent to the `$s$`  in the third row of following table. That is, they have to `randomly choose` the target timestep. While for PCM, we fixed the target timestep (i.e., $s:=s_{m}$) before training and `set them to pre-defined`. `This is exactly the core difference between PCM and TCD, CTM [3]`. Considering your concern, we are also willing to run additional comparison experiments with the official TCD and have a specific discussion on it in our paper.

We list the loss design comparison in the following table

| Methods | timesteps sampled                                            | adjacent timesteps | target timesteps                                             |
| ------- | ------------------------------------------------------------ | ------------------ | ------------------------------------------------------------ |
| CM      | $t_{n+1}\sim\textrm{Uniform}(\{t_1, t_2, \dots\})$           | $t_{n}$            | $s:=t_1:=0$ (`Fixed`)                                        |
| CTM     | $t_{n+1}\sim\textrm{Uniform}(\{t_1, t_2, \dots\})$           | $t_{n}$            | $s\sim\textrm{Uniform}(\{t_1, t_2, \dots, t_{n}\})$ (`Random`) |
| TCD     | $t_{n+1}\sim\textrm{Uniform}(\{t_1, t_2, \dots\})$ (Sampled from all possible timesteps) | $t_{n}$            | $s\sim\textrm{Uniform}(\{t_1, t_2, \dots, t_{n}\})$ (`Random`) |
| PCM     | $t_{n+1}\sim\textrm{Uniform}(\{s_m, \dots, s_{m+1}\})$ (Sampled within pre-defined phases) | $t_{n}$            | $s := s_{m}$ (`Fixed`,  no target timestep embedding)        |

From a more high-level perspective, as the sentence  "enabling `seamless transitions at any point` along the trajectory governed by the PF ODE"  stated in TCD,  CTM and TCD all try to learn arbitrary pairs.

While our method PCM `only learns transition within each sub-phase`, which not only easy to train but also does not require any target timestep embeddings.


**New Experiments**

Our paper is originally for high-resolution text-conditioned image and video generation, where we try to tackle the limitations of LCM in text-to-image generation and text-to-video generation, as stated in our paper. We did not test on the CIFAR and ImageNet since they are for unconditional or class-conditional low-resolution generation. For your request, we have tried our best to run as many experiments as possible on the CIFAR and ImageNet. Following the official CM,  the ImageNet experiments require training on 64 A100 GPUs for a week, which we can not afford in the limited rebuttal period.

[1] Trajectory consistency distillation

[2] Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps

[3] Consistency trajectory models: Learning probability flow ode trajectory of diffusion
I respectfully request that the other reviewers and the Area Chair take note of the potential plagiarism concerns regarding this paper.

I suspect that this paper may have plagiarized the work of [1] Zheng, Jianbin, et al. "Trajectory Consistency Distillation: Improved Latent Consistency Distillation by Semi-Linear Consistency Function with Trajectory Mapping." arXiv preprint arXiv:2402.19159 (2024). https://arxiv.org/abs/2402.19159v2

In accordance with Harvard’s Plagiarism Policy(https://usingsources.fas.harvard.edu/what-constitutes-plagiarism-0), this paper may be engaging in ***Inadequate Paraphrase*** and ***Uncited Paraphrasing***.


The reasons for my concerns are as follows:

   1. In the manuscript submitted by the author, from lines L134 to L161, the author asserts that their proposed parameterization is one of their core contributions. However, the equations presented, specifically Eq. 2 and 3, are identical to Eqs. 19 and 20 in [1]. Moreover, [1] was the first to apply Eq. 19 to the task of latent consistency distillation in text to image diffusion models. The description L134 to L161 also significantly overlaps with the parameterization discussion in Sec 4.2 of [1].
   2. The objective function detailed in lines L62 to L178 of the manuscript are consistent with Eq. 29 in [1], and the content and explanation in Sec 4.2 “training” of [1] closely resemble the author’s manuscript. 
   3.	The author frequently references [2] CTM, which conducted extensive experiments on CIFAR and ImageNet datasets during the experimental phase. However, the author’s manuscript does not mention these tasks in experiment at all. In their rebuttal, the author mentioned lacking sufficient computational resources to conduct experiments related to [2] CTM, which is particularly confusing to me. Furthermore, the tasks in the author’s manuscript are identical to those in [1].

And obviously, given the knowledge of [1], the innovation of this paper is significantly weakened both in task and experimental results.

***I am not able to make a definitive judgment at this stage. However, based on the current evidence, I believe this paper may be suspected of plagiarism.***

------
Here we paste the definition of Uncited Paraphrasing and Inadequate Paraphrase here:

1. Inadequate Paraphrase: When you paraphrase, your task is to distill the source's ideas in your own words. It's not enough to change a few words here and there and leave the rest; instead, you must completely restate the ideas in the passage in your own words. If your own language is too close to the original, then you are plagiarizing, even if you do provide a citation.

In order to make sure that you are using your own words, it's a good idea to put away the source material while you write your paraphrase of it. This way, you will force yourself to distill the point you think the author is making and articulate it in a new way. Once you have done this, you should look back at the original and make sure that you have represented the source’s ideas accurately and that you have not used the same words or sentence structure. If you do want to use some of the author's words for emphasis or clarity, you must put those words in quotation marks and provide a citation.

2. Uncited Paraphrasing: When you use your own language to describe someone else's idea, that idea still belongs to the author of the original material. Therefore, it's not enough to paraphrase the source material responsibly; you also need to cite the source, even if you have changed the wording significantly. As with quoting, when you paraphrase you are offering your reader a glimpse of someone else's work on your chosen topic, and you should also provide enough information for your reader to trace that work back to its original form. The rule of thumb here is simple: Whenever you use ideas that you did not think up yourself, you need to give credit to the source in which you found them, whether you quote directly from that material or provide a responsible paraphrase.


[1] Zheng, Jianbin, et al. "Trajectory consistency distillation: Improved Latent Consistency Distillation by Semi-Linear Consistency Function with Trajectory Mapping" arXiv preprint arXiv:2402.19159 (2024). https://arxiv.org/abs/2402.19159v2

[2] Kim, Dongjun, et al. "Consistency trajectory models: Learning probability flow ode trajectory of diffusion." arXiv preprint arXiv:2310.02279 (2023). https://arxiv.org/abs/2310.02279v3
Except for the validation on CIFAR and ImageNet. We hope to emphasize that we validate the design of PCM on much more baselines for all compared methods.

PCM (Ours): 1. `CIFAR-10`, 2. `ImageNet`, 3. `SD-v15`, 4. `SDXL`, 5.`Text-to-Video`

Instaflow: SD-v15

SDXL-Lightning: SDXL

CTM: CIFAR, ImageNet

CM: CIFAR, ImageNet

SDXL-Turbo: SDXL

LCM: SD v1-5, SDXL

AnimateLCM: Text-to-Video

We sincerely hope any further chance for discussion! Looking forward to your reply!
**The main metrics listed in the Table 1 of original CTM paper is augmented with more advanced sampling techniques while all the compared methods do not use them**. CTM applies a pretrained image classification model to score the generated samples, and only retain the sample with higher confidence and regenerate new samples if the confidence of the classification model is low. That means, `It needs to generate much more samples than all the compared methods`. It is noting that FID will generally become smaller when the number of samples become larger.
Besides, when comparing each baseline for fair, the principle design of CTM without GAN loss is even inferior to naive CM. 

The core difference of CM, CTM and PCM are listed as the following table. 

| Methods | timesteps sampled                                      | adjacent timesteps | target timesteps                                    |
| ------- | ------------------------------------------------------ | ------------------ | --------------------------------------------------- |
| CM      | $t_{n+1}\sim\textrm{Uniform}(\{t_1, t_2, \dots\})$     | $t_{n}$            | $s:=t_1:=0$                                         |
| CTM     | $t_{n+1}\sim\textrm{Uniform}(\{t_1, t_2, \dots\})$     | $t_{n}$            | $s\sim\textrm{Uniform}(\{t_1, t_2, \dots, t_{n}\})$ |
| PCM     | $t_{n+1}\sim\textrm{Uniform}(\{s_m, \dots, s_{m+1}\})$ | $t_{n}$            | $s := s_{m}$                                        |

In our paper, we have provided `thorough analysis and discussion with CTM`. The random choice design of target timestep make the design of CTM :
- more complex (additionally target timestep embedding)
- need more capacity ($\mathcal O (N^2)$ learning objectives)
- and harder to train (inferior performance without GAN loss)

Please let us know which part do you think is not convincing enough, we are willing for further discussion!
Thank you for your response!  Could you specify `which parts are not resolved properly`? We are willing for further reply and discussion.  For the data, we already showed that our model without applying any adversarial loss consistently surpassing the CTM w/ GAN on CIFAR through multistep refinement.  Additionally, we show that our model trained with only 3.5% training budget of the official CM achieved even superior performance.
I don’t believe the author’s response has adequately addressed my concerns. The paper still requires substantial experimental validation before it can be considered for publication. The author repeatedly compares their work with CTM throughout the paper, yet I found no clear advantages in the data provided in the rebuttal. This raises serious doubts about the paper’s contribution. As a result, I will be lowering my score.