Improved Training Technique for Latent Consistency Models
Authors: Quan Dao
Khanh Doan
Di Liu
Trung Le
Dimitris N. Metaxas
Venue: ICLR 2025 Poster (2025)
Keywords: Consistency Model
Diffusion Model
Primary area: generative models

Abstract: Consistency models are a new family of generative models capable of producing high-quality samples in either a single step or multiple steps. Recently, consistency models have demonstrated impressive performance, achieving results on par with diffusion models in the pixel space. However, the success of scaling consistency training to large-scale datasets, particularly for text-to-image and video generation tasks, is determined by performance in the latent space. In this work, we analyze the statistical differences between pixel and latent spaces, discovering that latent data often contains highly impulsive outliers, which significantly degrade the performance of iCT in the latent space. To address this, we replace Pseudo-Huber losses with Cauchy losses, effectively mitigating the impact of outliers. Additionally, we introduce a diffusion loss at early timesteps and employ optimal transport (OT) coupling to further enhance performance. Lastly, we introduce the adaptive scaling-$c$ scheduler to manage the robust training process and adopt Non-scaling LayerNorm in the architecture to better capture the statistics of the features and reduce outlier impact. With these strategies, we successfully train latent consistency models capable of high-quality sampling with one or two steps, significantly narrowing the performance gap between latent consistency and diffusion models. The implementation is released here: \url{https://github.com/quandao10/sLCT/}

Published as a conference paper at ICLR 2025
IMPROVED TRAINING TECHNIQUE FOR LATENT CON- SISTENCY MODELS
Quan Dao
Khanh Doan"
Rutgers University
Qualcomm Vietnam Company Limited khanhdn@que @qti qua comm. com
quan. dacê rut.gers. edu
Di Liu
Trung Le Monash University trung Lrunnnmeeeasshh.. edu
Dimitris Metaxas Rutgers University dnm@cs rutgers..ee.... edu
Rutgers University di 1..1irrutggrs.eed edu
ABSTRACT
Consistency models are family of generative models capable of producing high-quality samples either a single step or multiple steps. Recently, consis- tency models have demonstrated impressive performance, achieving results on par with diffusion models in the pixel space. However, the success of scal- ing consistency training to large-scall scale datasets, particularly for rext-to-image and video generation tasks, is determined by performance in the latent space. In this work, we analyze the statistical differences between pixel and latent spaces, dis- covering that latent data often contains highly impulsive outliers, which signifi- cantly degrade the performance of iCT in the latent space. To address this, we replace Pseudo-Huber losses with Cauchy losses, effectively mitigating the im- pact of outliers. Additionally, we introduce diffusion loss at early timesteps and employ optimal transport (OT) coupling to further enhance performance. Lastly, we introduce the adaptive scaling-c scheduler to manage the robust train- ing process and adopt Non-scaling LayerNorm in the architecture better cap- ture the statistics of the features and reduce outlier impact. With these strate- gies, successfully train latent consistency models capable of high-quality sam- pling with one or two steps, significantly narrowing the performance gap between latent consistency and diffusion models. The implemennation is released here: /tttp:://itte com quandao 0/sLCT/
INTRODUCTION
In recent years, generative models have gained significant prominence, with models like ChatGPT excelling language generation and Stable Diffusion (Rombach al.) 2027). computer vision, the diffusion model (Song et al.) 2020;[Song & Ermon 2019; Ho et al. 2020 Sohl-Dickstein et al. 2015) has quickly popularized and dominated the Adversarial Generative Model (GAN) (Good- fellow et al. 2014). It is capable of generating high-quality diverse images that beat SoTA GAN models (Dhariwal & Nichol 2021). Additionally, diffusion models are easier to train, as they avoid the common pitfalls of traiming instability and the need for meticulous hyperparameter tuning sociated with GANs. The application of diffusion spans the entire computer vision field, including text-to-image generation (Rombach al. 2021 Gu al. 2022), image editing (Meng al. 2021
Wu & la Torre 2023] Sluberman-Spiege
glas et al. 2024
Han et al. 2024 He et al. 2024).
text-
1o-3D generation (Poole al. 20222 Wang et 2024). personalization Ruiz et 2022 Van Le ct al. 2023 Kumari et al. (202)) and control generation (Zhang et al./ 2023b Brooks et al. 2023
Zhangli al.] 2024). Despite their powerful capabilities, they require thousands of function eval- uations for sampling, which is computationally expensive and hinders their application in the real world. Numerous efforts have been made to address this sampling challenge, either by proposing new training frameworks (Xiao et al. 2021 Rombach et al. 2021) or through distillation techniques
Equal contributions
Project Lead & Corresponding Author

---

Published as a conference paper at ICLR 2025
Meng et al. 2023 Yin et al. 2024 Sauer et al. 2023 Dao et al. 2024a).
However, methods like Xiao et | 2021 suffer from low recall due to the inherent challenges of GAN training, while
2021) still requires multi-step sampling. Distillation- based approaches, on the other hand, rely heavily on pretrained diffusion models and demand additional training.
(Rombach et al.
Recently, (Song al., 2023) introduced new family of generative models called the consistency
model. Compared to the diffusion model (Song & Ermon 2019; Song et al.) [020)] Ho et al. 2020), the consistency model could both generate high-quality samples in single step and multi-steps. The
consistency model could be obtained by either consistency distillation (CD) or consistency training (CT). In previous work (Song et al.) (023), CD significantly outperforms CT. However, the CD requires additional training budget for using pretrained diffusion, and its generation quality is inher- ently limited by the pretrained diffusion. Subsequent research (Song & Dhariwal 2023) improves the consistency training procedure, resulting in performance that not only surpasses consistency dis- tillation but also approaches SoTA performance of diffusion models. Additionally, several works (Kim et al. 2023 Geng et al. 22224) have further enhanced the efficiency and performance of CT, achieving significant results. However, all of these efforts have focused exclusively on pixel space, where data is perfectly bounded. In contrast, most large-scall scale applications of diffusion models, such as toxt-to--mage or video generation, operate in latent space (Rombach et al. [22T]] Gu et al.. 20022, as training on pixel space for large-scale datasets is impractical. Therefore, models for large datasets, the consistency must perform effectively in latent space. This work ad- dresses the key question: How well can consistency models perform in latent space? To explore this, we first directly applied the SoTA pixel consistency training method, iCT (Song & Dhariwal 2023), to latent space. The preliminary results were extremely poor, as illustrated in fig. motivating a deeper investigation into the underlying causes of this suboptimal performance. We aim to improve CT in latent space, narrowing the gap between the performance of latent consistency and diffusion.
scale consistency
We first conducted statistical analysis of both latent and pixel spaces. Our analysis revealed that the latent space contains impulsive outliers, which, while accounting for very small proportion, exhibit extremely high values akin to salt-and-pepper noise. We also drew a parallel between Deep O-Networks (DQN) and the Consistency Model, as both employ temporal difference (TD) loss. This could lead training instability compared to the Kullback-Leible (KL) loss used in diffusion models. Even in bounded pixel space, the TD loss still contains impulsive outliers, which (Song & 2023 addressed by proposing the use of Pseudo- Huber loss to reduce training instability. As shown in fig. the latent input contains extremely high impulsive outliers, leading to very large TD values. Consequently, the Pseudo-Huber loss fails to sufficiently mitigate these outliers, resulting in poor performance as demonstrated in fig. 5 To overcome this challenge, we adopt Cauchy loss, which heavily penalizes extremely impulsive outliers. Additionally, we introduce diffusion loss at early timesteps along with optimal transport (OT) matching, both of which significantly enhance the model's ' performance. Finally, we propose an adaptive scaling e schedule to effectively control the robustness of the model, and we incorporate Non-scaling LayerNorm into the architecture. With these techniques, we significantly boost the performance of latent consistency model compared to the baseline iCT framework and bridge the gap between the latent diffusion and consistency training.
Dhariwal
2 RELATED WORKS
Consistency model (Song et al. 2023 Song & Dhariwal 2023) proposes new type of generative model based on PF-ODE, which allows 1, 2 or multi-step sampling. The consistency model could be obtained by either training from scratch using an unbiased score estimator or distilling from pretrained diffusion model. Several works improve the training of the consistency model. ACT (Kong et al. (023). CTM (Kim et al. 2023) propose to use additional GAN along with consistency objective. While these methods could improve the performance of consistency training. they require an additional discriminator, which could need to tune the hyperparameters carefully. MCM (Heek) et al. 2024) introduces multistep consistency training, which is a combination of TRACT (Berth- clot et al. 2023) and CM (Song et al. 2023). MCM increases the sampling budget to 2-8 steps to tradeoff with efficient training and high- image generation. ECM (Geng et al. 2024) initial- izes the consistency model by pretrained diffusion model and fine-tuning it using the consistency training objective. ECM vastly achieves improved training times while maintaining good genera- tion performance. However, ECM requires pretrained diffusion model, which must use the same architecture as the pretrained diffusion architecture. Although these works successfully improve the
2

---

Published as a conference paper at ICLR 2025
performance and efficiency of consistency training, they only investigate consistency training on pixel space. As in the diffusion model, where most applications are now based on latent space, scal- ing the consistency training (Song et al., 2023 Song & Dhariwal 2023) to text-to-image or higher resolution generation requires latent space training. Otherwise, with pretrained diffusion model, we could either finetune consistency training (Geng et al., 2024) distill from diffusion model (Song) iet al. 2023; Luo et al. 2023). CM (Song et al. 2023) is the first work proposing consistency dis- tillation (CD) on pixel space. LCM (Luo et al. 2023) later applies consisteney technique on latent space and can generate high-quality images within few steps. However, LCM's 's generated images using 1-2 steps are still blurry (Luo al. (023). Recent works, such as Hyper-SD Ren et al.) (2024)
and TCD Zheng et al. (2024). have introduced notable improvements to latent consistency distilla-
tion. TCD Zheng et al. (2024) employed CTM Kim et al. (2023) instead of CD Song et al. (2023),
significantly enhancing the performance of the distilled student model. Building on this, Hyper-SD)
Ren et al. (202) divided the Probability Flow ODE (PF-ODE) into multiple components inspired by Multistep Consistency Models (MCM) Heek et al. (2024), and applied TCD Zheng et al. (2024)
to each segment. Subsequently, Hyper-SD Ren et al. (2024) merged these segments progressively
into final model, integrating human feedback learning and score distillation Yin et al. (2024) to
optimize one-step generation performance.
PRELIMINARIES
Denote Pdata (Xa) as the data distribution, the forward diffusion process gradually adds Gaus- sian noise with monotonically increasing standard deviation (t for E 10. T such that Pr (x,x xD) N(x,, o² (t) and (t) is handcrafted such that (0)) = min and (TT max By setting (1) I, the probability flow ODE (PF-ODE) from (Karras et al. 2022) is defined as:
dx.
Vx lx:) Xx: f(x:.
(1)
dt
where f: (xx, + Xo is the denoising function which directly predicts clean data x from given perturbed data Xt. (Song et al.) 2023) defines consistency model based on PF-ODE in eq. ( which builds a bijective mapping f between noisy distribution p(xx) and data distribution Pdata The bijective mapping f t) + Xo is termed the consistency function. A consistency model fo(x,, t) is trained approximate this consistency function fx,,,t) The previous works (Song et al. 2023 [Song& & Dhariwal 2023 Karras et al. 2022) impose the boundary condition by param- eterizing the consistency model as:
fx(x,, t) = Cakip + Cous (t) 8),
where Fx f) is a neural network to train. Note that, since (t) t, we hereafter use t and interchangeably. Cskip(()) (t) and (t) are time-dependent functions such that Cskip (Amin) = 1 and
Cout maxx)
To train or distill consistency model, (Song et al., firstly discretize the PF-ODE using sequence of noise levels J
2023 Song& & Dhariwal 2023 Karras et al.
2022
t,mi t
t
ty t max
Fmax. where t, uni + N +
i-1 (til
mlle) and p 7
Consistency Distillation Given the pretrained diffusion model (x,, t) =e Vx, (x,)) the con-
sistency model could be distilled from the pretrained diffusion model using the following CD loss:
(c)(0,0)) E ((f.))(ff(xx
te+1), fo- (x)),
(3)
where Xta1 = Xy + f4+1 with the xo ~ Pdata (xo) and z NN0 I) and Xt, = X (t: f++1) (441 V "x... log Pt.+, (X...) = Xt.41 - (t: t )l 1 1441).
Consistency Training The consistency model is trained by minimizing the following CT loss: CC 0-) = EE [f 1.+1). fx-(xx.. f:))].
(4)
where X,, Xo + t,z and Xt.,1 = Xo + f with the same X. ~ Pdata (X)) and z ~ I)
In eq. (3) and eq. (4. fo and fo- are referred as the online network and the target network, respectively. The target's parameter 0- is obtained by applying the Exponential Moving Average (EMA) to the student's parameter during the training and distillation as follows:
0 stopgrad(ju0 + (1 -=))),
(5)
3

---

Published as a conference paper at ICLR 2025
with 0 S μ 1 as the EMA decay rate, weighting function X(t,) for each timestep &, and d(:,,) is a predefined metric function.
In CM (Song et al. 2023). the consistency training still lags behind the consistency distillation and diffusion models. ICT (Song & Dhariwal, 2023) later propose several improvements that signifi- cantly boost the training performance and efficiency. First, the EMA decay rate fa is set to 0 for better training convergence. Second, the Fourier scaling factor of noise embedding and the dropout rate are carefully examined. Third, iCT introduces Pseudo-Huber losses to replace Lg and LPIPS since LPIPS introduces the undesirable bias in generative modeling (Song & Dhariwal 2023). >. Fur- thermore, the Pseudo- Huberr is more robust to outliers since it imposes a smaller penalty for larger errors than the L2 metric. Fourth, iCT proposes an exp curriculum for total discretization steps N, which doubles N after a predefined number of training iterations. Moreover, uniform weighting X(4)) = 1 is replaced by X(4+) = 1/(1++1 44). Finally, iCT adopts a discrete Lognormal distribu- tion for timestep sampling as EDM (Karras et al., (022). With all these improvements, CT is now
better than CD and performs on par with the diffusion models in pixel space.
4 METHOD
In this paper, we first investigate the underlying reason behind the performance discrepancy between latent and pixel space using the same training framework in section 4.1 Based on the analysis, we find out the root of unsatisfied performance on latent space could attributed to two factors: the impulsive outlier and the unstable temporal difference (TD) for computing consistency loss. To deal with impulsive outliers of TD on pixel space, (Song & Dhariwal 2023) proposes the Pseudo- Huber function as training loss. For the latent space, the impulsive outlier is even more severe, making Pseudo-Huber loss not enough to resist the outlier. Therefore, section 1.222..iiiuuueee Cauchy loss, which is more effective with extreme outliers. In the next section 4.3 and section 4.4
propose to use diffusion loss at early timesteps and OT matching for regularizing the overkill effect of consistency the early step and training variance reduction, respectively. Section 4.5 designs an adaptive scheduler of scaling e to control the robustness of the proposed loss function more carefully, leading to better performance. Finally, in section [466, we investigate the normalization layers of architecture and introduce Non-scaling LayerNorm to both capture feature statistic better and reduce the sensitivity to outliers.
4.1 ANALYSIS OF LATENT SPACE
2023) on the latent dataset CelebA 32 x 32 x 4 and pixel dataset CIFAR- 10 32 x 32 x 3. Hereafter, we refer to the latent iCT model as iLCT. We find that iCT framework works well on pixel datasets as claim (Song Dhariwal 2023). However, it produces worse results on latent datasets as in fig. 5 table|[] The iLCT gets a very high FID above 30 for both datasets, and the generative images are not usable in the real world. This observation raises concern about the sensitivity of CT algorithm with training data, and we should carefully examine the training dataset. In addition, we notice that the DQN and CM use the same TD loss, which update the current state using the future state. Furthermore, they also possess the training instability. This motivates to carefully examine the behavior of TD loss with different training data.
We first reimplement the iCT model (Song & Dhariwal
While the pixel data lies within the range [-1, 1] after being normalized, the range of latent data varies depending on the encoder model, which is blackbox and unbound. After normalizing latent data using mean and variance, we observe that the latent data contains high-magnitude values. We call them the impulsive outliers since they account for small probability but are usually very large values. In the bottom left of fig. []] the impulsive outlier of latent data is red, spanning from -9 to 7, while the first and third quartiles are just around 11.4 and 1.4, respectively. We evaluate how the iCT will be affected by data outliers by analyzing the temporal difference TD = fo(Xt f2+1) fo- ft). In the top right of fig. [ the impulsive outliers of pixel TD range from -1.5 to 1.7, which are not too far from the interquartile range compared to latent TD. The impulsive outliers of latent TD range is much wider from -3.2 to 5. iCT uses Pseudo Huber loss instead of Lg loss since the Huber is less sensitive outliers, see fig. 2 However, for latent data, the Huber's 's reduction in sensitivity to outliers is not enough. This indicates that even using Pseudo-Hubel loss, the iLCT training on latent space could still be unstable and lead to worse performance, which matches our
4

---

Published as a conference paper at ICLR 2025
Pixel
5
7
6
3
2
Latent
Data
Temporal Difference
Impulsive noise comparison between pixel and latent spaces. The right column shows the statistics of TT values at 21 discretization steps. Other discretization steps exhibit same behavior, where impulsive outliers are consistently present regardless of the total discretization steps. The blue boxes represent interquartile ranges of the data, while the green and orange dashed lines indicate inner and outer fences, respectively. Outliers are marked with red dots.
Figure 1: Box and Whisker Plot:
experiment results iLCT. Based the above analysis, hypothesize that the TD value statistic highly depends on the training data statistic.
To mitigate the impact of impulsive outliers, could use stable target updates like Polyak or periodic in TD loss Lee & He (2019), but they lead to very slow convergence, as shown in (Song 2023 Even though CM is initialized by pretrained diffusion model, the Polyak update still takes long time to converge. Therefore, using Polyak or periodic updates is computationally expensive, and we keep the standard target update as in (Song & Dhariwal (003)) Another direction is using special metric for latent like LPIPS on pixel space (Song et al. (023). (Kang et al.) 2024) proposes the E--LatentI.PIPS as a metric for distillation and performs well on distillation tasks. However, this requires training network as metric and using this metric during the training process will also increase the training budget. To avoid the overhead simple loss function like Pseudo-Huber but be more effective with outliers. We find that the Cauchy loss function (Black & Anandan 1996 Barron 2019) could be a promising candidate in place of Pseudo-Huber for latent space.
the training, we seek
4.2 CAUCHY LOSS AGAINST IMPULSIVE OUTLIER
In this section, we introduce the Cauchy loss (Black & Anandan 1996 Barron 2019) function to deal with extreme impulsive outliers. The Cauchy loss function has the following form:
dCanctty (x.y) = log (1+ [1+
||x-||lll y/1
(6)
2c²
and we also consider two additional robust losses, which are Pscudo-Huber (Song & Dhariwal 2023 Barron 2019) and Geman-McClure (Geman & Geman 1986 Barron 2019)
dpscudo-Huber (x. y) / ||x - 1 + c -c,
2 x y11
dGemann McCllre (x,y)
(8)
||x y1 + 4c
where e is the scaling parameter to control how robust the loss is to the outlier. We analyze their robustness behavior against outliers. As shown in fig. [2a] the Pseudo-Huber loss linearly increases like L1 loss for the large residuals x y. In contrast, the Cauchy loss only grows logarithmically, and the Geman-MeClure suppresses the loss value for the outliers.
The Pseudo-Huber loss works well if the residual value does not grow too high and, therefore, has good performance on the pixel space. However, for the latent space, as shown the bottom right of fig. [ the TD suffers from extremely high values coming from the impulsive outlier in the latent dataset, the Cauchy loss could be more suitable since significantly dampens the influence of extreme outliers. Otherwise, even Geman-McClure is very highly effective for removing outlier effects than two previous losses; it gives a gradient for high TD value and completely ignores the impulsive outliers as fig. 2b This is unexpected behavior because even though we call the high- value latent impulsive outlier, they actually could encode important information from original data. Completely ignoring them could significantly hurt the performance of training model. Based this analysis, we choose Cauchy loss as the default loss for latent CM for the rest of the paper. The loss ablation is provided in table
5

---

Published as a conference paper at ICLR 2025
Preado Rober
Cauchy
Deriniton
Derivativn
4.5
9.50
0.36
3.0
-3.25
(a) Robust Loss
(b) Derivative of Robust Loss
Figure 2: Analysis of robust loss: Pseudo-Huber, Cauchy, and Geman-McClure
4.3 DIFFUSION LOSS AT SMALL TIMESTEP
For small noise level the ground truth of (x) can be well approximated by Xo. but this does not hold for large noise levels. Therefore, for low-level noise, the consistency objective seems to be overkill and harms the model's performance since instead of optimizing fo(Xa. o) to approximated ground truth Xg. the consistency objective optimizes through a proxy estimator fa (x<os < a) leading error accumulation over timestep. To regularize this overkill, propose apply additional diffusion loss on small noise level as follows:
Laif = [[)((,,, 4) xo
VI S int(N r):
where the number training discretization steps and E [0; 1] is the diffusion threshold, and we heuristicly choose = 0.25. We do not apply diffusion loss for large noise levels since f(x,, o)) will differ greatly from the target Xo. leading to very high L2 diffusion loss. This could harm the training consistency process, misleading to the wrong solution. We provide the ablation study in table 2b] Furthermore, CTM (Kim et al. 2023) also proposes to use diffusion loss, but they use
them on both high and low-level noise, which is different from
4.4 OT MATCHING REDUCES THE VARIANCE
In this section, we adopt the OT matching technique from previous works (Pooladian et al. 2023
Lee al. 2023). (Pooladian et al., 2023) proposes use OT match noise and data the training batch, such as the moving L2 cost is optimal. On the other hand, (Lee et al.) (203)) introduces 3VAE for creating noise corresponding to data and train flow matching on the defined data-noise pairs. By reassigning noise-data pairs, these works significantly reduce the variance during the diffusion/flow matching training process, leading to a faster and more stable training process. According to (Zhang et al. 2023a), the consistency training and diffusion models produce highly similar images given the same noise input. Therefore, the final output solution of the consistency and diffusion models should be close to each other. Since OT matching helps reduce the variance during training diffusion, it could be useful to reduce the variance of consistency training. In our implementation, we follow (Pooladian al., 2023: Tong al., 2023) using the POT library to map from noise data in the training batch. The overhead caused by minibatch OT is relatively small, only around 0.93% training time, but gains significant performance improvement as shown in table 2a
4.5 ADAPTIVE e SCHEDULER
In this section, we examine the choice of scaling parameter in robust loss functions. The scal- ing parameter controls the robustness level, which is very important for model performance. The previous work (Song & Dhariwal 2023 proposes to use fixed constant co = 0.00054/dd, where the dimension of data. We find that using this simple fixed is not yet optimal for the training consistency model. Especially in this paper, we follow the Exp curriculum specified by eq. (TO) in (Song & Dhariwal (202)), which doubles the total discretization step after defined number of training iterations.
NFB(K) = min (002 + K'
K
(10)
loga
|$1/ /0p

---

Published as a conference paper at ICLR 2025
01>6
siaker
Tratning ng slapa (X1DP)
Thening stepa (x30')
Figure 3: Model convergence plot on different schedule. (Left) Our proposed e values. Perfor- mance on FID (Middle) and Recall (Right) of our proposed in comparison with different choices.
where l is current training iteration, K is total training iteration and so 10, 81 640. During training, we notice that the variance of TD is significantly reduced as doubling total discretization steps using eq. (TO). Since the more discretization steps, the closer distance of Xt, and X the TD value's range between them should be smaller. However, the impulsive outlier still exists regardless of the number of discretization steps. Intuitively, we propose heuristic adaptive e scheduler where the e is scaled down proportional to the reduction rate of TD variance as the number of discretization steps increases. We plot our scheduler versus discretization steps in fig. and fit the scheduler to get the scheduler equation following:
exp(-1.18 log((FE(()) - -1) 0.72)
(11)
4.6 NON-SCALING LAYERNORM
As mentioned in section the statistic of training data could play an important role in the success of consistency training. Furthermore, in architecture design, the normalization layer specifically handles the statistics of input, output, and hidden features. In this section, we investigate the nor- malization layer choice for consistency training, which is sensitive to training data statistics.
Song et al. 2023 use the UNet architecture from (Dhari- wal Nichol, 2021). UNet (Dhariwal & Nichol
Currently, both (Song & Dhariwal 2023
202T). GroupNorm is used every layer by groups of local channels, while the Lay- erNorm further captures the statistics' overall features. Therefore, LayerNorm is better at capturing fine-grained statistics over the entire feature. We further carry out the experiments for other types of normalization, such as LayerNorm, InstanceNorm, RMSNorm in table 2d and observe that the GroupNorm and InstanceNorm perform relatively well compared to others, especially LayerNorm. This could be due to that they are less sensitive to the outliers since they only capture the statistic groups of channels. Therefore, the impulsive features only affect the normalization of group containing them. For the LayerNorm, the impulsive features could negatively impact the overall features's normalization. We further look into the LayerNorm implementation and suspect that the scaling term could significantly amplify the outliers across features by serving as shared parameter. This observation is also mentioned in (Wei et al., 2022) for LLM quantization. In implementation. set the scaling term of LayerNorm to and disabled the gradient update for it equation 12 We refer as Non-scaling LayerNorm (NsLN) as (Wei al. 2022).
default. The GroupNorm only captures the statistics
x xx)(x)
" + B, NsLNg(x)== X --(())
LN,,a(X)
+ 3,
(12)
(x) +
(x2(x) +
where u(x) and (x) are mean and variance of x.
5 EXPERIMENT
5.1 PERFORMANCE OF OUR TRAINING TECHNIQUE
Experiment Setting: We measure the performance of our proposed technique on three datasets:
CelebA-HQ (Huang al. 2018), FFHQ (Karras al. 2019) and LSUN Church (Yu et al. 2015).
7

---

Published as a conference paper at ICLR 2025
Model
NFEL
FID
Recallf
Epochs
Total Bs
Kindi
APEL
FiB)
Reail!
Epadis
Toxal
Phat Difftodoon Model
Pixel Diftusion Model
Woethan
5:04
0.37
WaveDiff (Phung, etal.
2023
5.94
0.37
500
Scomn Sm1
DOISAN
Snewerrd Veary
4000
7:35
0.36
6.7K
Score SDE Sony al
2020
4000
7.23
6.2K
Lotest Difflssion Model
DDGAN Xigo
2021
7.64
0.36
800
LIMS LOMES
50
7:70
0.30
90
112
RDDOT arain 47244h
0.38
600
4.02
0.52
RDM Teng el al. 2023
270
3.15
0.55
4K
LSK
UNCSN++ KKm
2021
Latent Corsistency Model
Latent Diffusion Model
BAT
Scor & 1lbantnt
0.11
188
LFM-8
Dao
2023
0.41
500
731
0.27
0.58
18K
18K
256
LDM-4
Komnesch in 2021
200
5.11
0.49
600
48
15GM
Vandal
7.22
1K
(b) LSUN Church
DDMI
Fars e a. A024
1000
7.25
LDM-8 Phinng or 2024
DIMSUM
MT
73
3.76
0.56
395
32
Mode
XPE,
FiD;
Reall
Epedis
Total Bi
250
8.85
1.4K
128
Latest Difiuaio Model
LIM84
Don 2015
KoY
0.40
Latent Consistency Model
LIM-- RIWEROE O 30 A113
200
4:08
0.50
(18)
42
LUM5-
250
10.23
1.4k
128
LLCT Song & Dhariwal 2023
37.15
0.12
1.4K
128
Conekoency
ILCT Song Dharwal 2023
16.84
0.24
1.4K
128
BLT
Sitt & Dhariosl 2013
48
6:15
1,K
135
Ours Ours
7.27
0.50
1.4K
128
RLT
Heng & Mitrrio 2005
21
0.19
1.4K
6.93
0.52
1.4K
128
(a) CelebA-HQ
(c) FFHQ
Table I: Our performance on CelebA-HQ. LSUN Church, FFHQ datasets at resolution 256 x 256. (f) means training our machine with the same diffusion forward and equivalent architecture.
at the same resolution of 256 x 256. Following LDM (Rombach et al. 2021), we use pretrained VAE KL-8 to obtain latent data with the dimensionality of 32 x 32 x 4. We adopt the OpenAl UNet ar- chitecture (Dhariwal & Nichol, 202T) as the default architecture throughout the paper. Furthermore, we use the variance exploding (VE) forward process for all the consistency and diffusion experi- ments following (Song et al., 2023 Song & Dhariwal 2023). The baseline iCT is self-mmplemented based official implementation CM (Song al.) 2023) and iCT (Song & Dhariwal (023). We
refer to this bascline as iLCT. Furthermore, we also train the latent diffusion model for each dataset using the same VE forward noise process for fair comparisons with our technique. This LDM model is referred to LDM-8t in table 1 the same architecture
All three frameworks, including ours, iLCT, and LDM-81 use
During the evaluation, first generate 50K latent samples and then pass them through VAE's decoder to obtain the pixel images. We use two well-known metrics, Fréchet Inception Distance (FID) (Naeem et al. 2020) and Recall (Kynkäänniemi et al. 2019), , for measuring the performance of the model given the training data and 50K generated images.
Evaluation:
Model Performance: We report the performance of our model across all three datasets in table I primarily to compare it with the baseline iLCT (Song & Dhariwal 2023) and LDM (Rombach et al. 2021. For both 1 and NFE sampling, we observe that the FIDs of LLT for all datasets are notably high (over 30 for 1-NFE sampling and over 16 for 2-NFE sampling), consistent with the qualitative results shown in fig. [5] where the generated image is unrealistic and contain many artifacts. This poor performance of iL.TT'n in latent space is expected, as the Pseudo- Huber training losses are insufficient in mitigating extreme impulsive outliers, as discussed in section 4.1 T.Jaan and section 4.2. In contrast, our proposed framework demonstrates significantly better FID and Recall than iLCT. Specifically, achieve INFE sampling FIDs of 7. 7.27, 8.87, and 8.29 for CelebA-HQ. LSUN Church, and FFHQ, respectively. For 2-NFE sampling, our FID scores improve across all three datasets. Notably, our 1-NFE sampling outperforms LDM-81 using the same noise scheduler and architecture. However, our models still exhibit higher FIDs compared to LDM (Rombach et al. (02T) and LFM (Dao et al.) 2023). In contrast, we only need 1 or 2 timestep sampling, whereas they require multiple timesteps for high-fidelity generation. It's important to note that we employ the VE forward process, whereas these other methods use VP and flow-matching forward processes. Furthermore, the qualitative results of our framework, as shown in fig. highlight our ability to generate high-quality images.
8

---

Published as a conference paper at ICLR 2025
(a) CelebA-HQ
(b) LSUN Church
(c) FFHQ
Figure 4: Our qualitative results using I-NFE at resolution 256 256
(a) CelebA-HQ
(b) LSUN Church
(c) FFHQ
Figure 5: iLCT qualitative results using 1-NFE at resolution 256
256
5.2 ABLATION OF PROPOSED FRAMEWORK
We ablate our proposed techniques the CelebA-HQ 256 x 256 dataset, with all FID and Recall metries measured using 1-NFE sampling. All models are trained for 1,400 epochs with the same hyperparameters. As shown in table 2a replacing Pseudo-Huber losses with Cauchy losses makes our model's ' training less sensitive to impulsive outliers, resulting in a significant FID reduction from 37.15 to 13.02. This demonstrates the effectiveness of Cauchy losses in handling extremely high- value outliers, as discussed in section 4.2 Additionally, applying diffusion loss at small timesteps further reduces FID by approximately 4 points to 9.11, this loss term stabilizes the training process at small timesteps, as described in section 4.3 Introducing OT coupling during minibatch training reduces training variance, improving the FID to 8.89. Notably, by replacing the fixed scaling term (Song& & Dhariwal 2023) with an adaptive scaling schedule, our model achieves an addi- tional FID reduction of more than point, reaching 7.76, highlighting the importance the scaling term è in robustness control. Finally, we propose using NsLN, which removes the scaling term from LayerNorm to handle outliers more effectively. NsLN captures feature statistics while mitigating the negative impact of outliers, resulting in our best FID of 7.27.
Robustness Loss To analyze the impact of different robust loss functions, we conduct an ablation study using our best settings but replace the Cauchy loss with alternatives such as I.2, E-Latentl PIPS Kang al. (2024), the Huber and the that both Huber and Geman-McClure underperform compared to the Cauchy loss when applied in the latent space. This is because the Huber loss remains too sensitive to extremely impulsive outliers, while the Geman-McClure loss tends to ignore such outliers entirely, leading to loss of important information. This behavior is also discussed in section
loss. The results, shown table 2c indicate
Diffusion Threshold In this section, explore the impact of varying the threshold for applying the diffusion loss function in combination with the consistency loss. We observe that using the diffusion loss every timestep improves consistency training; however, it underperforms compared

---

Published as a conference paper at ICLR 2025
Loss
FID 4
Recall
Framework
FID
Recall f
L2
50.40
0.04
iLCT
37.15
0.12
E-LatentL
11.49
0.47
Cauchy
13.02
0.36
Huber
9.97
0,44
+ Diff
9.11
0,41
Geman McClure
11.28
0.44
+ OT
8.89
0.42
Cauchy
7.27
0.50
+ Scaled e
7.76
0,47
+ NsLN
7.27
0.50
(c) Robust losses.
Norm layer
FID
Recall f
(a) Components of proposed framework
r
FID +
Recall
GN
7.76
0.47
1.0
7.47
0.49
IN
8.47
0,43
0.6
7.33
0.49
LN
9.05
0.46
0.25
7.27
0.50
RMS
8.96
0,46
NsLN
7.27
0.50
(b) Threshold using Diffusion loss
(d) Norm Layer
Table 2: Ablation Studies CelebA-HQ 256 x 256 dataset at epoch 1400
to applying the diffusion loss selectively at smaller timesteps such as r = 0.25 as shown in table 2b This suggests that applying diffusion losses primarily at small noise levels improves performance as discussed section 4.3 At larger timesteps, the diffusion loss may conflict with the consistency loss, potentially guiding the model toward incorrect solutions, thereby reducing overall performance.
Scaling term e scheduler In this section, we compare the performance of our adaptive scaling scheduler with the fixed scaling scheduler proposed in (Song & Dhariwal] 2023). Our model demonstrates better convergence with the proposed adaptive scheduler. The rationale behind this improvement lies in the fact that, as the discretization steps increases using the exponential curricu- lum, the value of the TD scales down. Despite the reduced TD value, impulsive outliers still persist. fixed large scaling e is not effective in handling these outliers. To address this, we scale e down as discretization steps increases, which leads to better performance, shown in fig. 3
Normalizing Layer We denote GN, IN, LN, RMS, and NsLN as GroupNorm, InstanceNorm, LayerNorm, RMSNorm, and Non-scaling LayerNorm, respectively. The baseline UNet architecture from (Dhariwal & Nichol (02T) uses GroupNorm by default. We replace the normalization layers in the baseline with each of these types and train the model on CelebA-HQ using the best settings.
The results are reported in table 2d GN and IN only capture local statistics, making them more robust to outliers, outliers in region do not affect others. In contrast, LN captures statistics from all features, making it more vulnerable to outliers because an outlier affects all features through shared scaling term. By removing the scaling term in LN, we obtain NsLN, which is both effective NsLN outperforms the
in capturing feature statistics and resistant to outliers. As shown in table 2d second-best GN by 0.5 FID and significantly outperforms LN.
CONCLUSION
CT is highly sensitive to the statistical properties of the training data. In particular, when the data contains impulsive noise, such latent data, CT becomes unstable, leading to poor performance. In this work, we propose using the Cauchy loss, which is more robust to outliers, along with several im- proved training strategies to enhance model performance. As result, we can generate high-fidelity images from latent CT, effectively bridging the gap between latent diffusion models and consistency models. Future work could explore further improvements to the architecture, specifically by investi- gating normalization methods that reduce the impact of outliers. For example, removing the scaling term from group normalization or instance normalization may help mitigate outlier effects. Another promising future direction is the integration of this technique with Consistency Trajectory Models (CTM) Kim et al. (2023), as CTM has demonstrated improved performance compared to traditional Consistency Models (CM) Song et al. (2023).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
Research funded by research grants to Prof. Dimitris Metaxas from NSF: 2310966, 2235405, 2212301, 2003874, 1951890, AFOSR 23RT0630, and NIH 2101HL.
REFERENCES
Jonathan T Barron. A general and adaptive robust loss function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4331-4332 2019.
David Berthelot, Arnaud Autef, Jierui Lin, Dian Ang Yap. Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbott, and Eric Gu. Tract: Denoising diffusion models with transitive closure time-distillation. arXiv preprint arXiv:2303.04248. 2023.
Michael Black and Paul Anandan. The robust estimation of multiple motions: Parametric and piccewise-smooth flow fields. Computer vision and image understanding, 63(1):75-104, 1996.
Tim Brooks, Alcksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
Quan Dao, Hao Phung, Binh Nguyen, and Anh Tran. Flow matching in latent space. arXiv preprint S0Ii::3307.08698, 2023.
Quan Dao, Hao Phung, Trung Dao, Dimitris Metaxas, and Anh Tran. Self-corrected flow distillation for consistent one-step and few-step text-to-image generation. arXiv preprint 1rXiv:24/2.16990 2024a.
Quan Dao, Binh Ta, Tung Pham, and Anh Tran. high-quality robust diffusion framework for
corrupted dataset. In European Conference on Computer Vision, pp. 107-123. Springer, 2024b.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8777 8794, 2021.
Donald Geman and Stuart Geman. Bayesian image analysis. Disordered systems and biological organization, pp. 301-319. Springer. 1986.
Zhengyang Geng, Ashwini Pokle, William Luo, Justin Lin, and J Zico Kolter. Consistency models made easy. arXiv preprint arXiv:2406.1455 2024.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10696-10706, 2022.
Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpeng Song, Mengwei Ren, Ruijiang Gao, Anastasis Stathopoulos, Xiaoxiao He, Yuxiao Chen, et al. Proxedit: Improving tuning-fre free real image editing with proximal guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4291-4301, 2024.
Xiaoxiao He, Ligong Han, Quan Dao, Song Wen, Minhao Bai, Di Liu, Han Zhang, Martin Renqiang Min, Felix Juefei-Xu, Chaowei Tan, et al. Dice: Discrete inversion enabling controllable editing for multinomial diffusion and masked generative models. arXiv preprint arXiv:2410.08207, 2024.
Jonathan Heek, Emiel Hoogeboom, and Tim Salimans. Multistep consistency models. arXiv preprint ar 4rXhv:2003.06807, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.

---

Published as a conference paper at ICLR 2025
Huaibo Huang, zhihang li, Ran He, Zhenan Sun, and Tieniu Tan. Introvae: In- trospective variational autoencoders for photographic image synthesis. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. vances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL F1le/99856600222555777222222222.
and R. Garnett (eds.), Ad-
ngs nwwrips.r....e www.eiis.....aeepeeeeeee
cc/paper.pp.p... Ffles/paper/2015
Paper.ppp... pdf
Inbar Hluberman-Spiegeleee Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12469-12478, 2024.
Minguk Kang, Richard Zhang, Connelly Barnes, Sylvain Paris, Suha Kwak, Jaesik Park, Eli Shecht- man, Jun-Yan Zhu, and Taesung Park. Distilling Diffusion Models into Conditional GANs. In European Conference on Computer Vision (ECCV), 2024.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Proc. NeurIPS, 2022.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and II-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. arXiv preprint arXiv:2106.05522 2021.
Dongjun Kim, Chieh-Hsin Lai, Wei- Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning proba- bility flow ode trajectory of diffusion. arXiv preprint 2:Xiv:2110.02279, 2023.
Fei Kong, Jinhao Duan, Lichao Sun, Hao Cheng, Renjing Xu, Hengtao Shen, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. Act: Adversarial consistency models. arXiv preprint arXiv:2311. 140997, 2023.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Com- puler Vision and Patlern Recognition, pp. 1931-1941, 2023.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Donghwan Lee and Niao He. Target-based temporal-differecce learning. In International Confer- ence on Machine Learning, pp. 3713-3722. PMLR, 2019.
Sangyun Lee, Beomsu Kim, and Jong Chul Ye. Minimizing trajectory curvature of ode-based gen- erative models. arXiv preprini ar0i: /2003, 2023.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Syn- thesizing high-resolution images with few-step inference, 2023.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint 1oXiv:2/08.0107: 2021.
Chenlin Meng. Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14297-14306, 2023.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. ArXiv, abs/2002..99997 2020. https::::
semant cacholaaaaaa htt.://coooo.....oooooooooaaaaaa
12

---

Published as a conference paper at ICLR 2025
Dogyun Park, Sihyeon Kim, Sojin Lee, and Hyunwoo J Kim. Ddmi: Domain-agnostic latent diffusion models for synthesizing high-quality implicit neural representations. arXiv preprint arXiv:2401.12517, 2024.
Hao Phung, Quan Dao, and Anh Tran. Wavelet diffusion models are fast and scalable image gener- ators. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Pp. 10199-10208, June 2023.
Hao Phung, Quan Dao, Trung Dao, Hoang Phan, Dimitris Metaxas, and Anh Tran. Dimsum: Diffu- sion mamba-a scalable and unified Sreuuel-requuncc method for image generation. arXiv preprint wrXiv:2411.04166 2024.
Aram-Alexandre Pooladian, Heli Ben-Hamu, Carles Domingo-Ennn Brandon Amos, Yaron Lip- man, and Ricky TQ Chen. Multisample flow matching: Straightening flows with minibatch cou- plings. arXiv preprint arXiv:2304. 14772, 2023.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion, arXiv, 2022.
Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao. Hyper-sd: Trajectory segmented consistency model for efficient image synthesis. arXiv preprint arXiv:2404. 13686, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfrr Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint, 2022.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis- tillation. arXiv preprint arXiv:2311. 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265, PMLR, 2015.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint 10. 2023.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1, 13456, 2020.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 20023
Jiayan Teng, Wendi Zheng, Ming Ding, Wenyi Hong. Jianqiao Wangni, Zhuoyi Yang, and Jie Tang- Relay diffusion: Unifying diffusion process across resolutions for image synthesis. arXiv preprint arXiv:2309.033 03350, 2023.
Alexander Tong. Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks. Kilian Fatras, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. arXiv preprint arXiv:2300. 00482, 2023.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling latent space. Advances in neural information processing systems, 34: 1287-11302 2021.
Thanh Van Le, Hao Phung, Thuan Hoang Nguyen. Quan Dao. Ngoc N Tran, and Anh Tran. Anti- dreambooth: Protecting users from personalized -ontgee synthesis. In Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 2116-2127, 2023.
13

---

Published as a conference paper at ICLR 2025
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Feng- wei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer lan- guage models. Advances in Neural Information Processing Systems, 35:17402-17414 2022.
Chen Henry Wu and Fernando De la Torre. A latent space of stochastic diffusion models for zero- shot image editing and guidance. In ICCV, 2023.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arXiv:2112.078044 2021.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 661 6613-6623, 2024.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large- scale image dataset using deep learning with humans in the loop. ArXiv, 0abh/1506.03336 2015.
URL https: /tpp.app..../// wew.mmiccccooarr......a.
831744337
Huijie Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Peng Wang, Liyue Shen, and Qing Qu The
emergence of reproducibility and consistency in diffusion models. In Forty-firsl International Conference Machine Learning, 2023a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models, 2023b.
Qilong Zhangli, Jindong Jiang, Liu, Licheng Yu, Xiaoliang Dai, Ankit Ramchandani, Guan Pang, Dimitris N Metaxas, and Praveen Krishnan. Layout-agnostic scene text image synthesis with diffusion models. In 2024 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 7496-7506. IEEE Computer Society, 2024.
Jianbin Zheng, Minghui Hu, Zhongyi Fan, Chaoyue Wang, Changxing Ding, Dacheng Tao, and Tat-Jen Cham. Trajectory consistency distillation, arXiv preprint 191559 2024.
14

---

Published as a conference paper at ICLR 2025
A
APPENDIX
We provide additional uncurated samples of our models for three datasets: CelebaA-HQ (6 LSUN Church (8 Db. and FFHQ (IO) [T]. We also provide additional uncurated samples of our models on CelebaA-HQ trained with L2 loss (12) and E-LatentLPIPS loss (3).
Figure 6: One-step samples on CelebA-HQ HQ 256 x 256
BI
UIS
Figure 7: Two-step samples on CelebA-HQ 256 x 256
15

---

Published as a conference paper at ICLR 2025
Figure 8: One-step samples LSUN Church 256 256
Figure 9: Two-step samples LSUN Church 256 256
16

---

Published as a conference paper at ICLR 2025
Figure 10: One-step samples FFHQ 256 x 256
Figure 11: Two-step samples on FFHQ 256 x 256
17

---

Published as a conference paper at ICLR 2025
Figure 12: One-step samples on CelebA-HQ 256 256 (1.2 loss)
Figure 13: One-step samples on CelebA-HQ 256 x 256 (E-La loss)
18


Reviews:
[{"review_id": "GenSGxctiz", "reviewer": "Reviewer_fHyc", "summary": "The authors analyzed statistical differences between pixel space and potential space, and found that potential data often contained anomalous values of altitude pulses, which significantly reduced iCT performance in potential space. To solve this problem, they proposed the Cauchy loss, which effectively mitigated the effects of anomalies. Additionally, they introduce diffusion losses at early time steps and employ optimal transport coupling to further improve performance.", "questions": "1. The pseudo-huber loss also seems to be able to be characterized similarly to the Cauchy loss by adjusting c to adjust the degree of suppression of outliers. I'd like to know what the authors think.\n\n2. ﻿ The batch for consistency model training is usually large. What is the increase in total time when using POT to calculate optimal transport for noise data coupling?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The analysis and motivation of this manuscript is sound, and to the best of my knowledge, the effect of the potential spatial anomalies they first revealed on consistent model training.\n\n2. Each of the proposed techniques was well ablated.\n\n3. From the visualizations provided by the authors (Figures 4,5), several of the proposed techniques do significantly improve the visuals of data such as Celeba-HQ.", "weaknesses": "1. As an empirical paper, the authors seem to have compared only with iCTs that reproduce in hidden spaces. In fact, there have been many improvements on the consistent model of lifting hidden spaces, such as [1, 2], with which authors should compare or discuss.\n\n[1] Hyper SD: Trajectory Segmented Consistency Model for Effective Image Synthesis\n\n[2] Trajectory consistency disruption\n\n2. The authors' experiments are limited to some simple modal datasets such as the FFHQ, CELEBA-HQ datasets. Empirical evidence without multi-modal datasets weakens persuasion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lU1xeYvvTe", "reviewer": "Reviewer_2d6A", "summary": "This paper presents a series of techniques for enhancing the training of consistent models in potential space, including Cauchy loss, early diffusion loss, and optimal transport coupling between noise and samples.", "questions": "In my experience, some losses that are robust to outliers usually slow down the convergence of the model, due to the fact that these losses are equivalent to reduced gradients in large error regions, which may lead to slower convergence in early training than standard losses. Have the authors observed this phenomenon in their experiments?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The motivation of this article are very sound and the introductory section does a good job of presenting the motivation and contribution of the article. Given the popularity of potential spatial diffusion modeling, this research may have important implications for the practical application of accelerated diffusion.\n\n2. The authors found significant differences between latent space training and pixel space training. The latter is usually normalized and the former may have impulse noise. This factor may seem simple, but it has been long neglected.", "weaknesses": "1. The link to TD training in DQN in section 4.1 seems somewhat redundant. There is no evidence that DQN has a similar problem with impulse noise. And there is no solution proposed by the authors that is not derived from it.\n\n2. The authors mention in the introduction that the aim is to address the potential proliferation of large-scale applications such as text-to-image or video generation. However, instead of using a text-to-image model like LCM, the authors ended up experimenting on some simple pattern datasets, which is a minor shortcoming of the experimental evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sdhNk5vhD0", "reviewer": "Reviewer_7pHT", "summary": "The paper proposes a new strategy for consistency training of latent consistency models. It analyzes the reasons for the poor performance of improved consistency training in latent space, particularly addressing the issue of outliers by suggesting the use of Cauchy loss as a remedy. Additionally, by using x0 as the ground truth when t is small, the paper reduces the errors caused by model fitting. Compared to improved consistency training, the proposed method shows significant improvements in latent space.", "questions": "Please see the Weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe writing of the paper is clear.\n2.\tThe paper analyzes the reasons for the poor performance of improved consistency training in latent space.\n3.\tThe results of the proposed method achieved a great improvement than improved consistency training in latent space.", "weaknesses": "1.\tEquation (9) should be ||f(xt)-x0||^2.\n2.\tPlease briefly explain how the constant in Equation (11) is determined.\n3.\tBatch size has a significant impact on generative models, especially in consistency training; please include the batch size in Table 1.\n4.\tPlease include a comparison of the results from latent consistency distillation, including the resources used for training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N40VpqC4QS", "reviewer": "Reviewer_n8Mw", "summary": "This paper targets the efficient training problem for consistency models on latent space. They find that the highly impulsive outliers in the latent data significantly degrade the performance of iCT. Therefore, they employ a series of training tricks, including Cauchy loss, diffusion loss at early timesteps, optimal transport, adaptive scaling-c scheduler, and non-scaling layernorm to improve the performance.", "questions": "1. What is the correlation between the major contributions?\n2. Could you please provide more quantitative comparisons of your models with LCM and original CM?\n3. Why do you choose such a schedule in equation (11)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The performance reported in the paper is good compared to the self-implemented iCT.\n2. The motivation that delivers Cauchy loss, optimal transport, adaptive scaling-c scheduler, and non-scaling layernorm to handle highly impulsive outliers is convincing.", "weaknesses": "1. The soundness of the contribution in this paper is not good. The major contributions (Cauchy loss, optimal transport, adaptive scaling-c scheduler, and non-scaling layernorm) are mostly engineering training tricks. The correlation between the major contributions is not so strong, which makes the entire paper look like an A+B+C work. Since CM is already a complex model with so many hyper-parameters and training tricks, introducing more training tricks into CM doesn't appeal to me enough.\n\n2. The comparison of this paper with SOTA models is not sufficient. While the authors mention LCM in the paper, they do not compare their model with LCM in Table 1. This paper is mainly based on the impulsive outliers observed from their self-implemented iLCT, which is not so convincing to me. Does LCM also show impulsive outliers? Is it possible to adopt your training tricks to LCM? In addition, since iCT is not open-source, I recommend the authors show more comparisons of your model and the original CM. For example, what about L2 loss and LPIPS loss in Table 2 (b)?\n\n3. The ablation study to the hyper-parameters is not sufficient. Please provide more discussion on why you choose such a schedule in equation (11). \n\n4. The presentation of this paper is somehow repetitive and complicated. For example, Lines 164-175 are quite similar to the abstract and introduction part. And sections 4.4 and 4.6 are all descriptions with pure texts. Introducing more equations or figures may help to improve the presentation of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)