Improved Training Technique for Latent Consistency Models

Authors: Quan Dao, Khanh Doan, Di Liu, Trung Le, Dimitris N. Metaxas
Primary area: generative models
Keywords: Consistency Model, Diffusion Model

Abstract:
Consistency models are a new family of generative models capable of producing high-quality samples in either a single step or multiple steps. Recently, consistency models have demonstrated impressive performance, achieving results on par with diffusion models in the pixel space. However, the success of scaling consistency training to large-scale datasets, particularly for text-to-image and video generation tasks, is determined by performance in the latent space. In this work, we analyze the statistical differences between pixel and latent spaces, discovering that latent data often contains highly impulsive outliers, which significantly degrade the performance of iCT in the latent space. To address this, we replace Pseudo-Huber losses with Cauchy losses, effectively mitigating the impact of outliers. Additionally, we introduce a diffusion loss at early timesteps and employ optimal transport (OT) coupling to further enhance performance. Lastly, we introduce the adaptive scaling-$c$ scheduler to manage the robust training process and adopt Non-scaling LayerNorm in the architecture to better capture the statistics of the features and reduce outlier impact. With these strategies, we successfully train latent consistency models capable of high-quality sampling with one or two steps, significantly narrowing the performance gap between latent consistency and diffusion models. The implementation is released here: \url{https://github.com/quandao10/sLCT/}

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730345429571

thread_type:
official_review

<thread_object>
creator: reviewer_n8mw

<content>
summary:
This paper targets the efficient training problem for consistency models on latent space. They find that the highly impulsive outliers in the latent data significantly degrade the performance of iCT. Therefore, they employ a series of training tricks, including Cauchy loss, diffusion loss at early timesteps, optimal transport, adaptive scaling-c scheduler, and non-scaling layernorm to improve the performance.

strengths:
1. The performance reported in the paper is good compared to the self-implemented iCT.
2. The motivation that delivers Cauchy loss, optimal transport, adaptive scaling-c scheduler, and non-scaling layernorm to handle highly impulsive outliers is convincing.

weaknesses:
1. The soundness of the contribution in this paper is not good. The major contributions (Cauchy loss, optimal transport, adaptive scaling-c scheduler, and non-scaling layernorm) are mostly engineering training tricks. The correlation between the major contributions is not so strong, which makes the entire paper look like an A+B+C work. Since CM is already a complex model with so many hyper-parameters and training tricks, introducing more training tricks into CM doesn't appeal to me enough.

2. The comparison of this paper with SOTA models is not sufficient. While the authors mention LCM in the paper, they do not compare their model with LCM in Table 1. This paper is mainly based on the impulsive outliers observed from their self-implemented iLCT, which is not so convincing to me. Does LCM also show impulsive outliers? Is it possible to adopt your training tricks to LCM? In addition, since iCT is not open-source, I recommend the authors show more comparisons of your model and the original CM. For example, what about L2 loss and LPIPS loss in Table 2 (b)?

3. The ablation study to the hyper-parameters is not sufficient. Please provide more discussion on why you choose such a schedule in equation (11). 

4. The presentation of this paper is somehow repetitive and complicated. For example, Lines 164-175 are quite similar to the abstract and introduction part. And sections 4.4 and 4.6 are all descriptions with pure texts. Introducing more equations or figures may help to improve the presentation of this paper.

questions:
1. What is the correlation between the major contributions?
2. Could you please provide more quantitative comparisons of your models with LCM and original CM?
3. Why do you choose such a schedule in equation (11)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730618493659

thread_type:
official_review

<thread_object>
creator: reviewer_7pht

<content>
summary:
The paper proposes a new strategy for consistency training of latent consistency models. It analyzes the reasons for the poor performance of improved consistency training in latent space, particularly addressing the issue of outliers by suggesting the use of Cauchy loss as a remedy. Additionally, by using x0 as the ground truth when t is small, the paper reduces the errors caused by model fitting. Compared to improved consistency training, the proposed method shows significant improvements in latent space.

strengths:
1.	The writing of the paper is clear.
2.	The paper analyzes the reasons for the poor performance of improved consistency training in latent space.
3.	The results of the proposed method achieved a great improvement than improved consistency training in latent space.

weaknesses:
1.	Equation (9) should be ||f(xt)-x0||^2.
2.	Please briefly explain how the constant in Equation (11) is determined.
3.	Batch size has a significant impact on generative models, especially in consistency training; please include the batch size in Table 1.
4.	Please include a comparison of the results from latent consistency distillation, including the resources used for training.

questions:
Please see the Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730641071393

thread_type:
official_review

<thread_object>
creator: reviewer_2d6a

<content>
summary:
This paper presents a series of techniques for enhancing the training of consistent models in potential space, including Cauchy loss, early diffusion loss, and optimal transport coupling between noise and samples.

strengths:
1. The motivation of this article are very sound and the introductory section does a good job of presenting the motivation and contribution of the article. Given the popularity of potential spatial diffusion modeling, this research may have important implications for the practical application of accelerated diffusion.

2. The authors found significant differences between latent space training and pixel space training. The latter is usually normalized and the former may have impulse noise. This factor may seem simple, but it has been long neglected.

weaknesses:
1. The link to TD training in DQN in section 4.1 seems somewhat redundant. There is no evidence that DQN has a similar problem with impulse noise. And there is no solution proposed by the authors that is not derived from it.

2. The authors mention in the introduction that the aim is to address the potential proliferation of large-scale applications such as text-to-image or video generation. However, instead of using a text-to-image model like LCM, the authors ended up experimenting on some simple pattern datasets, which is a minor shortcoming of the experimental evaluation.

questions:
In my experience, some losses that are robust to outliers usually slow down the convergence of the model, due to the fact that these losses are equivalent to reduced gradients in large error regions, which may lead to slower convergence in early training than standard losses. Have the authors observed this phenomenon in their experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646830882

thread_type:
official_review

<thread_object>
creator: reviewer_fhyc

<content>
summary:
The authors analyzed statistical differences between pixel space and potential space, and found that potential data often contained anomalous values of altitude pulses, which significantly reduced iCT performance in potential space. To solve this problem, they proposed the Cauchy loss, which effectively mitigated the effects of anomalies. Additionally, they introduce diffusion losses at early time steps and employ optimal transport coupling to further improve performance.

strengths:
1. The analysis and motivation of this manuscript is sound, and to the best of my knowledge, the effect of the potential spatial anomalies they first revealed on consistent model training.

2. Each of the proposed techniques was well ablated.

3. From the visualizations provided by the authors (Figures 4,5), several of the proposed techniques do significantly improve the visuals of data such as Celeba-HQ.

weaknesses:
1. As an empirical paper, the authors seem to have compared only with iCTs that reproduce in hidden spaces. In fact, there have been many improvements on the consistent model of lifting hidden spaces, such as [1, 2], with which authors should compare or discuss.

[1] Hyper SD: Trajectory Segmented Consistency Model for Effective Image Synthesis

[2] Trajectory consistency disruption

2. The authors' experiments are limited to some simple modal datasets such as the FFHQ, CELEBA-HQ datasets. Empirical evidence without multi-modal datasets weakens persuasion.

questions:
1. The pseudo-huber loss also seems to be able to be characterized similarly to the Cauchy loss by adjusting c to adjust the degree of suppression of outliers. I'd like to know what the authors think.

2. ﻿ The batch for consistency model training is usually large. What is the increase in total time when using POT to calculate optimal transport for noise data coupling?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel strategy for training latent consistency models. The experiments demonstrate that the proposed strategy effectively enables high-quality sampling in just one or two steps, significantly reducing the performance gap between latent consistency models and diffusion models. The paper is well-written, well-organized, and makes a valuable empirical contribution to the community. A noted limitation is that, due to constraints in computational resources, the method was not evaluated on large-scale applications. However, the rebuttal systematically addressed most of the concerns raised by the reviewers in their initial assessments. This led to a consensus among all four reviewers, who agreed to accept the paper based on its innovation and empirical contributions. The AC concurs with the reviewers and recommends accepting the paper.