TLCM: Training- efficient Latent Consistency Model for Image Generation with 2-8 Steps

## Abstract

Abstract Distilling latent diffusion models (LDMs) into ones that are fast to sample from is attracting growing research interest. However, the majority of existing methods face two critical challenges: ( i ) They hinge on long training using a huge volume of real data.
( ii ) They routinely lead to quality degradation for generation, especially in text-image alignment.
This paper proposes a novel Training-efficient Latent Consistency Model (TLCM) to overcome these challenges.
Our method first accelerates LDMs via data-free multistep latent consistency distillation (MLCD), and then data-free latent consistency distillation is proposed to efficiently guarantee the inter-segment consistency in MLCD.
Furthermore, we introduce bags of techniques, e.g., distribution matching, adversarial learning, and preference learning, to enhance TLCM’s performance at few-step inference without any real data.
TLCM demonstrates a high level of flexibility by enabling adjustment of sampling steps within the range of 2 to 8 while still producing competitive outputs compared to full-step approaches.
Notably, TLCM enjoys the data-free merit by employing synthetic data from the teacher for distillation.
With just 70 training hours on an A100 GPU, a 3-step TLCM distilled from SDXL achieves an impressive CLIP Score of 33.68 and an Aesthetic Score of 5.97 on the MSCOCO-2017 5K benchmark, surpassing various accelerated models and even outperforming the teacher model in human preference metrics.
We also demonstrate the versatility of TLCMs in applications including image style transfer, controllable generation, and Chinese-to-image generation.

## 1 Introduction

Diffusion models (DMs) have made great advancements in the field of generative modeling, becoming the go-to approach for image, video, and audio generation (Ho et al., ; Kong et al., ; Saharia et al., ).
Latent diffusion models (LDMs) further enhance DMs by operating in the latent image space, pushing the limit of high-resolution image and video synthesis (Ma et al., ; Peebles & Xie, ; Podell et al., ; Rombach et al., ).
Despite the high-quality and realistic samples, LDMs suffer from frustratingly slow inference–generating a single sample requires tens to hundreds of evaluations of the model, giving rise to a high cost and bad user experience.

There is growing interest in distilling large-scale LDMs into more efficient ones. Concretely, progressive distillation (Lin et al., ; Meng et al., ; Salimans & Ho, ) reduces the sampling steps by half in each turn but finally hinges on a set of models for various sampling steps.
InstaFlow (Liu et al., ), UFO-Gen (Xu et al., ), DMD (Yin et al., ), and ADD (Sauer et al., ) target one-step generation, yet losing or weakening the ability to benefit from more (e.g., $>4$) sampling steps. Latent consistency models (LCMs) (Luo et al., ) apply consistency distillation (Song et al., ) on LDMs’ reverse-time ordinary differential equation (ODE) trajectories to conjoin one- and multi-step generation, but the image quality degrades substantially, especially in 2-4 steps.
HyperSD (Ren et al., ) applies consistency trajectory distillation (Kim et al., ) in segments of the ODE trajectory, yet suffers from a substantial performance drop in text-image alignment.
Besides, all these methods rely on a huge volume of high-quality data and long training time, hindering their applicability to downstream scenarios with rare computing and data.

Before presenting our proposal, we argue that one-step generation may not always be the optimal choice in practical applications. Empirically, sampling with 2-8 steps introduces less than 2 times additional computational time compared to one step for SDXL (Podell et al., ) but can notably enhance the upper limit of sampling quality.
Moreover, some practical applications typically have a low tolerance for quality degradation and hence can accept a moderate number of sampling steps. Thereby, this paper aims to develop a unified model with 2 to 8 sampling steps while achieving quality comparable to full-step models. We propose Training-efficient Latent Consistency Models (TLCMs) to achieve this at the expense of minimal computation and training data. Technically,
we introduce data-free multistep latent consistency distillation (MLCD) to reduce the sampling steps.
After MLCD, we further employ data-free latent consistency distillation (LCD) for global consistency.
To enhance LCD, we enforce the consistency of TLCM at sparse predefined timesteps instead of the entire timestep range, which reduces LCD’s learning difficulty and accelerates convergence.
A multistep solver is further explored to unleash the potential of the teacher in LCD.
Besides, we train a latent LPIPS model to constrain the perceptual consistency of the distilled model in latent space.
To optimize TLCM’s performance at few-step inference, we explore preference learning, distribution matching, and adversarial learning techniques for regularization in a data-free manner.

We have performed comprehensive empirical studies to evaluate TLCMs.
We first assess the image quality on the MSCOCO-2017 5K benchmark.
We observe the TLCM distilled from SDXL (Podell et al., ) gets an Aesthetic Score (AS) () of 5.97, and a CLIP Score (CS) (Hessel et al., ) of 33.68 with only 3 steps, substantially surpassing 4-step LCM, 8-step SDXL-Lightning (Lin et al., ), and 8-step HyperSD, comparable to 25-step DDIM.
Additionally, TLCM is obtained by only 70 A100 training hours without any real data, significantly reducing training costs.
We also demonstrate the versatility of TLCMs in applications including image stylization, controllable generation, and Chinese-to-image generation.

We summarize our contributions as follows:

- •
We propose TLCMs to accelerate LDMs to generate high-quality outputs within $2-8$ steps, at the expense of minimal training compute and data.
- •
We establish a data-free multistep latent consistency distillation and improved latent consistency distillation pipeline for fast LDM acceleration.
Besides, bags of data-free techniques are incorporated to boost rare-step quality.
- •
TLCM achieves a state-of-the-art CS (33.68) and AS (5.97) in 3 steps, surpassing competing baselines, such as 4-step LCM, 8-step SDXL-Lightning, and 8-step HyperSD.
- •
TLCMs’ versatility extends to scenarios such as image stylization, controllable generation, and Chinese-to-image generation, paving the path for extensive practical applications.

## 2 Related works

Diffusion models. (DMs) (Ho et al., ; Sohl-Dickstein et al., ; Song & Ermon, ; ; Song et al., ) progressively add Gaussian noise to perturb the data, then are trained to denoise noise-corrupted data.
In the inference stage, DMs sample from a Gaussian distribution and perform sequential denoising steps to reconstruct the data.
As a type of generative model, they have demonstrated impressive capabilities in generating realistic and high-quality outputs in text-to-image generation (Podell et al., ; Rombach et al., ; Saharia et al., ), video generation (Peebles & Xie, ).
To enhance the condition awareness in conditional DMs, the classifier-free guidance (CFG) (Ho & Salimans, ) technique is proposed to trade off diversity and fidelity.

Diffusion acceleration.
The primary challenges that hinder the practical adoption of DMs is the slow inference involving tens to hundreds of evaluations of the model.

Early works like Progressive Distillation (PD) (Salimans & Ho, ) and Classifier-aware Distillation (CAD) (Meng et al., ) explore the approaches of progressive knowledge distillation to compress sampling steps but lead to blurry samples within four sampling steps.
Consistency models (CMs) (Song et al., ),
consistency trajectory models (CTMs) (Kim et al., ) and Diff-Instruct (Luo et al., ) distill a pre-trained DM into a single-step generator, but they do not verify the effectiveness on large-scale text-to-image generation.

Recently, the distillation of large-scale text-to-image DMs has gained significant attention.
LCM (Luo et al., ) extends CM to text-to-image generation with few-step inference but synthesizes blurry images in four steps.
InstaFlow (Liu et al., ), UFOGen (Xu et al., ), BOOT (Gu et al., ), SwiftBrush (Nguyen & Tran, ), DMD (Yin et al., ), and Diffusion2GAN (Kang et al., ) propose one-step sampling for text-to-image generation but are unable to extend their sampler to multiple steps for better image quality.

More recently, SDXL-Turbo (Sauer et al., ), SDXL-Lighting (Lin et al., ), and HyperSD (Ren et al., ) are proposed to further enhance the image quality with a few-step sampling but their training depends on huge high-quality text-image pairs and expensive online training.
Our method not only enables the generation of high-quality samples using a 2-8 steps sampler but also enhances model performance with more inference cost.
Furthermore, our training strategy is resource-efficient and does not require any images.

Human preference for text-to-image model.
ImageReward (IR) (Xu et al., ) and Aesthetic Score  () are proposed to evaluate the human preference for the text-to-image model. Multi-dimensional Preference Score (MPS)  (Zhang et al., ) improves metrics by learning diverse preferences. To optimize TLCM towards human preference, we incorporate effective reward learning into TLCM to directly guide model tuning.

## 3 Preliminary

### 3.1 Diffusion Models

Diffusion models (DMs) (Ho et al., ; Sohl-Dickstein et al., ; Song et al., ) are specified by a predefined forward process that progressively distorts the clean data $x_{0}$ into a pure Gaussian noise with a Gaussian transition kernel.
It is shown that such a process can be described by the following stochastic differential equation (SDE) (Karras et al., ; Song et al., ):

$$ $dx_{t}=f(x,t)x_{t}dt+g(t)dw_{t},$ (1) $$

where $t\in[0,T]$, $w_{t}$ is the standard Brownian motion, and $f(x,t)$ and $g(t)$
are the drift and diffusion coefficients respectively.
Let $p_{t}(x_{t})$ denote the corresponding marginal distribution of $x_{t}$.

It has been proven that this forward SDE possesses the identical marginal distribution as the following probability flow (PF) ordinary differential equation (ODE) (Song et al., ):

$$ $dx_{t}=\left[f(x,t)x_{t}-\frac{1}{2}g^{2}(t)\nabla{x_{t}}\log p_{t}(x_{t}) \right]dt.$ (2) $$

As long as we can learn a neural model $\epsilon_{\theta}(x_{t},{t})$ for estimating the ground-truth score $\nabla{x_{t}}\log p_{t}(x_{t})$,
we can then draw samples that roughly follow the same distribution as the clean data by solving the diffusion ODE.
In practice, the learning of $\epsilon_{\theta}(x_{t},{t})$ usually boils down to score matching (Song et al., ).

The ODE formulation is appreciated due to its potential for accelerating sampling and has sparked a range of works on specialized solvers for diffusion ODE (Lu et al., ; ; Song et al., ).
Let $\Psi$ denote an ODE solver, e.g., the deterministic diffusion implicit model (DDIM) solver (Song et al., ).
The sampling iterates by

$$ $x_{t_{n-1}}=\Psi(\epsilon_{\theta}(x_{t_{n}},{t_{n}}),t_{n},t_{n-1}),$ (3) $$

where $\{t_{n}\}_{n=0}^{N}$ denotes a set of pre-defined discretization timesteps and $t_{N}=T,t_{0}=0$.

### 3.2 Consistency Models

Consistency model (CM) (Song & Dhariwal, ; Song et al., ) aims at generating images from Gaussian noise within one sampling step. Its core idea is to learn a model $f_{\theta}(x_{t},t)$ that directly maps any point $x_{t}$ on the trajectory of the diffusion ODE to its endpoint.
To achieve this, CMs first parameterizes $f_{\theta}(x_{t},t)$ as:

$$ $f_{\theta}(x_{t},t)=c_{skip}(t)x_{t}+c_{out}(t)F_{\theta}(x_{t},t),$ (4) $$

where $c_{skip}(t),c_{out}(t)$ is pre-defined to guarantee the boundary condition $f_{\theta}(x_{0},0)=x_{0}$, and $F_{\theta}(x_{t},t)$ is the neural network (NN) to train.

CM can be learned from a pre-trained DM $\epsilon_{\theta_{0}}$ via consistency distillation (CD) by minimizing (Song et al., ):

$$ $\mathcal{L}_{CD}=d\big{(}f_{\theta}(x_{t_{m}},t_{m}),f_{\theta^{-}}(x_{t_{n}}, t_{n})\big{)},$ (5) $$

where $t_{m}\sim\mathcal{U}[0,T]$, $x_{t_{m}}\sim p_{t_{m}}(x_{t_{m}})$, $t_{n}\sim\mathcal{U}[0,{t_{m}})$, $x_{t_{n}}=\Psi(\epsilon_{\theta_{0}}(x_{t_{m}},{t_{m}}),t_{m},t_{n})$, $d(.,.)$ is some distance function, and $\theta^{-}$ is the exponential moving average (EMA) of $\theta$.
Typically, $x_{t_{n}}$ is obtained by a single-step solver (SS) $\Psi$.

Multistep consistency models (MCMs) (Heek et al., ) generalize CMs by splitting the entire range $[0,T]$ into multiple segments and performing consistency distillation individually within each segment.
Formally, MCMs first define a set of milestones $\{t_{\mathrm{step}}^{s}\}_{\mathrm{s}=0}^{M}$ ($M$ denotes the number of segments), and minimize the following multistep consistency distillation (MCD) loss:

$$ $\mathcal{L}_{MCD}=d\big{(}\mathrm{DDIM}(x_{t_{m}},f_{\theta}(x_{t_{m}},t_{m}), t_{m},t_{\mathrm{step}}^{s}),\mathrm{DDIM}(x_{t_{n}},f_{\theta^{-}}(x_{t_{n}}, t_{n}),t_{n},t_{\mathrm{step}}^{s})\big{)},$ (6) $$

where $s$ is uniformly sampled from $\{0,\dots,M\}$, $t_{m}\sim\mathcal{U}[t_{\mathrm{step}}^{s},t_{\mathrm{step}}^{s+1}]$, $t_{n}=t_{m}-1$, and $\mathrm{DDIM}(x_{t_{m}},f_{\theta}(x_{t_{m}},t_{m}),t_{m},t_{\mathrm{step}}^{s})$ means one-step DDIM transformation from state $x_{t_{m}}$ at timestep $t_{m}$ to timestep $t_{\mathrm{step}}^{s}$ based on the estimated clean image $f_{\theta}(x_{t_{m}},t_{m})$ (Song et al., ).

Figure: Figure 2: The overview for training TLCM. Data-free multistep latent consistency distillation is first used to accelerate LDM, obtaining initial TLCM (left part of the overview). Then, improved data-free latent consistency distillation is proposed to enforce the global consistency of TLCM. MPS optimization, DM, and adversarial learning are exploited to promote TLCM’s performance in a data-free manner (right part of the overview). Note that we omit the Latent LPIPS model for brevity.
Refer to caption: x1.png

## 4 Methodology

Our target is to accelerate LDM into a few-step model, with performance competitive to the full-step teacher.
The learning procedure should be executed with
cheap cost in a data-free manner.
In this section, we propose a novel and unified Training-efficient Latent Consistency Model (TLCM) with 2-8 step inference.
We begin by introducing data-free multistep latent consistency distillation.
Subsequently, we discuss data-free latent consistency distillation to enforce the global consistency of TLCM.
Lastly, we explore various techniques to promote TLCM’s performance in a data-free manner. The overview of our training pipeline is presented in Figure 2.

### 4.1 Data-free Multistep Latent Consistency Distillation

We consider distilling representative pre-trained LDMs, e.g., SDXL (Podell et al., ).
Previous LCM (Luo et al., ) has distilled SDXL into a few-step model, but it results in a big performance drop since it is hard to learn the mapping between an arbitrary state of the entire ODE trajectory to the endpoint.
Drawing inspiration from MCM, we split the entire range [0, $T$]
into $M$ segments, and then only enforce the consistency at each separate segment.
To speed up convergence, we change the skipping step ($skip$) =1 in MCM into 20.
The EMA module is removed to save memory consumption.
Let $z_{t}$ denote the states at timestep $t$ in the latent space.
We abuse $\epsilon_{\theta_{0}}(z_{t},c,t)$ and $f_{\theta}(z_{t},c,t)$ to denote the pre-trained LDM and target model respectively, where $c$ refers to the generation condition.
We formulate the multistep latent consistency distillation (MLCD) loss as:

$$ $\mathcal{L}_{mlcd}=\|g_{\theta}(z_{t_{m}},t_{m},t_{step}^{s},c)-\text{nograd}( g_{\theta}(z_{t_{n}},t_{n},t_{step}^{s},c)\|_{2}^{2},$ (7) $$

where $g_{\theta}(z_{t_{m}},t_{m},t_{step}^{s},c)=\mathrm{DDIM}\big{(}z_{t_{m}},f_{
\theta}(z_{t_{m}},c,t_{m}),t_{m},t_{\mathrm{step}}^{s}\big{)}$ represents initial TLCM.
Given that CFG (Ho & Salimans, ) is critical for high-quality text-to-image generation, we integrate it to MLCD by:

$$ $z_{t_{n}}=\Psi(\hat{\epsilon}_{\theta_{0}}(z_{t_{m}},c,w,t_{m}),t_{m},t_{n}),$ (8) $$

where
$\hat{\epsilon}_{\theta_{0}}(z_{t},c,w,t):=\epsilon_{\theta_{0}}(z_{t},
\emptyset,t)+w(\epsilon_{\theta_{0}}(z_{t},c,t)-\epsilon_{\theta_{0}}(z_{t},
\emptyset,t))$ with $w$ as the guidance scale.

However, this training procedure depends on huge high-quality data, which limits its applicability in scenarios where such data is inaccessible.
To deal with this problem, we propose to draw samples from the teacher model as training data.
Specifically, instead of obtaining $z_{t_{m}}$ via adding noise to $z_{0}$ as in MCM and HyperSD,
we initialize $z_{T}$ as pure Gaussian noise $\epsilon$ and perform denoising with off-the-shelf ODE solvers based on the teacher model $\epsilon_{\theta_{0}}$ to obtain $z_{t_{m}}$.
Intuitively, with this strategy, we leverage and distill only the denoising ODE trajectory of the teacher without concerning the forward one.
The latent state $z_{t_{m}}$ can be acquired from $\epsilon$ by a single denoising step, but we empirically observe that this naive strategy is unable to accelerate LDM with desirable performance, due to poor quality of $z_{t_{m}}$.
Theoretically, $z_{t_{m}}$ contains less noise for smaller ${t_{m}}$. Therefore, we design a multistep denoising strategy (MDS) to predict $z_{t_{m}}$, which executes more sampling iterations for smaller ${t_{m}}$ to get cleaner $z_{t_{m}}$.
At this stage, the DDIM solver is used to estimate the ODE trajectory and generate samples from pure Gaussian noise.
We present the details in Algorithm 1 in Appendix A.1.

### 4.2 Improved Data-free Latent Consistency Distillation

After a round of distillation on $M$ segments, TLCM can naturally produce high-quality samples through $M$-step sampling. However, it is empirically observed that the performance decreases when using fewer steps, which is probably because of the larger discretization error caused by long sampling step sizes.
To alleviate this, we advocate explicitly teaching TLCM to capture the mapping between the states that cross segments. Upon this goal, we propose data-free latent consistency distillation to promote the model to be consistent across the predefined timesteps.

We do not compile TLCM to keep consistency across the entire timestep range [0, $T$] since it is hard to learn the mapping that transforms any point along the trajectory into real data. Instead, we improve raw LCD by only keeping consistency at the predefined $M$ timesteps, which makes LCD much easier to learn the mapping.
Naturally, the skipping step $skip$ is changed to $T/M$.
The big $skip$ offers an additional advantage that further accelerates model convergence.
Benefiting from the pre-trained TLCM, we can fast yield clean data $\hat{z}_{0}$ via a few-step ($q$-step) sampler, such as 4 steps, eliminating the requirement of real data.
The latent state $\hat{z}_{t_{m}}$ is obtained by adding noise to $\hat{z}_{0}$ in the forward diffusion process, where $t_{m}\in\{t_{\mathrm{step}}^{s}\}_{\mathrm{s}=1}^{M}$. We formulate this procedure as

$$ $\hat{z}_{t_{m}}=FD(TLCM(\epsilon,T,c),t_{m}),\quad\epsilon\in\mathcal{N}(0,I),$ (9) $$

where $FD$ and $TLCM$ denote forward diffusion and multistep iterations by TLCM.
Then, an ODE solver is used to estimate latent state $\hat{z}_{t_{n}}$ from $\hat{z}_{t_{m}}$.
Raw LCD adopts a one-step solver to predict $\hat{z}_{t_{n}}$.
We argue that it restricts the capability of the teacher due to discretization error, especially for big $skip$.
As a result, we explore a multistep solver (MS) to unleash the potential of the teacher.
Concretely, the time interval $T/M$ is uniformly divided into $p$ parts, and then $p$-step DDIM with CFG is used to calculate $\hat{z}_{t_{n}}$.
The improved data-free LCD loss in stage 2 is:

$$ $\mathcal{L}_{ilcd2}=\|\big{(}f_{\theta}(\hat{z}_{t_{m}},c,t_{m}))-\text{nograd }\big{(}f_{\theta}(\hat{z}_{t_{n}},c,t_{n}))\big{)}\|_{2}^{2}.$ (10) $$

We present the details in Algorithm 2 in appendix A.1.
Surprisingly, our improved data-free LCD only costs 2K-iteration training to achieve convergence.

### 4.3 Incorporating Bag of Techniques into TLCM in Data-free Manner

Latent LPIPS.
Typical LCD directly adopts mean square error loss ($\mathcal{L}_{mse}$) to enforce consistency in the latent space, but it can not capture perceptual features. LPIPS (Zhang et al., )
can extract the features matching human perceptual responses.
Meanwhile, it has been widely used as an effective regression loss across many image translation tasks.
Thereby, we aim to integrate LPIPS into our distillation pipeline to enhance TLCM’s performance.
However, LPIPS is built in the pixel space, and hence we have to reconstruct latent codes to pixel space to use LPIPS.
To reduce training time, we train a latent LPIPS (L-LPIPS) model, which
computes perceptual features in latent space.
The latent LPIPS model adopts the VGG network by
changing the input to 4 channels and removing the 3 max-pooling layers, as the latent space in LDM is already
8× downsampled.
The model is trained from scratch on BAPPS dataset (Zhang et al., ).
Based on L-LPIPS, the outputs of the model $g_{\theta}$ and $f_{\theta}$ are first fed into the L-LPIPS model, whose outputs are used to calculate consistency loss via Equation (7) or Equation (10).

MPS optimization.
Since TLCMs transform the points on the trajectory to clean samples $\hat{x}_{0}$, we can naturally directly maximize the feedback of the scorer on the sample $s(\hat{x}_{0},c)$.
Considering that multi-dimensional preference score (Zhang et al., ) can measure diverse human preferences, we leverage it to improve TLCM towards human preference.
Formally, we optimize the following MPS loss ($\mathcal{L}_{mps}$):

$$ $\mathcal{L}_{mps}=\max(s_{0}-s(\hat{x}_{0},c_{pos}),0)+\max(s(\hat{x}_{0},c_{ neg}),0),$ (11) $$

where $c_{pos}$ represents the text condition corresponding to the images while $c_{pos}$ denotes the irrelevant texts. $\mathcal{L}_{mps}$ maximizes $s(\hat{x}_{0},c_{pos})$ with margin $s_{0}$ and simultaneously minimizes $s(\hat{x}_{0},c_{neg})$ with margin 0.
The gradients are directly back-propagated from the scorer to model parameters $\theta$ for optimization.
We do not use ImageReward or AS to optimize TLCM, because we find IR tends to cause overexposure and AS results in oversaturation for generated images.

Distribution matching.
Distribution matching (Yin et al., ) is proposed to transform LDM into a one-step model.
We effectively integrate it into our distillation method to enhance the performance of TLCM.
To remove the need of real data, we exploit Equation 9 to get noisy latent $\hat{z}_{t}$.
Data-free DM loss in $\mathcal{L}_{dfdm}$ is applied to optimize TLCM at sparse-step inference as

$$ $\mathcal{L}_{dfdm}=-\mathbb{E}_{t,\epsilon,\hat{z}_{t}}[s_{real}(FD(f_{\theta} (\hat{z}_{t},t,c),t^{\prime}))-s_{fake}(FD(f_{\theta}(\hat{z}_{t},t,c),t^{ \prime}))\nabla_{\theta}f_{\theta}(\epsilon)],$ (12) $$

where $s_{real}$ and $s_{fake}$ denote the pre-trained score model and fake score model, both initialized by SDXL. The model $s_{fake}$ is finetuned on synthetic data $\hat{z}_{0}$ through noise prediction loss $\mathcal{L}_{diff}$ in DM (Yin et al., ).

Adversarial learning.
For high-resolution text-to-image
generation, considering the high data dimensionality and
complex data distribution, simply using MSE loss fails to capture data discrepancy precisely, thus providing imperfect consistency
constraints.
We propose to use GAN loss to enforce the distribution consistency.
Unlike previous methods needing real data to execute adversarial learning, we exploit Equation 9 to obtain $\hat{z}_{t}$.
The student model $f_{\theta}$ denoises $\hat{z}_{t}$ by one step, obtaining $\widetilde{z}_{0}$.
Through discriminator $D$, the GAN loss $\mathcal{L}_{gan}$ is formulated as

$$ $\mathcal{L}_{gan}=\log(D(FD(\hat{z}_{0},t^{\prime}))-\log(D(FD(\widetilde{z}_{ 0},t^{\prime}))).$ (13) $$

## 5 Experiments

### 5.1 Implementation Details

We use the prompts from LAION-Aesthetics- 6+ subset of LAION-5B (Schuhmann et al., ) to train our model. We train the
model with 12000 iterations for data-free MLCD and 2000 iterations for data-free LCD. After LCD, MPS optimization runs 500 iterations with a batch size of 8. Then, DM and adversarial learning are used to improve TLCM with 1000 iterations with a batch size of 4.
The whole procedure uses AdamW optimizer and 4 A100. Only MLCD adopts a learning rate of 1e-4 and the other stages use a learning rate of 1e-5. The discriminator adopts a learning rate of 1e-4 and AdamW optimizer.
The initial segment number $M$ is 8 and $s_{0}$ for MPS is 16. We set the guidance scale $w$ in CFG as 8.0, the denoising steps $p=3$ for the teacher to compute $\hat{z}_{t_{n}}$, and $q=4$ for TLCM to compute $\hat{z}_{0}$.
As for model configuration, we use SDXL (Podell et al., ) as teacher to estimate trajectory while student model $f_{\theta}$ is also initialized by SDXL. The discriminator is also initialized by SDXL.
We train a unified Lora instead of UNet in all the distillation stages for convenient transfer to downstream applications.

### 5.2 Main Results

We quantitatively compare our
method with both the DDIM (Song et al., ) baseline and acceleration
approaches including LCM (Luo et al., ), SDXL-Turbo (Sauer et al., ), SDXL-Lightning (Lin et al., ), HyperSD (Ren et al., ),
CS (Hessel et al., ) with ViT-g/14 backbone,
AS (), IR (Xu et al., ), and Fréchet Inception Distance (FID) are exploited as objective metrics.
The evaluation is performed on MSCOCO-2017 5K validation dataset (Lin et al., ).
All methods perform zero-shot validation except for HyperSD since it utilizes the MSCOCO-2017 dataset for training.
Only SDXL-Turbo produces 512-pixel images while the others generate 1024-pixel images. We only report FID for reference and do not analyze it since FID on COCO is not reliable for evaluating text-to-image models (Sauer et al., ; Ren et al., ).

**Table 1: Zero-shot performance comparison on MSCOCO-2017 5K validation datasets with the state-of-the-art methods. All models adopt SDXL architecture. Time: inference time (second) on A100. TH: Training hours using A100. TI: Training images.**
| Method | Step | Time | FID | CS | AS | IR | TH | TI |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DDIM (Song et al., ) | 25 | 3.29 | 23.29 | 33.97 | 5.87 | 0.82 | 0 | 0 |
| LCM (Luo et al., ) | 4 | 0.71 | 27.09 | 32.53 | 5.85 | 0.51 | - | - |
| SDXL-Turbo (Sauer et al., ) | 4 | 0.38 | 28.52 | 33.35 | 5.64 | 0.83 | - | - |
| SDXL-Turbo (Sauer et al., ) | 8 | 0.61 | 29.64 | 32.81 | 5.78 | 0.82 | - | - |
| SDXL-Lightning (Lin et al., ) | 4 | 0.71 | 27.90 | 32.90 | 5.63 | 0.72 | - | $>$12M |
| SDXL-Lightning (Lin et al., ) | 8 | 0.99 | 27.04 | 32.74 | 5.95 | 0.71 | - | $>$12M |
| HyperSD (Ren et al., ) | 4 | 0.71 | 34.45 | 32.64 | 5.52 | 1.15 | 600 | $>$12M |
| HyperSD (Ren et al., ) | 8 | 0.99 | 35.94 | 32.41 | 5.83 | 1.14 | 600 | $>$12M |
| TLCM | 2 | 0.58 | 27.50 | 33.18 | 5.90 | 0.97 | 70 | 0 |
| TLCM | 3 | 0.65 | 29.12 | 33.68 | 5.97 | 1.00 | 70 | 0 |
| TLCM | 4 | 0.71 | 30.33 | 33.52 | 6.06 | 1.01 | 70 | 0 |
| TLCM | 5 | 0.78 | 30.90 | 33.69 | 6.04 | 1.01 | 70 | 0 |
| TLCM | 6 | 0.85 | 30.98 | 33.71 | 6.07 | 1.01 | 70 | 0 |
| TLCM | 8 | 0.99 | 32.40 | 33.53 | 6.08 | 1.02 | 70 | 0 |

The metrics of various methods are listed in Table 1. We use “-” to represent a metric when it is missing in the corresponding paper.
We can observe that our TLCM only costs 70 A 100 training hours, even without any data. Compared to other methods, TLCM significantly reduces training resources, which is very valuable for most laboratories and scenarios when real data are inaccessible. our 3-step TLCM presents superior CS, AS, and IR than 4-8 step’s LCM (Luo et al., ), SDXL-Lightning (Lin et al., ).
These results indicate our TLCM’s synthetic images are much better aligned with texts and human preference than LCM, SDXL-Lightning. Excitingly, our 3-step TLCM outperforms the 25-step teacher in terms of AS and IR, and achieves comparable CS value, demonstrating that TLCM almost reserves all the information in the teacher and even introduces new human preference knowledge via the proposed distillation method.
Our 3-step TLCM shows much higher CS than the 4-8 step HyperSD, indicating HyperSD loses much information in the distillation procedure because it fails to sufficiently ensure consistency constraint.
We notice IR value of HyperSD is higher than our TLCM.
This is because IR model has been used to optimize HyperSD.
Moreover, we can see the performance of SDXL-Turbo drop with respect to CS and IR when increasing sampling steps.
This is because it is designed for specific steps.
Instead, our TLCM can improve at least one metric with additional steps.
This is valuable since image quality is the primary consideration when affordable computation resource is determined in real applications.

We present the visual comparisons in Figure 3.
Under the same conditions, we observe that the images generated by TLCM have better image quality and maintain higher semantic consistency on more challenging prompts, which also leads to greater human preference.

Figure: Figure 3: Visual comparison between our TLCM and the state-of-the-art methods. Zoom in for more details.
Refer to caption: extracted/5970485/figure/ddim/face.jpg

### 5.3 Ablation Study

To analyze the key components of our method, we make a
thorough ablation study to verify the effectiveness of the
proposed TLCM.
Table 2 depicts the performance of TLCM’s variants.

Data-free multistep latent consistency distillation.
As shown in Table LABEL:table:ablation, only using $\mathcal{L}_{lcd-s}$ which computes $z_{t_{m}}$ by single step for LCD achieves CS score of 31.61, AS of 5.89, indicating our data-free method is able to accelerate LDM with good quality.
Changing $\mathcal{L}_{lcd-s}$ to single-step denoising MLCD $\mathcal{L}_{mlcd-s}$, all metrics are improved. This result verifies that MLCD has a stronger capability to accelerate LDM than LCD.
This is because it is hard for data-free LCD to enforce consistency across the entire timestep range while data-free MLCD alleviates this by performing LCD within predefined multiple segments.

Denoising strategy.
We can observe from Table 2 that $\mathcal{L}_{mlcd-m}$
substantially enhances the performance of $\mathcal{L}_{mlcd-s}$, verifying that the proposed multistep denoising strategy is critical to perform data-free MLCD. The probable reason is our multistep MDS yields better initial latent codes, whereas the latent codes have better quality with smaller timesteps.

Latent LPIPS.
As outlined in Table 2, $\mathcal{L}_{mlcd-s}$ using L-LPIPS introduces gains on all metrics. This result denotes it is more powerful to enforce consistency in latent LPIPS space than raw latent space as latent LPIPS can make perceptual consistency.

Data-free latent consistency distillation in stage 2.
In  2, $\mathcal{L}_{lcd2}$ represents using multistep solver in LCD to enforce consistency across the entire timestep range. We can see that $\mathcal{L}_{lcd2}$ significantly improves CS values of TLCM trained in stage 1. This is because $\mathcal{L}_{lcd2}$ achieves inter-segment consistency of TLCM.
The performance is further enhanced by substituting $\mathcal{L}_{lcd2}$ with $\mathcal{L}_{ilcd2}$. The reason lies in that it is easier to make consistency along the sparse predefined timesteps than the entire timestep range.

MHP optimization.
Table 2 shows that adding $\mathcal{L}_{mhp}$ to the losses in line 7 introduces gains in terms of CS and IR. This result indicates that our MHP optimization method is capable of improving the text-image alignment and human preference of TLCM.

Data-free DM.
We can see in Table 2 using our data-free DM loss $\mathcal{L}_{dfdm}$ leads to the performance improvements on all metrics.
This result demonstrates that our DM in a data-free way is compatible with the proposed distillation method, boosting TLCM’s performance.

Discriminator.
We also observe in Table 2 that discriminator loss $\mathcal{L}_{gan}$ improves CS, AS, and IR since the discriminator facilitates consistency in probability distribution space, which is critical for the low-step regime.

Teacher’s inference steps of data-free latent consistency distillation in stage 2.
In Table 3, we study the effect concerning the teacher’s sampling steps of data-free LCD in stage 2. The results show as the sampling step increases from 1 to 4, the performance is consistently improved.
Therefore, it is crucial to perform multi-step denoising to estimate $\hat{z}_{t_{n}}$. The reason is that multi-step solvers are capable of reducing discretization error for big skipping step.

**Table 2: Ablation study of TLCM with respect to latent LPIPS, data-free LCD with single denoising step ($\mathcal{L}_{lcd-s}$), data-free MLCD with single denoising iteration ($\mathcal{L}_{mlcd-s}$), data-free MLCD with MDS ($\mathcal{L}_{mlcd-m}$), data-free LCD in stage 2 ($\mathcal{L}_{lcd2}$), improved data-free LCD in stage 2 ($\mathcal{L}_{ilcd2}$), data-free DM ($\mathcal{L}_{dfdm}$), multi-dimensional human preference ($\mathcal{L}_{mhp}$), adversarial ($\mathcal{L}_{gan}$). All the models adopt a 4-step sampler and SDXL backbone.**
| L-LPIPS | $\mathcal{L}_{lcd-s}$ | $\mathcal{L}_{mlcd-s}$ | $\mathcal{L}_{mlcd-m}$ | $\mathcal{L}_{lcd2}$ | $\mathcal{L}_{ilcd2}$ | $\mathcal{L}_{mhp}$ | $\mathcal{L}_{dfdm}$ | $\mathcal{L}_{gan}$ | CS | FID | AS | IR |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | ✓ |  |  |  |  |  |  |  | 31.61 | 32.90 | 5.89 | 0.41 |
|  |  | ✓ |  |  |  |  |  |  | 31.76 | 27.01 | 5.98 | 0.58 |
| ✓ |  | ✓ |  |  |  |  |  |  | 31.99 | 27.61 | 5.92 | 0.61 |
| ✓ |  |  | ✓ |  |  |  |  |  | 32.31 | 30.99 | 6.01 | 0.69 |
| ✓ |  |  | ✓ | ✓ |  |  |  |  | 32.74 | 32.05 | 6.00 | 0.72 |
| ✓ |  |  | ✓ |  | ✓ |  |  |  | 33.06 | 25.44 | 5.96 | 0.77 |
| ✓ |  |  | ✓ |  | ✓ | ✓ |  |  | 33.16 | 28.40 | 6.01 | 0.90 |
| ✓ |  |  | ✓ |  | ✓ | ✓ | ✓ |  | 33.32 | 30.58 | 6.03 | 0.97 |
| ✓ |  |  | ✓ |  | ✓ | ✓ | ✓ | ✓ | 33.52 | 30.33 | 6.06 | 1.01 |

**Table 3: Performance comparison of the teacher’s sampling steps for data-free LCD in stage 2.**
| Step | CS | FID | AS | IR | Step | CS | FID | AS | IR |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | 32.78 | 26.19 | 5.95 | 0.66 | 2 | 32.97 | 25.73 | 5.95 | 0.71 |
| 3 | 33.06 | 25.44 | 5.96 | 0.77 | 4 | 33.10 | 25.18 | 5.97 | 0.78 |

## 6 Conclusion

In this paper, we propose Training-efficient Latent Consistency Model (TLCM), a novel approach for accelerating text-to-image
latent diffusion models using only 70 A100 hours, without requiring any text-image paired data.
TLCM can generate high-quality, delightful images with only 2-8 sampling steps and achieve better image quality than baseline methods while being compatible with image style transfer, controllable generation, and Chinese-to-image generation.

## Appendix A Appendix

### A.1 Algorithms

Figure: Algorithm 1 Data-free multistep latent consistency distillation

Figure: Algorithm 2 Data-free latent consistency distillation in stage 2

Figure: Figure 4: TLCM with image style transfer. The styles are presented at the top, and we apply image style transfer on the source image with our TLCM. Two-step sampling can produce highly stylized images with excellent results.
Refer to caption: extracted/5970485/figure/stylish/panda.png

Figure: Figure 5: TLCM with ControlNet. Our TLCM can be incorporated into ControlNet pipeline and produce satisfactory results with 2 steps sampling.
Refer to caption: extracted/5970485/figure/controlnet/canny/dog.jpg

### A.2 Application

#### A.2.1 Acceleration of Image Style Transfer

Our TLCM LoRA is compatible with the pipeline of image style transfer (Mou et al., ).
We present some examples in Figure 4 with only 2-step sampling.

#### A.2.2 Acceleration of Controllable generation

Our TLCM LoRA is compatible with Controlnet, enabling accelerated controllable generation.
We utilize canny and depth ControlNet based on SDXL-base, together with TLCM LoRA in Figure 5.
The results are sampled in 2 steps.
We observe our model achieves superior image quality and demonstrates compatibility with other models, e.g. ControlNet, while also providing enhanced acceleration capabilities.

Figure: Figure 6: TLCM for Chinese-to-image generation. With 3 steps sampling, our TLCM model can produce images that align with Chinese semantic meaning. The first line presents images in general Chinese contexts, while the second line showcases images in specific Chinese cultural settings.
Refer to caption: x2.png

#### A.2.3 Acceleration of Chinese-to-image Generation

Our TLCM can accelerate the generation speed of the Chinese-to-image diffusion model (Ma et al., ).
We present some examples in Figure 6.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: As in weaknesses.
soundness: 2
strengths: -	The authors propose a two-stage distillation scheme that progressively distills the model to a few-step regime with reduced training costs.
-	The proposed method achieves superior results with SDXL on COCO-5k generation.
summary: The authors propose TLCM (Training-efficient Latent Consistency Model) for accelerating text-to-image latent diffusion models in a data-free manner with a small amount of training time. The key innovation is a two-stage distillation process. A data-free multistep latent consistency distillation (MLCD) is proposed to accelerate the base model, followed by another improved data-free latent consistency distillation to ensure global consistency. The authors enhance the model's performance through several techniques including latent LPIPS for perceptual consistency, multi-dimensional preference learning, distribution matching, and adversarial learning. Their empirical results show that TLCM can generate high-quality images in just 2-8 inference steps.
weaknesses: -	The submission is basically a technical report on achieving the SOTA few-step image generation performances with data-free multi-step consistency distillation, which is a successful practical combination of [1] and MCM. The second distillation stage also uses a series of SOTA techniques including ADD, DMD, and preference tuning. There are very limited scientific insights or technical innovations.
-	The current presentation is poor and needs substantial improvement. For instance, there is extensive and unnecessary usage of acronyms, making the submission so confusing and hard to read.
-	Limited empirical evaluation. The method is only validated on SDXL. How does it perform on other diffusion models with different architectures, e.g., Pixart?

References

1.	Kohler, Jonas, et al. "Imagine flash: Accelerating emu diffusion models with backward distillation." arXiv preprint arXiv:2405.05224 (2024).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: see above
soundness: 3
strengths: Advantages:
1. While straightforward, the idea shows some novelty
2. The writing is acceptable, though the innovation isn't immediately apparent
summary: This paper presents a direct combination of Multistep Consistency Models [1] and Latent Consistency Models [2], resulting in a Multistep Latent Consistency Model.

Advantages:
1. While straightforward, the idea shows some novelty
2. The writing is acceptable, though the innovation isn't immediately apparent

Disadvantages:
1. The experimental results appear counterintuitive: contrary to normal expectations where more sampling steps yield better results, this paper shows deteriorating FID scores with increased sampling steps
2. The paper lacks directness - the core innovations are hard to identify, as the authors seem to attempt highlighting multiple contribution points
3. The generated images predominantly show a cyberpunk style, lacking photorealism and overall quality

Technical Issues:
1. Equation 1 appears incorrect - contains an extra x_t term
   Should be: dx_t = f(x_t, t)dt + g(t)dw_t

2. Results Inconsistencies:
   - Tables 1 and 3 show FID increasing with more sampling steps, contradicting common understanding
   - Ablation study shows puzzling results: combining all techniques significantly worsens FID scores
   - Equations 11-13 look unnecessary, just traditional confrontational losses.

Writing Concerns:
The paper lacks focus on core contributions. It is hard for me to distinguish core contributions.

References:
[1] Heek et al., "Multistep consistency models," 2024
[2] Luo et al., "Latent consistency models," 2023
[3] Song et al., "Consistency models," 2023
weaknesses: Disadvantages:
1. The experimental results appear counterintuitive: contrary to normal expectations where more sampling steps yield better results, this paper shows deteriorating FID scores with increased sampling steps
2. The paper lacks directness - the core innovations are hard to identify, as the authors seem to attempt highlighting multiple contribution points
3. The generated images predominantly show a cyberpunk style, lacking photorealism and overall quality

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: (L144-145) This is a vague statement. What is the meaning of the effectiveness?
soundness: 2
strengths: - Interesting idea to use synthetic data for distillation.
- Reasonable motivations.
summary: Existing methods for distilling the pretrained knowledge of Diffusion Models have several drawbacks; expensive training time, necessity of large scale real data, and image generation quality given a few steps to generate an image. To this end, the authors propose a data-free distillation pipeline named as Training Efficient Latent Consistency Model (TLCM). Briefly, the initial parameter of TLCM is obtained by the proposed first process, and the second process boosts global consistency of TLCM. The experiment results show that TLCM has benefits over the baselines for the short training time and the data-free mechanism.
weaknesses: - Overview figure is very complex to understand. My suggestion is that it would be better if the right figure is decomposed into two parts; section 4.2 and section 4.3
- Weak explanations on the experiment settings.
    - What is the meaning of each measure? 
    - Is higher better or lower better? 
    - How many samples are used? 
    - What was the input to generate the samples to measure?
    - It sounds more reasonable if the proposed methods contain only IS or FID like [1]. The aim of the work is to distill the knowledge of the pretrained models and reduce the required sampling steps to some extent. Shouldn’t it enough If the generated images by TLCM  (2-8 steps) are as realistic as DDIM? Please provide justifications why the other metrics are needed in detail.  
- Weak interpretations of the experiment results.
    - (L364-L365) The proposed methods use synthetic data to distill the pretrained knowledge while the baselines are using real data. How does the proposed method show better performance than the baselines? It sounds more reasonable if the baseline is a sort of upper bound.
    - FID of TLCM (2 steps) is better than FID of TLCM (8 steps). It also shows the pattern that the # of steps and FID are inversely proportional, which is not reasonable. 
    - Prioritize what information is important for each table and emphasize those numbers.

- Too many engineering techniques are applied for improving the marginal performance. They dilute the main point of the paper.. 

Overall, the proposed methods are interesting, but the results (FID) are not convincing enough.

[1] PROGRESSIVE DISTILLATION FOR FAST SAMPLING OF DIFFUSION MODELS, ICLR’22

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: - After the emergence of rectified flow transformer, I think these kinds of distillation methods should be re-assessed. Apart from the non-predictable nature of the original diffusion trajectory, we now can learn the straightened flow of latent space and the distillation must be a lot easier. Even in the paper of SD3, it says that if the model capacity is given enough, reducing timesteps into a small number sacrifices little performance. This is not a critical comment about this paper and I would recommend the authors to think about this aspect.
soundness: 3
strengths: - Explanation about existing methods have been given enough. It is easy to follow and the application of each component sounds reasonable.
- Resulting images with only 4 steps look quite impressive.
- Necessary ablation studies to make this method solid have been provided.
summary: - The authors propose a multi-step distribution matching diffusion model to generate samples within few steps. By dividing the whole iterations into several predefined steps, the consistency model can focus on matching distribution in narrower range.
- Adversarial loss, latent-LPIPS loss, and human preference loss have also been adopted to boost the performance.
weaknesses: - Even though the results are quite impressive, this method is a combination of Multistep Consistency Models(MCM), Distribution Matching Distillation(DMD), and DiffusionGAN. It is hard to ignore novelty issue and this would be the main reason of my decision.
- Too many notations made me really hard to follow this paper. It would be much better if some expressions can be trimmed.

Minor issue
- The caption of the 3rd row in Fig.3 should be modified.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: 1. Does the 70GPU-hour training time include the time spent on data generation? The overall cost of data generation is needed to justify L241 'with cheap cost in data-free manner.'
soundness: 2
strengths: The paper presents the method in a very intuitive way with very informative equations and figures. The overall writing quality of this paper is good.
summary: This paper presents a method for latent diffusion model distillation. The core idea of this paper is mainly inspired by latent consistency distillation. The authors implemented some improvements based on the previous success of diffusion distillation, such as sparse timesteps. To prevent the cost of collecting real-world data, the proposed method is trained on generated data from the teacher model instead. A bag of independent techniques such as adversarial training are further included to help the model match the SoTA performance. The evaluations are performed on SDXL.
weaknesses: - Novelty

This paper presents insufficient novelties. The entire method seems to be an ensemble of diffusion distillation techniques already proven effective. For example, this paper advocates the advantage of relying on no training data and uses synthesized data instead. However, this has already been proven useful in SD3-turbo [1], and learning from generated data is by no means a new idea under the context of model distillation. 

- Empirical evaluations

This paper is presented with insufficient evaluations. SDXL, which is a relatively old model, is the only one evaluated in the paper. And there are some important comparisons missing such as SD3-turbo, DMD, and instaflow. 

The authors claim, 'We only report FID for reference and do not analyze it since FID on COCO is not reliable to evaluate text-to-image models.' I agree that FID is not the best way for diffusion evaluation. However, the values do reveal something about the proposed methods. I noticed that the FID values increase with the number of steps for the proposed method. In my understanding, this is a result of learning distillation from generated data instead of real data, as the modeled distribution deviates further from the real-world data distribution. 

- Supports to claims 

I find some claims in the paper require further support. For example, the authors claimed 'They (other distillation methods) need to perform long-time learning with a huge volume of real data.' I find it very hard to agree with this. According to my experience, methods such as LCM can be trained very fast with a small portion of training data. The authors are strongly encouraged to fill in the 'TH' value for LCM in Table 1 for a direct comparison. For the rest of the metrics, I don't see how the proposed method clearly outperforms the rest, except for IR, which is a score that the model is explicitly optimized for. 


[1] Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation, arxiv.