You Only Sample Once: Taming One-Step Text-to-Image Synthesis by Self-Cooperative Diffusion GANs

## Abstract

Abstract Recently, some works have tried to combine diffusion and Generative Adversarial Networks (GANs) to alleviate the computational cost of the iterative denoising inference in Diffusion Models (DMs).
However, existing works in this line suffer from either training instability and mode collapse or subpar one-step generation learning efficiency.
To address these issues, we introduce YOSO , a novel generative model designed for rapid, scalable, and high-fidelity one-step image synthesis with high training stability and mode coverage.
Specifically, we smooth the adversarial divergence by the denoising generator itself, performing self-cooperative learning. We show that our method can serve as a one-step generation model training from scratch with competitive performance.
Moreover, we extend our YOSO to one-step text-to-image generation based on pre-trained models by several effective training techniques (i.e., latent perceptual loss and latent discriminator for efficient training along with the latent DMs; the informative prior initialization (IPI), and the quick adaption stage for fixing the flawed noise scheduler). Experimental results show that YOSO achieves the state-of-the-art one-step generation performance even with Low-Rank Adaptation (LoRA) fine-tuning.
In particular, we show that the YOSO-PixArt- α 𝛼 \alpha italic_α can generate images in one step trained on 512 resolution, with the capability of adapting to 1024 resolution without extra explicit training, requiring only ~10 A800 days for fine-tuning. Our code is provided at https://github.com/Luo-Yihong/YOSO.

## 1 Introduction

Diffusion models (DMs) (Sohl-Dickstein et al., ; Ho et al., ; Song et al., ) have recently emerged as a powerful class of generative models, demonstrating state-of-the-art results in many generative modeling tasks, such as text-to-image (Rombach et al., ; Xu et al., ; Chen et al., ; Feng et al., ), text-to-video (Blattmann et al., ; Hong et al., ), image editing (Hertz et al., ; Brooks et al., ; Meng et al., ) and controlled generation (Zhang et al., ; Mou et al., ). However, the generation process of DMs requires iterative denoising, leading to slow generation speed.
Coupled with the intensive computational requirements of large-scale DMs, they constitute a substantial barrier to their practical application and wider adoption.

Sampling from DMs can be regarded as solving a probability flow ordinary differential equation (PF-ODE) (Song et al., ).
Some previous works (Song et al., ; Lu et al., ; ; Bao et al., ) focus on developing advanced ODE-solvers, for reducing the sampling steps. However, they still require 20+ steps to achieve high-quality generation. Another line is distilling from pre-trained PF-ODEs (Song et al., ; Liu et al., ; Luo et al., ; ; ; Salimans & Ho, ), aiming to predict multi-step solution of PF-ODE solver by one step. Existing methods (Luo et al., ; ) can achieve reasonable sample quality with 4+ steps. However, it is still challenging to generate high-quality samples with one step.

In contrast, generative adversarial networks (GANs) (Goodfellow et al., ; Radford et al., ) are naturally built on one-step generation with fast sampling speed. However, it is hard to extend GANs on large-scale datasets due to training challenges (Sauer et al., ; Kang et al., ),
resulting in worse sample quality compared to DMs (Sauer et al., ; Kang et al., ). In this work, we present a novel approach to combine diffusion process and GANs. The key to achieving access is finding a way to smooth the adversarial divergence for stabilize the training while maintaining effective one-step learning.
Previous works (Xiao et al., ; Xu et al., ; ; Sauer et al., ) have developed some variants for combining diffusion and GANs.
However, existing works either directly perform adversarial divergence against real data without smoothing strategy which suffers from unstable training and mode collapse, or rely on adding noise to smooth the adversarial divergence to stabilize the training which delivers less effective one-step generation learning.
To enjoy the best of both worlds, we propose to smooth the adversarial divergence by denoising the generator itself. In particular, we regard the one-step denoising generation based on less corrupted samples as ground truth and regard the one-step denoising generation based on more corrupted samples as student distribution to perform adversarial divergence.
The approach can not only naturally reduce the distance between target distribution and student distribution to stabilize the training, but also form effective one-step learning on clean samples. The learning process can be viewed as a self-cooperative process (Xie et al., ), as the generator learns from itself. Such an innovative design enables stable training and effective learning for one-step generation. Hence, we name our model YOSO, short for You Only Sample Once.

Moreover, we extend our YOSO to one-step text-to-image generation based on pre-trained models and introduce several effective training techniques (i.e., latent perceptual loss and latent discriminator for efficient training along with the latent DMs; the informative prior initialization (IPI), and the quick adaption stage for fixing the noise scheduler).
Thanks to our effective designs, we can efficiently and effectively fine-tune existing pre-trained text-to-image DMs (i.e., stable diffusion (Rombach et al., ) and PixArt-$\alpha$ (Chen et al., )) for high-quality one-step generation (see Fig. 1).
Furthermore, we are the pioneers in supporting Low-Rank Adaptation (LoRA) (Hu et al., ) fine-tuning in one-step text-to-image generation for enhanced efficiency, delivering state-of-the-art performance.

Figure: (a) Framework of YOSO.
Refer to caption: x1.png

Our work presents several significant contributions:

- •
We introduce YOSO, a novel generative model that can generate high-quality images with one-step inference, with a stable training process and good mode coverage.
- •
We further scale up YOSO by several principled and effective proposed training techniques that enable low-resource fine-tuning of pre-trained text-to-image DMs for one-step text-to-image, requiring only ~10 A800 days.
- •
We conduct extensive experiments to demonstrate the effectiveness of the proposed YOSO, including image generation from scratch, text-to-image generation fine-tuning, compatibility with existing image customization and image controllable modules.

## 2 Background

Diffusion models.
Diffusion models (DMs) (Sohl-Dickstein et al., ; Ho et al., ) define a forward process that gradually transforms samples from data distribution to Gaussian distribution by adding noise in $T$ steps with variance schedule $\beta_{t}$: $q({\mathbf{x}}_{t}|{\mathbf{x}}_{t-1})\triangleq\mathcal{N}({\mathbf{x}}_{t};
\sqrt{1-\beta_{t}}{\mathbf{x}}_{t-1},\beta_{t}\textbf{I})$.
The corrupted samples can be directly obtained by ${\mathbf{x}}_{t}=\bar{\alpha}_{t}{\mathbf{x}}_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon$, where $\bar{\alpha}_{t}=\prod_{s=1}^{t}(1-\beta_{T})$ and $\epsilon\sim\mathcal{N}(0,\mathbf{I})$.
The parameterized reversed diffusion process is defined to gradually denoise:
$p_{\theta}({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t})\triangleq\mathcal{N}({\mathbf{
x}}_{t-1};\mu_{\theta}({\mathbf{x}}_{t},t),\sigma_{t}^{2}\textbf{I})$.
The model can be trained by minimizing the negative ELBO (Ho et al., ; Kingma et al., ): $\mathcal{L}=\mathbb{E}_{t,q({\mathbf{x}}_{0})q({\mathbf{x}}_{t}|{\mathbf{x}}_{
0})}\text{KL}(q({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t},{\mathbf{x}}_{0})||p_{
\theta}({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t}))$
where $q({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t},{\mathbf{x}}_{0})$ is Gaussian posterior derived in (Ho et al., ). A key assumption in diffusion is that the denoising step size from $t$ to $t-1$ is sufficiently small. This assumption ensures the true $q({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t})$ follows Gaussian distribution, enabling the effectiveness of modeling $p_{\theta}({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t})$ with Gaussian distribution.

Diffusion-GAN hybrids.
An issue in DMs is that the true $q({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t})$ does not follow Gaussian distribution when the denoising step size is not sufficiently small.
Therefore, in order to enable large denoising step size, Diffusion GANs (Xiao et al., ) propose to minimize the adversarial divergence between model $p_{\theta}({\mathbf{x}}_{t-1}^{\prime}|{\mathbf{x}}_{t})$ and $q({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t})$: $\min_{\theta}\mathbb{E}_{q({\mathbf{x}}_{t})}[D_{\mathrm{adv}}(q({\mathbf{x}}_
{t-1}|{\mathbf{x}}_{t})||p_{\theta}({\mathbf{x}}_{t-1}^{\prime}|{\mathbf{x}}_{
t}))]$,
where $p_{\theta}({\mathbf{x}}_{t-1}^{\prime}|{\mathbf{x}}_{t})\triangleq\int p_{
\theta}({\mathbf{x}}_{0}|{\mathbf{x}}_{t})q({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t
},{\mathbf{x}})d{\mathbf{x}}_{0}$ and $p_{\theta}({\mathbf{x}}_{0}|{\mathbf{x}}_{t})$ is imposed by a GAN generator.
The capability of a GAN-based formulation enables much larger denoising step sizes (i.e., 4 steps).

## 3 Method: Self-Cooperative Diffusion GANs

A key issue in Diffusion-GAN hybrid models (Xiao et al., ; Xu et al., ; ) is that they match the generator distribution $p_{\theta}({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t})\triangleq\mathbb{E}_{p_{\theta
}({\mathbf{x}}_{0}|{\mathbf{x}}_{t})}q({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t},{
\mathbf{x}}_{0})$ with the corrupted data distribution.
This formulation only indirectly learns the $p_{\theta}({\mathbf{x}}_{0}|{\mathbf{x}}_{t})$ and
$p_{\theta}({\mathbf{x}}_{0})=\int q({\mathbf{x}}_{t})p_{\theta}({\mathbf{x}}_{
0}|{\mathbf{x}}_{t})d{\mathbf{x}}_{t}$, which are the distributions used for one-step generation, making the learning process less effective.

### 3.1 Our Design

To enable more effective learning for one-step generation, we propose to directly construct the learning objectives over clean data. We first construct a sequence distribution of clean data as follows:

$$ $\displaystyle p_{\theta}^{(t)}({\mathbf{x}}_{0})$ $\displaystyle=\int q({\mathbf{x}}_{t})p_{\theta}({\mathbf{x}}_{0}|{\mathbf{x}} _{t})d{\mathbf{x}}_{t},\quad 0<t\leq T;\quad p_{\theta}^{(0)}({\mathbf{x}}_{0} )\triangleq q({\mathbf{x}}_{0}),$ (1) $$

where $q({\mathbf{x}}_{0})$ is the data distribution, $p_{\theta}({\mathbf{x}}_{0}|{\mathbf{x}}_{t})\triangleq\mathcal{N}(G_{\theta}(
{\mathbf{x}}_{t},t),\sigma^{2}\mathbf{I})$ and $G_{\theta}$ is the denoising generator.
Note that $G_{\theta}({\mathbf{x}}_{t},t)$ is our denoising generator that predicts clean samples. If the network $\epsilon_{\theta}$ is parameterized to predict noise, we have $G_{\theta}({\mathbf{x}}_{t},t)\triangleq\tfrac{{\mathbf{x}}_{t}-\sqrt{1-\bar{
\alpha}_{t}}\epsilon_{\theta}({\mathbf{x}}_{t},t))}{\bar{\alpha}_{t}}$.

Given the constructed distribution, we can formulate the optimization objective as follows:

$$ $\displaystyle\mathbb{E}_{t}[D_{\mathrm{adv}}(q({\mathbf{x}})||p^{(t)}_{\theta} ({\mathbf{x}}))+\lambda\cdot\text{KL}(q({\mathbf{x}}_{0},{\mathbf{x}}_{t})||p_ {\theta}({\mathbf{x}}_{0},{\mathbf{x}}_{t}))]$ (2) $\displaystyle=\mathbb{E}_{t}[D_{\mathrm{adv}}(q({\mathbf{x}})||p^{(t)}_{\theta }({\mathbf{x}}))+\lambda_{t}\cdot\text{KL}(q({\mathbf{x}}_{0})q({\mathbf{x}}_{ t}|{\mathbf{x}}_{0})||q({\mathbf{x}}_{t})p_{\theta}({\mathbf{x}}_{0}|{\mathbf{ x}}_{t}))],$ $$

where $q({\mathbf{x}}_{0},{\mathbf{x}}_{t})\triangleq q({\mathbf{x}}_{0})q({\mathbf{x
}}_{t}|{\mathbf{x}}_{0})$ and $p_{\theta}({\mathbf{x}}_{0},{\mathbf{x}}_{t})\triangleq q({\mathbf{x}}_{t})p_{
\theta}({\mathbf{x}}_{0}|{\mathbf{x}}_{t})$. The optimization objective is constructed by combining adversarial divergence and KL divergence. Specifically, the adversarial divergence focuses on matching over the distribution level, ensuring the generation quality, and the KL divergence focuses on matching over point level, ensuring the mode coverage.

However, it is hard to directly learn the adversarial divergence over clean data distribution, akin to the challenges with GAN training. To tackle these challenges, previous Diffusion GANs (Xiao et al., ; Xu et al., ) have pivoted towards learning the adversarial divergence over the corrupted data distribution. Unfortunately, as analyzed before, such an approach fails to directly match $p_{\theta}({\mathbf{x}}_{0})$, curtailing the efficacy of one-step generation. Moreover, it also compels the discriminator to fit different levels of noise, leading to limited capability.

Recall that $p^{(t)}_{\theta}({\mathbf{x}})$ is defined as $\int q({\mathbf{x}}_{t})p_{\theta}({\mathbf{x}}|{\mathbf{x}}_{t})d{\mathbf{x}}
_{t}$. The quality of the distribution has two key factors: 1) the ability of the trainable generator $G_{\theta}$; 2) the information given by ${\mathbf{x}}_{t}$.

Hence given the generator $G_{\theta}$ fixed, if we increase the information in ${\mathbf{x}}_{t}$, it is supposed that we can get a better distribution. In other words, it is highly likely that $p^{(t_{k})}_{\theta}({\mathbf{x}})$ is superior to $p^{(t)}_{\theta}({\mathbf{x}})$, where $t_{k}=\max\{t-k,0\}<t$. Motivated by the cooperative approach (Xie et al., ; ; ; Hill et al., ) which uses an MCMC revised version of model distribution to learn the generator, we suggest to use $p^{(t_{k})}_{\theta}({\mathbf{x}})$ as ground truth to learn $p^{(t)}_{\theta}({\mathbf{x}})$, which constructs the following training objective:

$$ $\displaystyle\min_{\theta}\mathcal{L}_{\theta}\triangleq\mathbb{E}_{t}\mathbb{ E}_{q({\mathbf{x}})q({\mathbf{x}}_{t}|{\mathbf{x}})}\lambda_{t}\times||G_{ \theta}({\mathbf{x}}_{t},t)-{\mathbf{x}}||_{2}^{2}+\mathbb{E}_{t}(D_{\mathrm{ adv}}(p^{(t_{k})}_{\theta}(\texttt{sg}({\mathbf{x}}))||p^{(t)}_{\theta}({ \mathbf{x}})),$ (3) $$

where $\texttt{sg}[\cdot]$ denotes stop-gradient operator and the second term is named as cooperative adversarial loss. This training objective can be regarded as a self-cooperative approach, since the ‘revised’ samples from $p^{(t_{k})}_{\theta}({\mathbf{x}})$ and samples from $p^{(t)}_{\theta}({\mathbf{x}})$ are generated by the same network $G_{\theta}$. Note that we only replace data distribution as $p^{(t_{k})}_{\theta}({\mathbf{x}})$ in the adversarial divergence for smoothing the learning objective, as recent work (Luo et al., ) has found that it is beneficial to learn the generator with a mix of real data and revised data.

We briefly verify the theoretical rationality of the cooperative adversarial divergence below.

###### Proposition 1

The optimal solution of the cooperative adversarial loss reaches $p^{(T)}_{\theta}({\mathbf{x}})=p_{d}({\mathbf{x}})$.

This proposition tells that the proposed cooperative adversarial loss can recover the real data distribution when the network’s capability is sufficiently large, demonstrating the theoretical rationality of the proposed objective. See proof in the Appendix C.

In the distribution matching objective above, we apply the non-saturating GAN objective to minimize the adversarial divergence of marginal distributions. And the KL divergence for point matching can be optimized by $L_{2}$ loss. Hence a tractable training objective is formulated as follows:

$$ $\displaystyle\min_{\theta}\max_{\phi}\mathbb{E}_{t}[\mathbb{E}_{p_{\theta}^{(t _{k})}({\mathbf{x}})}\log D_{\phi}(\texttt{sg}({\mathbf{x}}),t)-\mathbb{E}_{p_ {\theta}^{(t)}({\mathbf{x}})}\log D_{\phi}({\mathbf{x}},t)]+\lambda_{t}\mathbb {E}_{q({\mathbf{x}})q({\mathbf{x}}_{t}|{\mathbf{x}})}||G_{\theta}({\mathbf{x}} _{t},t)-{\mathbf{x}}||_{2}^{2},$ $$

where $D_{\phi}$ is the discriminator network. We find that the self-cooperative approach is connected with Consistency Training (Song et al., ). However, Consistency Training considers the ${\mathbf{x}}_{t_{k}}$ as an approximated ODE solution of ${\mathbf{x}}_{t}$ to perform a point-to-point match.
In contrast, our proposed objective matches $p^{(t)}_{\theta}({\mathbf{x}})$ and $p^{(t_{k})}_{\theta}({\mathbf{x}})$ in marginal distribution level, which avoids the approximated error of ODE.

To further ensure the mode cover of the proposed model, we can add consistency loss to our objective as regularization, which constructs the following loss:

$$ $\displaystyle\min_{\theta}\max_{\phi}\mathbb{E}_{t}\big{\{}\mathbb{E}_{p_{ \theta}^{(t_{k})}({\mathbf{x}})}[\log D_{\phi}({\mathbf{x}},t)-\mathbb{E}_{p_{ \theta}^{(t)}({\mathbf{x}})}\log D_{\phi}({\mathbf{x}},t)]$ (4) $\displaystyle+\mathbb{E}_{q({\mathbf{x}})q({\mathbf{x}}_{t_{k}}|{\mathbf{x}})q ({\mathbf{x}}_{t}|{\mathbf{x}}_{t_{k}},{\mathbf{x}})}[\lambda(t)||G_{\theta}({ \mathbf{x}}_{t},t)-{\mathbf{x}}||_{2}^{2}+\lambda^{\mathrm{con}}_{t}||G_{ \theta}({\mathbf{x}}_{t},t)-\texttt{sg}(G_{\theta}({\mathbf{x}}_{t_{k}},t_{k}) )||_{2}^{2}]\big{\}},$ $$

where $\lambda^{\mathrm{con}}_{t}$ and $\lambda(t)$ are pre-defined hyper-parameters.

###### Proposition 1

## 4 Try It On CIFAR-10 Before Scaling Up For Saving Money!

In this section, we evaluate the performance of the proposed YOSO on CIFAR-10 (Yu et al., ) to verify its effectiveness under both training from scratch and fine-tuning settings.

### 4.1 training strategies

Before starting training, we introduce some effective training strategies in following for taming YOSO.

Decoupled scheduler. We find that the optimal scheduler for performing consistency loss and adversarial loss are not identical. This is due to the cooperative adversarial loss not involving the approximated error regarding the timestep skips, enabling a substantial skip to maximize its efficacy. In contrast, the consistency loss is susceptible to approximated error from timestep skips, necessitating a more conservative skip to preserve its effectiveness. Hence, to better unleash the capability of each loss, we propose using decoupled schedulers to construct our final training objective:

$$ $\displaystyle\min_{\theta}\max_{\phi}\mathbb{E}_{t}\big{\{}\mathbb{E}_{p_{ \theta}^{(t_{k})}({\mathbf{x}})}[\log D_{\phi}({\mathbf{x}},t)-\mathbb{E}_{p_{ \theta}^{(t)}({\mathbf{x}})}\log D_{\phi}({\mathbf{x}},t)]$ (5) $\displaystyle+\mathbb{E}_{q({\mathbf{x}})q({\mathbf{x}}_{t_{m}}|{\mathbf{x}})q ({\mathbf{x}}_{t}|{\mathbf{x}}_{t_{m}},{\mathbf{x}})}[\lambda(t)||G_{\theta}({ \mathbf{x}}_{t},t)-{\mathbf{x}}||_{2}^{2}+\lambda^{\mathrm{con}}_{t}||G_{ \theta}({\mathbf{x}}_{t},t)-\texttt{sg}(G_{\theta}({\mathbf{x}}_{t_{m}},t_{m}) )||_{2}^{2}]\big{\}},$ $$

where $t_{k}=\mathrm{max}(t-k,0)$, $t_{m}=\mathrm{max}(t-m,0)$, and we let $k=250$ and $m=25$ in experiments.

**Table 1: Unconditional generation results on CIFAR-10.**
| Model | FID$\downarrow$ | NFE $\downarrow$ |
| --- | --- | --- |
| Distillation or Fine-tuning |  |  |
| YOSO | 1.81 | 1 |
| Tract (Berthelot et al., ) | 3.78 | 1 |
| Diff-Instruct (Luo et al., ) | 4.53 | 1 |
| DMD (Yin et al., ) | 2.66 | 1 |
| CTM (Kim et al., ) | 1.98 | 1 |
| SiD (Zhou et al., ) | 1.92 | 1 |
| Training from scratch |  |  |
| YOSO | 2.26 | 1 |
| DDGANs (Xiao et al., ) | 3.75 | 4 |
| CT (Song et al., ) | 8.70 | 1 |
| iCT (Song & Dhariwal, ) | 2.83 | 1 |
| DDPM (Ho et al., ) | 3.21 | 1000 |
| EDM (Karras et al., ) | 2.04 | 36 |
| DDIM (Song et al., ) | 4.67 | 50 |
| StyleGAN2 (Karras et al., ) | 2.92 | 1 |

Annealing strategy. Since our target is to obtain a powerful one-step denoised generation model, however, as the KL loss and consistency loss will enforce point-level matching, this may hurt performance in cases where the model capacity is insufficient. Therefore, we suggest reducing the weight of these two losses to zero as training progresses. Specifically, we let $\lambda=(1-\frac{\lfloor n/\frac{N}{K}\rfloor}{K-1})\lambda^{\prime}$ for decreasing the weight to zero with $K$ times, where $n$ is the current iterations and $N$ is the total training iterations.

### 4.2 Empirical Evaluation

Experiment Setting.
We use the EDM (Karras et al., ) architecture for both UNet and discriminator. For the evaluation metric, we choose the Fréchet inception distance (FID) (Heusel et al., ). We train the proposed model on the CIFAR-10 dataset (Yu et al., ). We only employ MSE to compute image distance.

Results. The quantitative results are shown in Tab. 1. We observe that our method provides state-of-the-art performance with only a one-step manner on both training from scratch and fine-tuning settings, outperforming previous existing accelerated DMs, and GANs. In particular, our YOSO in fine-tuning EDM (Karras et al., ) even delivers better performance compared to EDM itself with ODE sampling. This is because our YOSO performs fine-tuning instead of distillation, the optimal solution for our YOSO is data distribution instead of teacher distribution.

## 5 Towards One-Step Text-to-Image Synthesis

Since training a text-to-image model from scratch is quite expensive, we suggest using pre-trained text-to-image DMs as initialization with Self-Coopeartive Diffusion GANs. In this section, we introduce several principled designs for developing a generation model that enables one-step text-to-image synthesis based on pre-trained DMs.

### 5.1 Using pre-trained models for Training

Latent Perceptual Loss. Prior research (Hou et al., ; Hoshen et al., ; Song et al., ) has confirmed the effectiveness of perceptual loss in various domains. Notably, recent studies (Liu et al., ; Song et al., ) have found that the LPIPS loss (Zhang et al., ) is crucial for obtaining few-step DMs with high sample quality. However, a notable drawback is that the LPIPS loss is computed in the data space, which is expensive. In contrast, the popular SD operates in the latent space to reduce computational demands. Hence, using LPIPS loss in training latent DMs is pretty expensive, as it requires not only the computation of LPIPS loss in the data space but also an additional decoding operation. Realizing that the pre-trained SD can serve as an effective feature extractor (Xu et al., ), we suggest using pre-trained SD to perform latent perceptual loss. However, SD is a UNet, whose final layer predicts epsilon with dimensions identical to the data. Hence we propose using the bottleneck layer of the UNet for computing:

$$ $d({\mathbf{z}}_{\theta},{\mathbf{z}})=||\mathrm{HalfUNet}({\mathbf{z}}_{\theta },c,t=0)-\mathrm{HalfUNet}({\mathbf{z}},c,t=0)||_{2}^{2},$ (6) $$

where ${\mathbf{z}}$ is the latent encoded images by VAE and $c$ is the text feature. We note that the benefit of computing latent perceptual loss by SD is not only the computational efficiency but also the incorporation of text features, which are crucial in text-to-image tasks.

Latent Discriminator. Training GANs for the text-to-image on large-scale datasets faces serious challenges. Specifically, unlike unconditional generation, the discriminator for the text-to-image task should justify based on both image quality and text alignment. This challenge is more obvious during the initial stage of training. To address this issue, previous pure GANs (Kang et al., ) for text-to-image propose complex learning objectives and require expensive costs for training.
The learning of GANs has been shown to benefit from using a pre-trained network as the discriminator. As discussed above, the pre-trained SD has learned representative features. Hence, we suggest applying the pre-trained SD for constructing the latent discriminator. Similar to latent perceptual loss, we only use half UNet for the discriminator followed by a simple predict head. The advantages of the proposed strategy are twofold: 1) we use the informative pre-trained network as initialization; 2) the discriminator is defined over latent space, which is computationally efficient.
Unlike previous work (Sauer et al., ) that defines a discriminator over data space which required decode latent and backward from decoder, yielding expensive computational cost.
By applying the Latent Discriminator, we observe a stable training process with fast convergence.

### 5.2 Fixing the Noise Scheduler

A common issue in DMs is that the final corrupted samples are not pure noise.
For example, the noise scheduler used by the SD makes the corrupted samples at the final timestep as:
${\mathbf{x}}_{T}=0.068265\cdot{\mathbf{x}}_{0}+0.99767\cdot\epsilon$, where the terminal Signal-to-noise ratio (SNR) is $\tfrac{\bar{\alpha}_{T}}{1-\bar{\alpha}_{T}}=0.004682$, effectively creates a gap between training and inference.
Previous work (Lin et al., ) has only observed that this makes DMs unable to produce pure black or white images. However, we find that this issue yields serious problems in one-step generation. As shown in Fig. 2, there are notable artifacts in one-step generation if we directly sample noise from standard Gaussian. The reason may be that in multi-step generation, the gap can be gradually fixed in sampling, while one-step generation reflects the gap more directly. To address this issue, we provide two simple yet effective solutions.

Figure: (a) w/o IPI
Refer to caption: extracted/5941826/figs/noise_corgi.jpg

Informative Prior Initialization (IPI). The non-zero terminal SNR issue is similar to prior hole issues in VAEs (Klushyn et al., ; Bauer & Mnih, ; Kingma et al., ). Therefore, we can use informative prior instead of non-informative prior to effectively address this issue. For simplicity, we adopt a learnable Gaussian Distribution $\mathcal{N}(\bm{\mu},\sigma^{2}\mathbf{I})$, whose optimal formulation is given below: $\epsilon^{\prime\prime}=\bar{\alpha}_{T}\cdot(\mathbb{E}_{\mathbf{x}}{\mathbf{
x}}+\mathrm{Std}({\mathbf{x}})\times\epsilon^{\prime})+\sqrt{1-\bar{\alpha}_{T
}}\cdot\epsilon$,
where $\mathbb{E}_{\mathbf{x}}{\mathbf{x}}$ and $\mathrm{Std}({\mathbf{x}})$ can be efficiently estimated by finite samples, and $\epsilon^{\prime}$ follows standard Gaussian distribution.
As shown in Fig. 2, the artifacts in one-step generation are immediately removed after applying IPI. We note that the performance is achieved with minimal adjustment, enabling the possibility of developing one-step text-to-image generation by LoRA fine-tuning.

Figure: Figure 3: Predicting $\epsilon$ fails.
Refer to caption: extracted/5941826/figs/pixart_eps_pre.jpg

Quick Adaption to V-prediction and Zero Terminal SNR. The IPI suffers from numerical instability when terminal SNR is pretty low. As shown in Fig. 3, we fine-tune PixArt-$\alpha$ (Chen et al., ) whose terminal SNR is 4e-5 with introduced technique and $\epsilon$-prediction, failing in one-step generation.
Following (Lin et al., ), we suggest switching to v-prediction (Salimans & Ho, ) and zero terminal SNR. However, we found that direct transition convergences slowly (see Appendix F).
This is unacceptable in solving large-scale text-to-image with limited computational resources.
To address this, we propose a quick adapt-stage:

Adapt-stage-I switch to v-prediction : $L(\theta)=\lambda_{t}||v_{\theta}({\mathbf{x}}_{t},t)-v_{\phi}({\mathbf{x}}_{t
},t)||_{2}^{2},$
where
$v_{\phi}({\mathbf{x}}_{t},t)=\bar{\alpha}_{t}\epsilon_{\phi}({\mathbf{x}}_{t},
t)-\sqrt{1-\bar{\alpha}_{t}}{\mathbf{x}}_{\phi}^{t}$, ${\mathbf{x}}_{\phi}^{t}=\tfrac{{\mathbf{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}
\epsilon_{\phi}({\mathbf{x}}_{t},t)}{\bar{\alpha}_{t}}$
, $v_{\theta}(\cdot,\cdot)$ denotes the desired v-prediction model, and $\epsilon_{\phi}(\cdot,\cdot)$ denotes the frozen pre-trained model.

Adapt-stage-II switch to zero terminal SNR : $L(\theta)=\lambda_{t}||v_{\theta}({\mathbf{x}}_{t},t)-v_{\phi}({\mathbf{x}}_{t
}^{\prime},t)||_{2}^{2}.$
Note that in this stage, we only change the scheduler to zero terminal SNR for student model. This avoids numerical instability issues of $\epsilon$-prediction with zero terminal SNR. We note that the zero terminal SNR scheduler has lower SNR than the original schedule at each timestep, formulating an effective distillation objective.

We note that this adapt stage converges rapidly, typically requiring only 1k iterations to initialize training YOSO. This enables a quick adaption to v-prediction and zero terminal SNR from pre-trained $\epsilon$-prediction DMs, thereby comprehensively mitigating the non-zero terminal SNR issue in the noise scheduler.

## 6 Experiments

In this section, we evaluate the performance of the proposed YOSO. Sec. 6.1 examines YOSO in the context of text-to-image generation by fine-tuning pre-trained PixArt-$\alpha$ (Chen et al., ) and Stable Diffusion (Rombach et al., ). In Sec. 6.3, we show that the YOSO can be used for several downstream applications. Moreover, we conduct ablation studies in Sec. 6.2 to highlight the effectiveness of our proposed algorithm and proposed training techniques.

### 6.1 Text-to-Image Generation

Experiment Setting.
We initialize the GAN generator by pre-trained PixArt-$\alpha$ which is a diffusion transformer with 0.6B parameters. For the GAN discriminator, we construct the latent discriminator by pre-trained SD 1.5 using the proposed construction introduced in Sec. 5.1. We switch the pre-trained PixArt-$\alpha$ to v-prediction by the proposed technique introduced in Sec. 5.1, followed by training on the JourneyDB dataset (Pan et al., ) with resizing as 512 resolution. We apply a batch-size of 256 and a constant learning rate of 2e-5 during training. We observe the training convergence fast, requiring only 30k iterations and around 10 A800 days to be trained. We apply a batch-size of 256 and a constant learning rate of 2e-5 during training. We also conduct experiments on fine-tuning SD 1.5 via LoRA to show the effectiveness of the proposed YOSO, with a batch-size of 128 and a constant learning rate of 2e-5.

Figure: Figure 4: Qualitative comparisons of YOSO against competing methods.
Refer to caption: extracted/5941826/figs/com_new.png

Figure: Figure 5: Qualitative comparisons of YOSO against competing methods. It can be seen that both DMD and SD-Turbo suffer from mode collapse, while YOSO achieves better sample quality and mode cover. The prompt is ”A cute dinosaur, cartoon style, white background”.
Refer to caption: extracted/5941826/figs/mode-new.png

Evaluation. We employ Aesthetic Score (AeS) (Schuhmann et al., ) to evaluate image quality and adopt the Human Preference Score (HPS) v2.1 (Wu et al., ) to evaluate the image-text alignment and human preference. The AeS is computed by an aesthetic score predictor trained on LAION (Schuhmann et al., ) datasets, without considering the image-text alignment. HPS is trained to predict human preference given image-text pairs, considering the image-text alignment and human aesthetic.
Additionally, we include ImageReward score (Xu et al., ), and
CLIP score (Hessel et al., ) to provide a more comprehensive evaluation of the model performance. We mainly compare our model against the open-source state-of-the-art (SOTA) models, e.g., SD-Turbo (Sauer et al., ), PixArt-DMD (Chen et al., ; Yin et al., ), and Hyper-SD (a concurrent work) (Ren et al., ).

**Table 2: Comparison of machine metrics on text-to-to-image generation across state-of-the-art methods. HFL denotes human feedback learning which might hack the machine metrics.**
| Model | Backbone | HFL | Steps | HPS$\uparrow$ | AeS$\uparrow$ | IR$\uparrow$ | CS$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Base Model | SD 2.1 | No | 25 | 27.28 | 5.66 | 0.36 | 33.46 |
| SD Turbo (Sauer et al., ) | SD 2.1 | No | 1 | 27.06 | 5.31 | 0.40 | 32.21 |
| Base Model | SD 1.5 | No | 25 | 24.72 | 5.49 | 0.20 | 31.88 |
| InstaFlow (Liu et al., ) | SD 1.5 | No | 1 | 26.18 | 5.27 | -0.22 | 30.04 |
| Hyper-SD-LoRA (Ren et al., ) | SD 1.5 | Yes | 1 | 28.01 | 5.79 | 0.29 | 30.87 |
| YOSO-LoRA (Ours) | SD 1.5 | No | 1 | 28.33 | 5.97 | 0.43 | 31.33 |
| LCM-LoRA (Luo et al., ) | SD 1.5 | No | 4 | 22.77 | 5.66 | -0.37 | 30.36 |
| PeRFlow (Yan et al., ) | SD 1.5 | No | 4 | 22.43 | 5.64 | -0.35 | 30.77 |
| TCD-LoRA (Zheng et al., ) | SD 1.5 | No | 4 | 22.24 | 5.45 | -0.15 | 30.62 |
| Hyper-SD-LoRA (Ren et al., ) | SD 1.5 | Yes | 4 | 30.24 | 5.55 | 0.53 | 31.07 |
| YOSO-LoRA (Ours) | SD 1.5 | No | 4 | 30.50 | 6.05 | 0.56 | 31.52 |
| Base Model | PixArt-$\alpha$ | No | 25 | 31.07 | 6.12 | 1.05 | 33.81 |
| DMD (Yin et al., ) | PixArt-$\alpha$ | No | 1 | 29.78 | 6.02 | 0.97 | 32.90 |
| YOSO (Ours) | PixArt-$\alpha$ | No | 1 | 30.52 | 6.19 | 1.02 | 32.95 |

Quantitative Results. The quantitative results are presented in Tab. 2.
As shown in Tab. 2, YOSO clearly beat previous state-of-the-art methods across all metrics (including HPS, AeS, Image Reward score, and Clip Score). It is important to highlight that for the SD 1.5 backbone, we only use LoRA for fine-tuning SD 1.5, where only less than 10% parameters are tuned.
Moroever, our method without human feedback learning (HFL) even outperforms Hyper-SD with HFL which potentially hacks the machine metric.

Qualitative Comparison.
To provide more comprehensive insight in understanding the performance of our YOSO, we further provide the qualitative comparison in Fig. 4.
The comparison shows that: YOSO clearly beats existing baselines in terms of both image quality and prompt alignment, which is achieved by LoRA-finetuning.
Specifically, the samples by SD-Turbo are blurry to some extent, and the samples by Hyper-SD are over-saturated and with notable artifacts.
Moreover, the advantage of YOSO in text-image alignment is significant.
For example, in the last row of Fig. 4 related to a cow-people worker is coding, previous methods including SD 1.5 itself cannot produce samples following the prompt, while YOSO precisely follows the prompt to generate a half cow half people worker.
Additionally, it is important to highlight that both SD-Turbo and DMD seem to have a serious issue in mode collapse. As shown in Fig. 5, both SD-Turbo and DMD generate samples that are extremely close to each other.
Overall, based on quantitative and qualitative results, we conclude that our YOSO is better in sample quality, prompt alignment, and mode cover compared with the SOTA one-step text-to-image models.

#### 6.1.1 Zero-Shot One-Step 1024 Resolution Generation

Due to the computational resource limitation, we trained YOSO on 512 resolution. However, there is a 1024-resolution version of PixArt-$\alpha$, obtained by continuing training on the 512-resolution version of PixArt-$\alpha$. Motivated by the effectiveness of the LoRA combination (Luo et al., ), we suggest constructing a similar combination as follows:

$$ $\mathbf{W}_{\mathrm{YOSO_{1024}}}=\mathbf{W}_{\mathrm{PixArt_{512}}}+\alpha \cdot(\mathbf{W}_{\mathrm{PixArt_{1024}}}-\mathbf{W}_{\mathrm{PixArt_{512}}})+ \beta\cdot(\mathbf{W}_{\mathrm{YOSO_{512}}}-\mathbf{W}_{\mathrm{PixArt_{512}}}),$ (7) $$

where $\alpha,\beta\in(0,1]$ and $\mathbf{W}$ means the model weight. We let $\alpha=\beta=1$ in our experiments. As shown in Fig. 1(b), the YOSO constructed in this way can generate high-quality images with 1024 resolution. Note that YOSO even changes the predicted objective to v-prediction. The impressive performance indicates the robust generalization ability of our proposed YOSO.

Figure: Figure 6: Qualitative comparison against competing methods and applications in down-stream tasks.
Refer to caption: extracted/5941826/figs/lora_app.png

Figure: Table 3: Ablation study on CIFAR-10 with smaller backbone.

### 6.2 Ablation Studies

We provide additional insight into the effectiveness of the proposed YOSO by performing ablation studies on some potential variants on CIFAR-10 and on text-to-image generation. We call YOSO without decoupled scheduler and annealing strategy as YOSO-base. See more details in Appendix.

Effect of Consistency Loss. For the effect of consistency loss in YOSO, our results suggest that it can improve the image quality, as indicated by the FID scores in Sec. 6.1.1. Notably, the removal of the consistency loss does not lead to a substantial decline in generation performance, which indicates the effectiveness of our proposed adversarial formulation.

Effect of Latent Perceptual Loss. To investigate the effect of latent perceptual loss proposed in Sec. 5.1, we conduct experiments on text-to-image tasks by using it to compute the consistency loss. As shown in Sec. 6.1.1, the proposed latent perceptual loss clearly improves the HPS from 15.22 to 20.08 and AeS from 5.23 to 5.45, highlighting its effectiveness.

Effect of Decoupled Scheduler. We evaluate the impact of the proposed decoupled scheduler by adding it to YOSO-base. As evidenced by Sec. 6.1.1 and Sec. 6.1.1, the incorporation of the decoupled scheduler significantly enhances sample quality, as reflected by improvements in FID, HPS, and AeS metrics. Notably, the FID score on CIFAR-10 decreases from 3.82 to 3.30 and the HPS in text-to-image tasks is improved from 26.42 to 27.10, demonstrating the superior efficacy of the decoupled scheduler.

Effect of Annealing Strategy. We also assess the effect of the proposed annealing strategy by removing it from the final formulated YOSO. As illustrated in Sec. 6.1.1 and Sec. 6.1.1, the removal of the annealing strategy degrades FID from 3.05 to 3.30 and reduces HPS from 28.33 to 27.10. Such results demonstrate the effectiveness of the annealing strategy in training YOSO.

Figure: Figure 7: The discriminator loss curve.
Refer to caption: extracted/5941826/figs/loss_ablation_new.png

Adversarial Divergence $D_{\mathrm{adv}}(p_{d}||p_{\theta}^{t})$. The key design of our proposed YOSO is smoothing the adversarial divergence by using self-generated data instead of real data as ground truth to perform adversarial divergence.We evaluate the variant of using $D_{\mathrm{adv}}(p_{d}||p_{\theta}^{t})$ as adversarial divergence. As shown in Sec. 6.1.1, the performance of the variant is significantly worse than YOSO-Base, i.e., FID degrades from 3.82 to 10.85. Moreover, besides sample quality, we find that the training of our proposed YOSO is much more stable than the variant. As shown in Fig. 7, the discriminator loss of the variant is not stable and is much smaller than ours. We also compare the variant in the text-to-image task. The results are given in Sec. 6.1.1, from which we can see that replacing our cooperative adversarial loss with $D_{\mathrm{adv}}(p_{d}||p_{\theta}^{t})$ yields serious performance deterioration, i.e., HPS decreases from 26.42 to 24.88 and AeS decreases from 5.85 to 5.69.
This indicates that our approach can effectively smooth the adversarial divergence by narrowing the gap between fake and real distribution, which makes discrimination harder and hence increases discriminator loss. Furthermore, we observe mode collapse in the training of this variant. Note that a similar case also occurs in SD-Turbo, a strong one-step text-to-image model, which performs adversarial divergence directly against real data (See Fig. 4). Overall, these results indicate the superiority of the proposed YOSO.

Comparison to UFOGen. Similar to us, UFOGen does not directly perform adversarial divergence against real data either. Instead, they employ adversarial divergence over corrupted data (i.e., $D_{\mathrm{adv}}(q({\mathbf{x}}_{t-1})||\int q({\mathbf{x}}_{t-1}|{\mathbf{x}}
=G_{\theta}({\mathbf{x}}_{t},t))q({\mathbf{x}}_{t})d{\mathbf{x}}_{t})$).
However, we argue that such a formulation is less effective in constructing one-step generation model as the adversarial divergence it employs only indirectly aligns with one-step generation at the level of clean data. We implement UFOGen on CIFAR-10 on our own. Note that we also perform consistency loss in the UFOGen variant to ensure a fair comparison, as the consistency loss has been found useful in YOSO. As shown in Sec. 6.1.1, UFOGen only obtains 45.15 FID on CIFAR-10, significantly worse than YOSO with 3.82 FID. This further underscores the superiority of our proposed YOSO. Through comparison with different adversarial divergence formulations, we find that our approach not only stabilizes training by smoothing the adversarial divergence but also facilitates effective learning for one-step generation.

### 6.3 Application

One promising and attractive property of DMs is that they can be used in multiple downstream tasks. In this section, we show the capability of our proposed YOSO in various downstream tasks, keeping the unique advantages of DMs: 1) Image-to-image Editing: As shown in Fig. 6, our YOSO-LoRA is capable of performing high-quality image-to-image editing (Meng et al., ) in one step; 2) Compatibility with ControlNet and Different Base Models: As shown in Fig. 6, our YOSO-LoRA is compatible with ControlNet (Zhang et al., ), following the condition well. Our YOSO-LoRA is also compatible with different base models (e.g., Dreamshaper and Toonyou) fine-tuned from SD 1.5, preserving their style well.
We find that when applying YOSO-LoRA to new base models, it fails to produce reasonable samples in one step. This may be due to the capability of LoRA and the distribution shift of training data.

## 7 Related Works

Few-Step Text-to-Image Generation.
LCM (Luo et al., ) adapts consistency distillation (Song et al., ) for stable diffusion (SD) and enables image generation in 4 steps with acceptable quality.
InstaFlow (Liu et al., ) adapts rectified flows (Liu et al., ; Liu, ) for SD, facilitating the generation of text-to-image in a mere single step. Despite this, the fidelity of the generated images is still poor.
Some works (Yin et al., ) adapts variation score distillation (Wang et al., ) and diff-instruct (Luo et al., ) for SD, which requires training an additional DMs for the generator distribution, akin to the discriminator in GANs. We found models trained in this way seem to have serious mode collapse issues (see Fig. 5).
Recently, combining DMs and GANs for one-step text-to-image generation has been explored.
UFOGen (Xu et al., ) extends DDGANs (Xiao et al., ) and SSIDMs (Xu et al., ) to SD by modifying the computation of reconstruction loss from corrupted samples to clean samples.
However, it still performs adversarial matching using corrupted samples.
ADD (Sauer et al., ) introduces a one-step text-to-image generation based on SD. It follows earlier research (Sauer et al., ) by employing a pre-trained image encoder, DINOv2 (Oquab et al., ), as the backbone of the discriminator to accelerate the training.
However, the discriminator design moves the training from latent space to pixel space, which substantially heightens computational demands. Moreover, they directly perform adversarial matches at clean real data, increasing the challenge of training.
This requires the need for a better but expensive discriminator design and expensive R1 regularization to stabilize the training.
SDXL-Lightning (Lin et al., ) combines progressive distillation and adversarial training in a progressive framework.
Recently, a concurrent work Hyper-SD (Ren et al., ) combines consistency distillation and adversarial training in a progressive framework, while our work can be trained end-to-end.
However, their work performs adversarial point-level matching via consistency loss, and they still perform adversarial training via injecting noise. This results in less effective one-step learning, thus they rely on human feedback learning for achieving better performance.
In contrast, we perform the adversarial training at the distribution level by replacing real data with self-generated data to smooth the adversarial divergence, which not only can stabilize the training without injecting noise but also form effective one-step generation learning.
Moreover, compared with above mentioned diffusion-GAN hybrid models, our approaches can be trained from scratch to perform one-step generation, which is not demonstrated by them.
Additionally, we extend our method not only to SD but also to PixArt-$\alpha$ (Chen et al., ) which is based on diffusion transformer (Peebles & Xie, ). This demonstrates the wide application of our proposed YOSO.

## 8 Conclusion

In summary, we present YOSO, a new generative model enabling high-quality one-step generation. Our novel designs combine the diffusion process and GANs, enabling not only one-step generation training from scratch but also one-step text-to-image generation fine-tuning from pre-trained models. We will release our model to advance the research of text-to-image synthesis.

## Appendix A Broader Impacts

This work presents YOSO, a method that accelerates multi-step large-scale text-to-image diffusion models into a one-step generator. On the positive side, while this is academic research, we believe the proposed YOSO can be widely applied in industry, and its high efficiency can lead to energy savings and environmental benefits. However, when these rapid generation models are manipulated by malicious actors, they can also simplify and accelerate the creation of harmful information. Although our work focuses on scientific research, we will take actions to reduce the harmful information, such as filtering out harmful content in the dataset.

## Appendix B Limitation

Our model, like most text-to-image diffusion models, may exhibit shortcomings in terms of fairness, as well as in handling specific details and accurately controlling the number of targets. We plan to explore these unresolved issues in the generation field in our future work, in order to enhance the model’s capabilities in text generation, fairness, detail control, and quantitative control.

## Appendix C Proof

Proof: The cooperative adversaril loss is defined as: $\mathbb{E}_{t}(D_{\mathrm{adv}}(p^{(t_{k})}_{\theta}(\texttt{sg}({\mathbf{x}})
)||p^{(t)}_{\theta}({\mathbf{x}}))$.
Since the divergence is non-negative, and the $p^{(0)}_{\theta}$ is defined as $p_{d}$, there exist a global optimal solution $p_{\theta}^{(T)}({\mathbf{x}})=p_{\theta}^{(T-1)}({\mathbf{x}})=\cdots=p_{d}({
\mathbf{x}})$ such that $\mathbb{E}_{t}(D_{\mathrm{adv}}(p^{(t_{k})}_{\theta}(\texttt{sg}({\mathbf{x}})
)||p^{(t)}_{\theta}({\mathbf{x}}))$ = 0. This completes the proof.

## Appendix D Training Algorithm

Figure: Algorithm 1 YOSO training from scratch.

We present the algorithm for training YOSO from scratch in Algorithm 1.

## Appendix E Additional Related Work

Text-to-image Diffusion Models. Since Diffusion models have shown a stable training process and are well-suited for scaling up generative models, numerous works have been proposed to extend DMs for text-to-image generation (Ramesh et al., ; Balaji et al., ; Rombach et al., ; Xue et al., ). Latent DMs (Rombach et al., ; Podell et al., ) are widely adopted for high-resolution image and video generation due to their computational efficiency.

## Appendix F The Effect of Quick-Adaption

Figure: (a) w/o quick adaption.
Refer to caption: extracted/5941826/figs/wo_quick_adaption.png

We present a qualitative comparison between YOSO w/o quick adaption and YOSO w/ quick adaption in Fig. 8. Note that the YOSO and variants here are trained with only 5k iterations. And for YOSO w/ quick adaption, we incorporate the 1k iterations used for quick adaptation into the total 5k training iterations to ensure a fair comparison. As shown in Fig. 8, YOSO w/ quick adaption demonstrates significantly better image quality compared to YOSO w/o quick adaption. The generated images of YOSO w/o quick adaption exhibit over-smoothing and severe artifacts. This indicates that the convergence of direct transition to v-prediction and zero terminal SNR is relatively slower.
Additionally, it is worth highlighting that although there are only 5k training iterations, the one-step generation quality of YOSO is already reasonable, which demonstrates the effectiveness and quick convergence of the proposed quick adaption stage and YOSO.

## Appendix G Experiment Setting Details

### G.1 Unconditional Generation Experiments

For the generator, we use the Adam optimizer with $\beta_{1}$ = 0.9 and $\beta_{2}$ = 0.999; for the discriminator, we use the Adam optimizer with $\beta_{1}$ = 0. and $\beta_{2}$ = 0.999. We adopt a constant learning rate of 2e-4 for both the discriminator and the generator in training from scratch. We adopt a constant learning rate of 2e-5 for both the discriminator and the generator in fine-tuning. We apply EMA with a coefficient of 0.9999 for the generator. We let the $\lambda_{t}=\text{SNR}(t)$ and $\lambda_{t}^{\text{con}}=\frac{1}{\frac{1}{\text{SNR}(t)}-\frac{1}{\text{SNR}(
t-1)}}$.

### G.2 Text-To-Image Experiments

For the generator, we use the AdamW optimizer with $\beta_{1}$ = 0.9 and $\beta_{2}$ = 0.999; for the discriminator, we use the AdamW optimizer with $\beta_{1}$ = 0. and $\beta_{2}$ = 0.999. We adopt a constant learning rate of 2e-5 for both the discriminator and the generator. We apply gradient norm clipping with a value of 1.0 for the generator only. We use batch size 256. For full fine-tuning, we apply EMA with a coefficient of 0.9999 for the generator. For LoRA fine-tuning, we apply EMA with a coefficient of 0.999 for the generator. Generally, the training is done within 30k iterations for full fine-tuning, while within 5k iterations for LoRA fine-tuning. We let the $\lambda_{t}=\text{SNR}(t)$ and $\lambda_{t}^{\text{con}}=\frac{1}{\frac{1}{\text{SNR}(t)}-\frac{1}{\text{SNR}(
t-m)}}$.

For full fine-tuning, we train YOSO on the JourneyDB dataset (Pan et al., ), by resizing to 512 resolution. And we only use the square image. For LoRA fine-tuning, we use an internally collected dataset and the caption of the JourneyDB dataset (Pan et al., ) to generate data for training, we only generate one image for one caption. For the evaluation, we evaluate the HPS score on its benchmark, and we evaluate other metrics based on COCO-5k (Lin et al., ) datasets.

### G.3 Ablation Study

For the ablation study on CIFAR-10, we use the same UNet and discriminator architecture as used in DDGANs (Xiao et al., ). For the generator, we use the Adam optimizer with $\beta_{1}$ = 0.9 and $\beta_{2}$ = 0.999; for the discriminator, we use the Adam optimizer with $\beta_{1}$ = 0. and $\beta_{2}$ = 0.999. We adopt a constant learning rate of 2e-4 for both the discriminator and the generator. We apply EMA with a coefficient of 0.9999 for the generator. We let the $\lambda_{t}=\text{SNR}(t)$ and $\lambda_{t}^{\text{con}}=\frac{1}{\frac{1}{\text{SNR}(t)}-\frac{1}{\text{SNR}(
t-1)}}$.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: Questions are embedded in the weakness section.
soundness: 4
strengths: 1. The proposed self-cooperative technique to train a one-step sampler seems novel in the diffusion literature, and the author proved it to be very effective. The fact that the less noisy (teacher) samples guide more noisy (student) samples during training is quite interesting.

2. It is interesting to see the connection between the self-cooperative objective and consistency models.

3. The experimental results are quite strong. They outperform other one-step baseline models on CIFAR-10 and also show improvement in the large-scale text-to-image setting.

4. The mathematical analysis is clear and robust, which supports each of its contributions. The derivations for Self-Cooperative Learning and the teacher-student dynamic seems well-structured.
summary: This paper presents YOSO (You Only Sample Once), a novel approach for high-quality, one-step text-to-image generation that enhances training stability and efficiency by combining elements from diffusion and GAN models. YOSO’s design includes Self-Cooperative Learning, where less noisy “teacher” samples guide more noisy “student” samples to stabilize training, along with a Decoupled Scheduler that applies separate optimization schedules to Adversarial Loss and Consistency Loss, improving sample quality. Additionally, Informative Prior Initialization (IPI) provides a more data-informed starting point for faster convergence and reduced artifacts, while Quick Adaptation facilitates smooth adjustments to v-prediction and zero terminal SNR, essential for stable, one-step sampling. Unlike prior models that require multiple steps, YOSO achieves single-step synthesis with refined adversarial learning, using teacher-student distributions within the model to mitigate instability and mode collapse risks. This efficient and stable framework positions YOSO as a significant advancement for rapid, high-fidelity text-to-image generation.
weaknesses: 1. Lack of the ablation study on training techniques. The paper introduces training techniques like Informative Prior Initialization (IPI) and v-prediction, which are valuable training methods that could also benefit other models. However, the lack of an ablation study on these components makes it difficult to assess their specific impact and to directly compare YOSO's performance with other models that could potentially leverage these techniques.

2. Hyperparameter sensitivity. The timestep difference between the teacher and student latents, a key factor in the Decoupled Scheduler, is a hyperparameter that directly impacts training stability and eventual performance. Does it need to be tuned for different datasets? Could you show how robust the method is to this hyperparameter? 

[minor comments]
1. Table 2 caption: text-to-to -> text-to
2. Eq (3): remove ‘x’
3. In Eq (3): why did you use different lambda for different t?

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - Why were only SD-LoRA and PixArt-Full experiments conducted in the experimental setup, without including SD-Full and PixArt-LoRA related experiments?

- I am curious about how the zero-shot one-step 1024 resolution is effective. As far as I know, 1024 has a different ar_size compared to 512, and the posemb scale is also different. Wouldn't these factors significantly impact the weight distribution of the 1024 model?

- Without retraining, will increasing the sampling steps, similar to LCM, improve the results? If further enhancements are desired for better generated image quality, what would be the subsequent optimization directions based on the proposed approach?
soundness: 3
strengths: - In terms of qualitative metrics, YOSO demonstrates certain advantages across various tasks and metrics, and it is capable of transferring to higher resolutions, image editing, and conditional generation tasks.

- The simple and effective training strategies proposed during practical implementation have certain reference significance for generative tasks.

- The paper shows many analyses and easy to follow.
summary: This paper combines diffusion models and GANs to propose the YOSO framework for one-step image generation. It employs various training strategies to enhance training stability and has been validated on both unconditional generation and text-to-image tasks.
weaknesses: - The presentation of the visualization results in this paper is somewhat insufficient, including a comparison with YOSO-PixArt-alpha and other Few-Step models, such as PixArt-delta, and it might be worth attempting to compare with the original models as well.

- The clarity of the narrative in this paper needs improvement; currently, the narrative logic resembles an experimental report. When encountering issues in a new task (unconditional and text-to-image generation / finetuning) or a new model (U-Net or DiT based), a specific strategy is proposed to address the problem. This approach may not be conducive to the coherence expected in a complete paper.

- Regarding writing:
  - The abbreviation "NEF" is not explained.
  - Should the steps for Hyper-SD-LoRA and YOSO-LoRA in Table 2 be equal to 1?
  - Figure 5 does not specify the comparative configuration for model inference—does it keep the prompt unchanged while changing the same seed?
  - The reference formatting in the paper should be kept consistent as possible.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: See Weaknesses.
soundness: 2
strengths: 1. The article is well written and easy to understand.
2. The visualization results look very good.
summary: This paper introduces YOSO, a novel generative model designed for rapid, scalable, and high-fidelity one-step image synthesis with high training stability and mode coverage. Specifically, the paper smooths the adversarial divergence by the denoising generator itself, performing self-cooperative learning. This method can serve as a one-step generation model training from scratch with competitive performance. Moreover, the authors extend YOSO to one-step text-to-image generation based on pre-trained models by several effective training techniques (i.e., latent perceptual loss and latent discriminator for efficient training along with the latent DMs; the informative prior initialization (IPI), and the quick adaption stage for fixing the flawed noise scheduler). Experimental results show that YOSO achieves state-of-the-art one-step generation performance even with Low-Rank Adaptation (LoRA) fine-tuning.
weaknesses: 1. There is a lack of important references in the Background, such as [1,2,3], which are all about combining Diffusion models and GANs.  The differences between the proposed method and these methods [1,2,3] need to be emphasized and discussed.
2. In the experimental part, it is also necessary to compare with these methods [1,2,3].
3. There is no comparison with the current SOTA methods in terms of model complexity (such as the number of network parameters and training inference time).
4. In the TEXT-TO-IMAGE GENERATION experiment, there is no comparison with the GAN-based methods, such as [4]. It is also worth discussing whether the proposed method is better or more efficient than the GAN-based methods.

[1]Wang, Zhendong, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. "Diffusion-gan: Training gans with diffusion." arXiv preprint arXiv:2206.02262 (2022).
[2]Kang, Minguk, Richard Zhang, Connelly Barnes, Sylvain Paris, Suha Kwak, Jaesik Park, Eli Shechtman, Jun-Yan Zhu, and Taesung Park. "Distilling Diffusion Models into Conditional GANs." ECCV (2024).
[3]Xu, Yanwu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. "Ufogen: You forward once large scale text-to-image generation via diffusion gans." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8196-8206. 2024.
[4]Kang, Minguk, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. "Scaling up gans for text-to-image synthesis." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10124-10134. 2023.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please refer to the weakness section
soundness: 3
strengths: 1. Creative Approach to Stability in One-Step Synthesis: YOSO’s self-cooperative learning framework is an inventive attempt to stabilize adversarial training by smoothing divergence via self-generated data, a strategy distinct from standard GAN-DM hybrids. This approach may address some instability challenges faced by prior one-step latent diffusion models.
2. Efficient Resource Utilization: YOSO’s compatibility with LoRA allows it to produce high-resolution images without the significant computational burden typically associated with GAN-based models, making it adaptable for resource-constrained settings.
summary: This paper, "You Only Sample Once: Taming One-Step Text-to-Image Synthesis by Self-Cooperative Diffusion GANs (YOSO)", proposes a novel generative method that merges diffusion models (DMs) with GANs to perform efficient, high-quality one-step text-to-image synthesis. YOSO leverages a self-cooperative learning framework, where a generator smooths adversarial divergence by referencing self-generated denoised samples, combined with a latent perceptual loss and a decoupled scheduler to stabilize training. The paper positions YOSO as an efficient alternative to multi-step latent diffusion methods (LDMs) with discriminator-based losses, aiming to achieve comparable quality while reducing computational overhead. Experimental results demonstrate competitive performance with state-of-the-art methods, though the model would benefit from clearer differentiation from similar single-step latent diffusion approaches.
weaknesses: 1. Clarity of Key Contribution: This paper lacks clarity on whether YOSO is a new model architecture that can be trained from scratch or a step distillation methodology that should be applied on pre-trained models. Further elaboration, possibly with additional diagrams, would make the paper more accessible and clear.

2. Experimental Scope and Robustness: This paper introduces YOSO both as a new model and a novel distillation techniques for pre-trained models. However, the former part was only evaluated on the CIFAR-10 dataset, additional benchmarks are necessary to substantiate its effectiveness across more diverse and complex datasets.

2. Novelty in Methodology: Besides self-cooperative learning, the adoption of decoupled scheduling, perceptual and consistency loss isn't novel to the field, and while the rationale is sound, these sections could benefit from more detailed comparison with previous works and how it is different from them.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The authors did not provide information about the dataset used for the test metrics in Table 2.  Although coco2014 is mentioned in the appendix, I am not certain if it is the dataset used for these metrics.
2. The ablation study is unconvincing, as the authors used CIFAR-10 for validation. However, CIFAR-10 has a low resolution and only consists of 10 simple categories, lacking the richness of text prompts. It would be better to conduct the ablation study on the COCO 2014 and COCO 2017 datasets, similar to existing one-step text-to-image generative methods (e.g., InstaFlow, SD-Turbo, SwiftBrush).
soundness: 2
strengths: 1. Applying the Latent Discriminator leads to a stable training process with fast convergence.
2. The paper tackles a relevant application (i.e., ControlNet) with one-step.
summary: The paper proposes a novel one-step generative model named YOSO, which offers higher training stability and model coverage compared to existing methods. This approach aims to achieve high-quality generation while maintaining performance metrics in a single step. Experiments are conducted on CIFAR-10 using HPS and AeS evaluation metrics.
weaknesses: 1. As mentioned in the paper, "It is hard to extend GANs on large-scale datasets due to training challenges," yet there are existing works that successfully train GANs with large-scale datasets (e.g., GigaGAN with 2.7 billion training image-prompt pairs, Diffusion2GAN with 12 million training image-prompt pairs). It is important to compare these methods. While GigaGAN and Diffusion2GAN do not provide source code, the authors can still conduct comparisons using the COCO 2014 dataset, as GigaGAN provides inference results for this dataset.

GigaGAN: Scaling up GANs for Text-to-Image Synthesis (CVPR 2023)

Diffusion2GAN: Distilling Diffusion Models into Conditional GANs.

2. Why use the bottleneck layer of the UNet to compute the latent perceptual loss? Some works (e.g., DIFT, PnP, FasterDiffusion) suggest that encoder features are more important than both the encoder and bottleneck features.

DIFT: Emergent Correspondence from Image Diffusion (NeurIPS'23)

PnP: Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation (CVPR 2023)

FasterDiffusion: Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS'24)

3. It’s better to provide experiments comparing with exiting SOTA one-step generative methods (e.g., SwiftBrush) to prove that YOSO enhances the inference efficiency and high-quality generation.

SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation (CVPR'24)

4. The effectiveness of IPI should be validated on datasets like COCO 2014 using FID and CLIPscore to avoid cherry-picking. FID and CLIPscore is wildy used in one-step generation methods like SD-Turbo, InstaFlow, SwiftBrush.

Meta-review: The paper introduces a framework for efficient and high-quality one-step image generation by combining diffusion models with GANs. The authors propose several techniques, including self-cooperative learning, a decoupled scheduler, and informative prior initialization. The paper demonstrates promising results on multiple benchmark datasets.

Following the authors-reviewers discussion, two reviewers increased their ratings. However, other reviewers leaning toward rejection with concerns about: 1) high-resolution generation on certain datasets, 2) inferior FID results, and 3) insufficient ablations to understand the optimal layers/features used for optimization. Subsequently, the authors provided additional responses addressing these three concerns.

Given the promising results, the paper is recommended for acceptance. However, the authors should include the discussions with reviewers in the final version.

Decision: Accept (Poster)