You Only Sample Once: Taming One-Step Text-to-Image Synthesis by Self-Cooperative Diffusion GANs
Authors: Yihong Luo
Xiaolong Chen
Xinghua Qu
Tianyang Hu
Jing Tang
Venue: ICLR 2025 Poster (2025)
Keywords: One-step text-to-image generation; Diffusion Models; Generative Adversarial Networks
Primary area: generative models

Abstract: Recently, some works have tried to combine diffusion and Generative Adversarial Networks (GANs) to alleviate the computational cost of the iterative denoising inference in Diffusion Models (DMs). 
However, existing works in this line suffer from either training instability and mode collapse or subpar one-step generation learning efficiency. 
To address these issues, we introduce YOSO, a novel generative model designed for rapid, scalable, and high-fidelity one-step image synthesis with high training stability and mode coverage. 
Specifically, we smooth the adversarial divergence by the denoising generator itself, performing self-cooperative learning. We show that our method can serve as a one-step generation model training from scratch with competitive performance. 
Moreover, we extend our YOSO to one-step text-to-image generation based on pre-trained models by several effective training techniques (i.e., latent perceptual loss and latent discriminator for efficient training along with the latent DMs; the informative prior initialization (IPI), and the quick adaption stage for fixing the flawed noise scheduler). Experimental results show that YOSO achieves the state-of-the-art one-step generation performance even with Low-Rank Adaptation (LoRA) fine-tuning.
In particular, we show that the YOSO-PixArt-$\alpha$ can generate images in one step trained on 512 resolution, with the capability of adapting to 1024 resolution without extra explicit training, requiring only \textasciitilde10 A800 days for fine-tuning. Our code is available at:  [https://github.com/Luo-Yihong/YOSO](https://github.com/Luo-Yihong/YOSO)

Published as a conference paper at ICLR 2025
YOU ONLY SAMPLE ONCE: TAMING ONE-STEP TEXT- TO-IMAGE SYNTHESIS BY SELF-COOPERATIVE DIF- FUSION GANS
Yihong Luo', Xiaolong Chen², Xinghua Qu',
Tianyang Hu', Jing Tang2.1*
HKUST HKUST(GZ)) Bytedance Seed
NUS
ABSTRACT
Recently, some works have tried combine diffusion and Generative Adversarial Networks (GANs) to alleviate the computational cost of the iterative denoising inference in Diffusion Models (DMs). However, existing works in this line suffer from either training instability and mode collapse subpar one-step generation learning efficiency. To address these issues, we introduce YOSO, a novel genera- tive model designed for rapid, scalable, and high-fidelity one-step image synthesis with high training stability and mode coverage. Specifically, we smooth the ad- versarial divergence by the denoising generator itself,, performing celf-cooperative learning. We show that our method can serve as a one-step generation model train- ing from scratch with competitive performance. Moreover, to one-step text-to-imaae generation based on pre-trained models by several ef- fective training techniques (i.e., latent perceptual loss and latent discriminator for efficient training along with the latent DMs; the informative prior initialization (IPI), and the quick adaption stage for fixing the flawed noise scheduler). Exper- imental results show that YOSO achieves the state-of-the-art one-step generation particular, performance even with Low-Rank Adaptation (LoRA) fine-tuning. we show that the YOSO-PixArt-a can generate images in one step trained on 512 resolution, with the capability of adapting to 1024 resolution without extra explicit training, requiring only - 10 A800 days for fine-tuning. Our code is available at: http:///i.aa..oaoooooooooooo////
extend our YOSO
com Luo Yihong/YYSO
INTRODUCTION
Diffuusion models (DMs) (Sohl-Dickstein et al. 2013 Ho et al. 2020: Song et al.
202[) have re-
cently emerged powerful class of generative models, demonstrating of the-art results in
many generative modeling tasks, such as text-to-image (Rombach et al. 2022 Xu et al. 2023c
Chen et al. 2024: Feng et al. 2023), , text-to-video (Blatmmann al. 2023 Hong et al. 2022 image
editing (Hertz et al. [2022] Brooks et al. [023 Meng et al.) 2022) and controlled generation (Zhang
al. 2023 Mou et al. 2023). However, the generation process of DMs requires iterative denoising.
leading to slow generation speed. Coupled with the intensive computational requirements of large-
scale DMs, they constitute a substantial barrier to their practical application and wider adoption.
Sampling from DMs can be regarded as solving a probability flow ordinary differential equation (PF-ODE) (Song et al., 202T). Some previous works (Song et al., 2020 Lu al. 2022a Bao et al. (022) focus on developing advanced ODE-solvers, for reducing the sampling steps. However, they still require 20+ steps to achieve high-quality generation. Another line distilling from pre-
trained PF-ODEs (Song et al.) 2023 Liu et al.) 20233 Luo et al.) [2023a blc; Salimans & Ho, 2022)
aiming to predict multi-step solution of PF-ODE solver by one step. Existing methods (Luo et al. 2023a b) achieve reasonable sample quality with 4+ steps. However, it is still challenging to generate high-quality samples with one step.
In contrast, generative adversarial networks (GANs) (Goodiellow et al. 2014 Radford et al. 2016)
are naturally built on one-step generation with fast sampling speed. However, is hard to extend
GANs on large-scale datasets due training challenges (Sauer al. 2022 Kang al.) 2023
sulting in worse sample quality compared to DMs (Sauer et al. 2023b: Kang et al. 2023). In this
"Coresponding Author: Jing Tang (jingtang@uss

---

Published as a conference paper at ICLR 2025
consisteneyy loss
&t loss
grasdend
Generator
Generalor
Diserim vondor
rwel/ jake?
Yogk graalen
forward diffrssio procers
(a) Framework of YOSO.
(b) 1024 resolution.
512 resolution.
Figure I: One-step generated images by YOSO under different configurations (Bottom). The model trained by fine-tuning PixArt-a (Chen et al. 2024) on 512 resolution with our proposed algorithm. Bottom Left is generated by YOSO adapting to 1024 resolution with Eq. without extra explicit training.
work, we present novel approach to combine diffusion process and GANs. The key to achieving access finding way to smooth the adversarial divergence for stabilize the training while maintain- ing effective one-step learning. Previous works (Xiao et al. 2022, Xu et al. 2023b c; Sauer et al.) 2023cp have developed some variants for combining diffusion and GANs. However, existing works either directly perform adversarial divergence against real data without smoothing strategy which suffers from unstable training and mode collapse, or rely on adding noise to smooth the adversarial divergence to stabilize the training which delivers less effective one-step generation learning. To enjoy the best of both worlds, we propose to smooth the adversarial divergence by denoising the generator itself. In particular, samples as ground truth and regard the one-step denoising generation based on more corrupted sam- ples student distribution perform adversarial divergence. The approach reduce the distance between target distribution and student distribution to stabilize the training, but also form effective one-step learning on clean samples. The learning process can be viewed self-cooperative process (Xie et al. 2018), as the generator learns from itself. Such an innovative design enables stable training and effective learning for one-step generation. Hence, we name our model YOSO, short for You Only Sample Once.
regard the one-step denoising generation based on less corrupted
only naturally
Moreover, extend our YOSO to one-step text-to-image generation based on pre-trained models and introduce several effective training techniques (i.e., latent perceptual loss and latent discrim- inator for efficient training along with latent DMs; the informative prior initialization (IPI), and the quick adaption stage for fixing the noise scheduler). Thanks to effecttve designs, can efficiently and effectively fine-tune existing pre-traines text-to-image DMs (i.e., stable diffu- sion (Rombach et al. 2022 and PixArt-a (Chen al., 2024)) for high-quality one-step generation (see Fig. []. Furthermore, we are the pioneers in supporting Low-Rank Adaptation (LoRA) (Hu al. 2022) fine-tuning in one-step text-to-image generation for enhanced efficiency, delivering state-of-the-art performance.
Our work presents several significant contributions:
2

---

Published as a conference paper at ICLR 2025
We introduce YOSO, novel generative model that generate high-quality images with one- step inference, with a stable training process and good mode coverage.
We further scale up YOSO by several principled and effective proposed training techniques that enable low-resource fine-tuning of pre-trained text-to-image DMs for one-step text-to-image, requiring only ~10 A800 days.
We conduct extensive experiments to demonstrate the effectiveness of the proposed YOSO, including image generation from scratch, texi-to-image generation fine-tuning, compatibility with existing image customization and image controllable modules.
2 BACKGROUND
Diffusion models. Diffusion models (DMs) (Sohl-Dickstein al. 2015 Ho al., 2020) define forward process that gradually transforms samples from data distribution to Gaussian distribution by adding noise in I steps with variance schedule By: (xx xx- _ (xx; vI f,x;, (00e-1.1111 3,1). The corrupted samples can be directly obtained by x, o; + v - agc, where â₂ = -II ( 3r) and E (O001 I). The parameterized reversed diffusion process is defined to gradually denoise: t). o21). The model can be trained by minimizing the negative ELBO (Ho et al. 2020 Kingma et al. 2021 ; L = (xxo(()))((s (xc) (L(g(xx-1(xx, (x,, pp - (x))
p
xx) _ (1x
where g x Xt, Xo) Gaussian posterior derived in (Ho et al. 2020). A key assumption in diffu-
sion is that the denoising step size from / to t is sufficienlly small. This assumption ensures the true g x)) follows Gaussian distribution, enabling the effectiveness of modeling po(x,-1|x, with Gaussian distribution.
GANN hybrids. An issue in DMs is that the true ((xx-1(xe) does not follow Gaussian distribution when the denoising step size is not sufficiently small. Therefore, in order to enable large denoising step size, Diffusion GANs (Xiao al. 2022) propose to minimize the adversarial diver- gence between model Po (x xx) and g X ) ming [P(x.))Davv (((x)v(((x11)))))) [ where pe (xx) & S po (x - X2, x)dx, and po (x xx,) is imposed by GAN generator. The capability of a GAN-based formulation enables much larger denoising step sizes (i.e., 4 steps).
3 METHOD: SELF--COPERA DIFFUSION GANS
A key issue in Diffusion-GAA hybrid models (iiao et al. 2022 Xu et al 2023b c) is that they match the generator distribution (x (xx) ) A (xs(xx)xx,) (x)x))x)))(( ) Xt- Xr. Xo) with the corrupted data distribu-
tion. This formulation only indirectly learns the px(xo (xx) and pa px(x)) ʃ qxx1 )o (xo xx, )dx;,
which are the distributions used for one-step generation, making the learning process less effective.
3.1 OUR DESIGN
To enable more effective learning for one-step generation, we propose to directly construct the learn- ing objectives over clean data. We first construct sequence distribution of clean data follows:
Pe 10 (xx) - (xx)))o
<TST:
Pe (xo) & q(xo),
where (xxo) the data distribution, pe xx) & NV (),,,), o²I) and Go the denoising gen- erator. Note that Go (Xe, t) is our denoising generator that predicts clean samples. If the network eæ parameterized to predict noise, we have Go(x,, t) x
de
Given the constructed distribution, we can formulate the optimization objective follows:
Th [Dadv (1(x) (2) (xx) + X xx)((((xx xx))]
(2)
= (aav (q(x) (x)) + Xx (g(()))))( [xx) 9 (x: (xa
where q(xo,x/) _ q (x)) and pe (x0, xx) q(x, (x xx). The optimization objective is constructed by combining adversarial divergence and KL divergence. Specifically, the adversarial divergence focuses matching the distribution level, ensuring the generation quality, and the KL divergence focuses on matching over point level, ensuring the mode coverage.
)
However, it is hard directly learn the adversarial divergence over clean data distribution, akin to the challenges with GAN training. To tackle these challenges, previous Diffusion GANs (Xiao et al.)
2022 Xu al. 2023b) have pivoted towards learning the adversarial divergence over the corrupted
3

---

Published as a conference paper at ICLR 2025
data distribution. Unfortunately, analyzed before, such approach fails to directly match pe (xu), curtailing the efficacy of one-step generation. Moreover, it also compels the discriminator to fit different levels of noise, leading to limited capability.
(1) (x) is defined as I ((xx))o (x(x,)xxx The quality of the distribution has two key Recall that Pg factors: 1) the ability of the trainable generator Go; 2) the information given by X
Hence given the generator Go fixed, if we increase the information Xx is supposed that we can
(t) get a better distribution. In other words, it is highly likely that Pa (tx) (x) is superior to pa (x). where
t max k, 0} Motivated by the cooperative approach (Xie et al. 2018 2021 2022 Luo
et al. (024a) which uses an MCMC revised version of model distribution to learn the generator,
suggest to use Pg (x) as ground truth learn p6 (x). which constructs the following training objective:
(fu)
min Lo - E, (xxc)aaa (x) (x t) - x11 Ba Ra (D) (r)")) (sg(x))ll)) (sx()))) (x)))))))) (x)),
(3)
where sq denotes stop-gres -raddgadient operator and the second term is named as cooperative adversarial loss. This training objective can be regarded as self-cooperative approach, since the 'revised' samples from Ps [tx) (x) and samples from Pe (x) are generated by the same network Gg. Note that we only replace data distribution as ((x) (x) in the adversarial divergence for smoothing the learning objective, as recent work (Luo et al. (024b)) has found that it is beneficial to learn the generator with mix of real data and revised data.
We briefly verify the theoretical rationality of the cooperative adversarial divergence below.
Proposition 1 The optimal solution of the cooperative adversarial loss reaches Pg (T) (x) Pa(x).
This proposition tells that the proposed cooperative adversarial loss can recover the real data distri- bution when the network's capability is sufficiently large, demonstrating the theoretical rationality of the proposed objective. See proof in the Appendix C
In the distribution matching objective above, we apply the non-saturating GAN objective mini- mize the adversarial divergence of marginal distributions. And the KL divergence for point matching can be optimized by L2 loss. Hence tractable training objective is formulated as follows:
min max E, (1x) (xx)(x)
P9 (x) log (x(ss(x),, 1) (x) log D ()] + A Axx))xa [Go(x,, 1) x112.
where Do is the discriminator network. We find that the self-cooperative approach is connected with Consistency Training (Song et al., 2023). However, Consistency Training considers the xx, as an approximated ODE solution of X; to perform a point-to-point match. In contrast, our proposed (t) objective matches ph (x) and Po (fx) (x) in marginal distribution level, which avoids the approximated
error of ODE.
To further ensure the mode cover of the proposed model, can add consistency loss to our objective as regularization, which constructs the following loss:
min max E, {E ste) [log D (x, t) - E po log D (x, 4) P0
#
(4)
Bo (xx b x)) (X()) x112 X. [[Ga(x), -
where Xçon and (L) are pre-defined hyper-parameters.
4 TRY IT ON CIFAR-10 10 BEFORE SCALING UP FOR SAVING MONEY!
In this section, we evaluate the performance of the proposed YOSO on CIFAR-10 (Yu et al. 2015) to verify its effectiveness under both training from scratch and fine-tuning settings.
4.1 TRAINING STRATEGIES
Before starting training, introduce some effective training strategies in following for taming YOSO.
Decoupled scheduler. We find that the optimal scheduler for performing consistency loss and ad- versarial loss are not identical. This is due to the cooperative adversarial loss not involving the ap- proximated error regarding the timestep skips, enabling substantial skip to maximize its efficacy.
4

---

Published as a conference paper at ICLR 2025
In contrast, the consistency loss is susceptible approximated error from timestep skips, necessi- tating a more conservative skip to preserve its effectiveness. Hence, to better unleash the capability of each loss, we propose using decoupled schedulers to construct our final training objective:
min max E, {E, (tt)) [lg (x
E (0)) log (x, t)]
(5)
+ +B B. (x) Nem [A(t)] - x 22 + x: co [Go(x,, (xt.) tm ))])))
where E2- = max(t- k 0), fm = max(t m, 0), and we let & = 250 and m = 25 in experiments.
Annealing strategy. Since our target is to obtain a pow- erful one-step denoised generation model, however, as the KL loss and consistency loss will enforce point-level matching, this may hurt performance in cases where the model capacity is insufficient. Therefore, we suggest re- ducing the weight of these two losses to zero as training
Table I: Unconditional generation re- sults on CIFAR-I0.
Model
FID
NFE
Distillation or Fine-luning YOSO
1.81
Tract Berthele et al. 2023
3.78
progresses. Specifically, we let = (1- K-I ) for de-
Dift-tmtraet
2123c
4.53
DMD
2.66
creasing the weight to zero with K times, where n is the current iterations and N is the total training iterations.
CTM Kim et al. 2024
1.98
Zhou et
1.92
Training from scratch
4.2 EMPIRICAL EVALUATION
YOSO
2:26
DDGANs Xiao et al
2022
3.75
4
Experiment Setting. We the EDM (Karras al. 2022) architecture for both UNet and discriminator. For the evaluation metric, choose the Fréchet inception distance (FID) (Heusel et al., 2017). We train the pro- posed model the CIFAR- 10 dataset (Yu al., 2015). We only employ MSE to compute image distance.
CT Song
8.70
ICT
SSng & Dhurwall 2024
2,83
DDPM Ho 2020
3.21
EDM
(Kartas et mi 2072
2.04
36
DDIM Song
4.67
50
StyleGAN2 (Karras et al. 2019
2.92
The quantitative results are shown in Tab. I We observe that our method provides state- of-the-art performance with only a one-step manner on both training from scratch and fine-tuning settings, outperforming previous existing accelerated DMs, and GANs. In particular, our YOSO in fine-tuning EDM (Karras et al., (022) even delivers better performance compared to EDM itself with ODE sampling. This because our YOSO performs fine-tuning instead of distillation, the optimal solution for our YOSO is data distribution instead of teacher distribution.
Results.
5 TOWARDS ONE-STEP TEXT-TO-IMAGE SYNTHESIS
Since training text-to-image model from scratch is quite expensive, we suggest using pre-trained text-to-image DMs as initialization with Self- Cofe-Coopeertive Diffusion GANs. In this section, we introduce several principled designs for developing generation model that enables one-step text- to-image synthesis based on pre-trained DMs.
5.1 USING PRE-TRAINED MODELS FOR TRAINING
Latent Perceptual Loss Prior research (Hou et al., 2017 Hoshen et al.) 2019 Song ct al. 2023) )
has confirmed the effectiveness of perceptual loss in various domains. Notably, recent studies (Liu et al. 2023 Song et al., 2023) have found that the LPIPS loss (Zhang et al. 2018 is crucial for obtaining few-step DMs with high sample quality. However, a notable drawback is that the LPIPS loss is computed in the data space, which is expensive. In contrast, the popular SD operates in the latent space to reduce computational demands. Hence, using LPIPS loss in training latent DMs is pretty expensive, as it requires not only the computation of LPIPS loss in the data space but also an additional decoding operation. Realizing that the pre-trained SD can serve as an effective feature extractor (Xu et al., 2023a), we suggest using pre-trained SD to perform latent perceptual loss. However, SD is a UNet, whose final layer predicts epsilon with dimensions identical to the data. Hence we propose using the bottleneck layer of the UNet for computing:
d(zo,z) z) ||HalUUNet(zo, c,
HallUNet(z,,,
where is the latent encoded images by VAE and is the text feature. We note that the benefit of computing latent perceptual loss by SD is not only the computational efficiency but also the incorporation of text features, which are erucial in text-to-image tasks.
Latent Discriminator. Training GANs for the text-to-image on large-scale datasets faces serious challenges. Specifically, unlike unconditional generation, the discriminator for the text-to-image
5

---

Published as a conference paper at ICLR 2025
task should justify based on both image quality and text alignment. This challenge is more obvious during the initial stage of training. To address this issue, previous pure GANs (Kang et al. (023) for text-to-image propose complex learning objectives and require expensive costs for training. The learning of GANs has been shown to benefit from using pre-trained network as the discriminator. As discussed above, the pre-trained SI) has learned representative features. Hence, we suggest ap- plying the pre-trained SD for constructing the latent discriminator. Similar to latent perceptual loss, we only use half UNet for the discriminator followed by simple predict head. The advantages of the proposed strategy are twofold: 1) we use the informative pre-trained network as initialization; 2) the diseriminator is defined over latent space, which is computationally efficient. Unlike previ- ous work (Sauer et al., 2023c) that defines a discriminator over data space which required decode latent and backward from decoder, yielding expensive computational cost. By applying the Latent Discriminator, we observe a stable training process with fast convergence.
5.2 FIXING THE NOISE SCHEDULER
A common issue DMs that the final corrupted samples are not pure noise. For example, the noise scheduler used by the SD makes the corrupted samples at the final timestep as: Xy = 0.068265 I-ar 0.004682, effectively
Xo + 0. 0.99767- 6, where the terminal Signal-to-noise ratio (SNR) is är creates a gap between training and inference. Previous work (Lin et al. 2024a) has only observed that this makes DMs unable to produce pure black or white images. However, we find that this issue yields serious problems in one-step generation. As shown in Fig. 2 there are notable artifacts in one-step generation if we directly sample noise from standard Gaussian. The reason may be that in multi-step generation, the gap can be gradually fixed in sampling, while one-step generation reflects the gap more directly. To address this issue, provide two simple yet effective solutions.
Informative Prior Initialization (IPI). The non-zero terminal SNR issue is similar to prior hole issues in VAEs (Klushyn et al. 2019 Bauer& & Mnih, 2019, Kingma et al., 2016). Therefore, we use informative prior instead of non-informative prior to effee- tively address this issue. For simplicity, we adopt a learnable Gaus- sian Distribution N I²)), whose optimal formulation is given below: e" = arr (Exx + Std(x) x e') + vi OT where Exx and (td can be efficiently estimated by finite samples, and e' follows standard Gaussian distribution. As shown in Fig, [ the arti- facts in one-step generation are immediately removed after applying IPI. We note that the performance is achieved with minimal adjust- ment, enabling the possibility of developing one-step text- text-to- to--mage generation by LoRA fine-tuning.
(a) w/o IPI
(b) w/ IPI
Figure 2: Samples by YOSO- LoRA with one-step infer- ence from different initializa- tion.
Quick Adaption to V-prediction and Zero Terminal SNR. The IPI suffers from numerical instability when terminal SNR is pretty low. As shown in Fig. 3] we fine-tune PixArt-o (Chen et al. 2024) whose terminal SNR is 4e-5 with introduced technique and e-prediction, failing in one-step generation. Following (Lin et al., 2024a), we suggest switching to v-prediction (Salimans & Ho! 2022) and zero ter- minal SNR. However, we found that direct transition convergences slowly (see Appendix(F). This is unacceptable in solving large-scale text-to-imaat with limited computational resources. To address this, we propose a quick adapt-stage:
Figure 3: Pre- dicting fails.
Adapt-stage-I switch to v-prediction : I((O) Xel1vo(X1,, t) us (x,, 1) where us (xx, 1) (x6o(x;, () v1 a,x!, x², X: aa-viiaa,e,la.. ..((,)) denotes the desired v-prediction model, and (.)))) denotes the frozen tre-ted model.
Adapt-stage-Il switch to zero terminal SNR [(() = Xx)lex(xe, - (x(xx, Note that in this stage, we only change the scheduler to zero terminal SNR for student model. This avoids numerical instability issues of e-prediction with zero terminal SNR. We note that the zero terminal SNR scheduler has lower SNR than the original schedule at each timestep, formulating an effective distillation objective.
We note that this adapt stage converges rapidly, typically requiring only 1k iterations to initialize training YOSO. This enables quick adaption v-prediction and zero terminal SNR from pre- trained e-reddcctonn DMs, thereby comprehensively mitigating the non-zero terminal SNR issue in the noise scheduler.

---

Published as a conference paper at ICLR 2025
1.5 NFE, 25 steps
LGM steps
TCD NFE, staps
PeRFlow stops
SD Turbo
Hyper-SD NFE,
YOSO (LoRA) NFE, step
NFE, step
stop
dog. best quality
monkey making
Close-up of lody sunglosses
worker looks like and working type code
Figure 4: Qualitative comparisons of YOSO against competing methods. NFE denotes the Number of Function Evaluations.
DMD-PiwArt
50-Turbe
YOSO-PixArt
VO50-SD
Figure 5: Qualitative comparisons of YOSO against competing methods. can be seen that both DMD and SD-Turbo suffer from mode collapse, while YOSO achieves better sample quality and mode cover. The prompt is "A cute dinosaur, cartoon style, white background".
EXPERIMENTS
In this section, evaluate the performance of the proposed YOSO. Sec. 6._ examines YOSO in the context of text-to-image generation by fine-tuning pre-trained PixArt-a (Chen et al., 2024) and Stable Diffusion (Rombach et al.) (022). In Sec. [6.3 we show that the YOSO can be used for several downstream applications. Moreover, we conduct ablation studies in Sec. 6.2 highlight the effectiveness of proposed algorithm and proposed training techniques.
6.1 TEXT-TO-IMAGE GENERATION
Experiment Setting. We initialize the GAN generator by pre-trained PixArt-a which is diffusion transformer with 0.6B parameters. For the GAN discriminator, construct the latent discriminator by pre-trained SD 1.5 using the proposed construction introduced Sec. [5.]] We switch the pre- trained PixArt-ar to v-prediction by the proposed technique introduced in Sec. 5.1, followed by training on the JourneyDB dataset (Pan al.] 2023) with resizing as 512 resolution. We apply batch-size of 256 and constant learning rate of 2e-5 during training. We observe the training convergence fast, requiring only 30k iterations and around 10 A800 days to be trained. We apply batch-size of 256 and constant learning rate of 2e-5 during training. We also conduct experiments on fine-tuning SD 1.5 via LoRA to show the effectiveness of the proposed YOSO.
Evaluation. We employ Aesthetic Score (AeS) (Schuhmann et al. 2022 to evaluate image quality and adopt the Human Preference Score (HPS) v2.1 (Wu et al. 2023) to evaluate the image-text alignment and human preference. The AeS is computed by aesthetic predictor trained

---

Published as a conference paper at ICLR 2025
LAION (Schuhmann et al. 2022 datasets, without considering the image-text alignment. HPS is trained to predict human preference given image-text pairs, considering the image-text alignment and human aesthetic. Additionally, we include ImageReward score (Xu et al., (024). and CLIP score (Hessel et al., 2021) provide a more comprehensive evaluation of the model performance. We mainly compare our model against the open-source state-of-the oftt-of-the-rtt aht (SOTA) models, e.g., SD- Turbo (Sauer et al.] 2023c). PixArt-DMD (Chen al., 2024
Yin et al. 2023). and Hyper-SD (a
concurrent work) (Ren et al. 2024).
Quantitative Results. The quantita- tive results are presented in Tab. 2 As shown in Tab. [ YOSO clearly beat previous state-of-the-art meth- ods across all metrics (including HPS, AeS, Image Reward score, and Clip Score). It is important to highlight that for the SD 1.5 back- bone, we only use LoRA for fine- tuning SD 1.5, where only less than 10% parameters are tuned. Mo- roever, our method without human feedback learning (HFL) even out- performs Hyper-SD with HFL which potentially hacks the machine metric.
Table 2: Comparison of machine metrics on -mmae generation across state-of-the-an methods. HFL denotes hu- man feedback learning which might hack the machine met- rics.
Model
Backhone
HHL
Stops
HPST
AcS7
CST
Bose Medel
33.46
(tueer 2003
Broe Medel
SD 155
No
25
24.72
5:49
0.20
31.88
InstaFlow [ Linetal h023
SD 1.5
No
26.18
5.27
-0.22
30.04
Hyper SD Loxc (cea ac 2024
SD 1.5
Yes
28.01
5.79
0.29
30.87
YOSO LoRA (oRA)
SD 1.5
No
28.35
5.97
0.43
31.33
LCM LoRA et 803bb
SD 1.5
No
22.77
5.66
-0.37
30.36
PotHoow (Yarel 2025))
SD 1.5
No
22.43
5.64
-0.25
30.77
TCD-LaRX 2arneer 2004
SD 1.5
No
22.24
5.45
-0.15
30.62
Hyper-SD-I.i S.... (ixx (oare Si 2014
SD 1.5
Yos
50.34
5.65
0.53
$1.07
YOSO LoRA
SD
No
30,50
6,05
0.56
31.52
Dase Model
O
No
25
31.07
6.12
1.05
33.81
DMD cineral 2003
PixArte
No
29.78
6.02
0.97
$2.90
YOSUTORIST
PixAn-
No
30.52
6.19
1.002
32.95
Qualitative Comparison. To provide more comprehensive insight in understanding the perfor- mance of our YOSO, we further provide the qualitative comparison in Fig. [ The comparison shows that: YOSO clearly beats existing baselines in terms of both image quality and prompt align- ment, which is achieved by LoRA-finetuning. Specifically, the samples by SD-Turbo are blurry to some extent, and the samples by Hyper-SD are over-saturated and with notable artifacts. Moreover, the advantage of YOSO in text-image alignment is significant. For example, in the last row of Fig. related to a cow-people worker is coding, previous methods including SD 1.5 itself cannot produce samples following the prompt, while YOSO precisely follows the prompt to generate half cow half people worker. Additionally, it is important to highlight that both SD-Turbo and DMD seem to have a serious issue mode collapse. As shown in Fig. S both SD-Turbo and DMD generate samples that are extremely close to each other. Overall, based on quantitative and qualitative results, we conclude that our YOSO is better in sample quality, prompt alignment, and mode cover compared with the SOTA one-step text-to-image models.
6.1.1 ZERO-SHOT OTE- 1024 RESOLUTION GENERATION
Due to the computational resource limitation, we trained YOSO on 512 resolution. However, there 1024-resolution version of PixArt-a, obtained by continuing training on the 512-resolution version of PixArt-o. Motivated by the effectiveness of the LoRA combination (Luo et al. 2023b
we suggest constructing a similar combination as follows:
W WYos01024 W WixxAr01a + a (W PixArt WPinArtai)) + 3- 33 WPiNAn (7)
where a, B E (0. 1] and W means the model weight. We let a = 3 = 1 in our experiments. As shown in Fig. 1b the YOSO constructed in this way can generate high-quality images with 1024 resolution. Note that YOSO even changes the predicted objective to v-prediction. The impressive performance indicates the robust generalization ability of our proposed YOSO.
6.2 ABLATION STUDIES
We provide additional insight into the effectiveness of the proposed YOSO by performing ablation studies on some potential variants on CIFAR- 10 and on text-to-image generation. We call YOSO without decoupled scheduler and annealing strategy YOSO-base. See more details in Appendix.
Effect of Consistency Loss. For the effect of consistency loss in YOSO, our results suggest that improve the image quality, as indicated by the FID scores in Tab. Notably, the removal of the consistency loss does not lead to a substantial decline in generation performance, which indicates the effectiveness of our proposed adversarial formulation.
8

---

Published as a conference paper at ICLR 2025
YO5a,
Dreansruper. 2 steps
Teunytu, 2 sileps
Irage Edit: Deg Lion,
ControlNet: Scribble
ControlNet:
Figure 6: Qualitative comparison against competing methods and applications in down-stream tasks.
Table Ablation study on CIFAR-10 Table 4: Ablation study one-step text-to-image synthesis. with smaller backbone.
Model
FID,
NFE J
Lper
IPI
Naive GAN
Coop GAN
HPS
AeS
YOSO
3,05
15.22
5.23
YOSO-Bad decoupled scheduler
20.08
5.45
YOSO-Base
3.82
20.63
YOSO Base weo consistency less
1.11
24.88
5.69
YOSO-Bay wio LPPSS loss
4.63
26.42
YOSO-Base (aull)))
27.10
5.94
UPOGln
28.33
5.97
DDGAN
Xno er
3.75
Effeet of Latent Perceptual Loss. To investigate the effect of latent perceptual loss proposed in Sec. [. we conduct experiments text-to-image tasks by using it to compute the consistency loss. As shown in Tab. 4 the proposed latent perceptual loss clearly improves the HPS from 15.22 to 20.08 and AeS from 5.23 to 5.45, highlighting its effectiveness.
Effect of Decoupled Scheduler. We evaluate the impact of the proposed decoupled scheduler by adding it YOSO-base. As evidenced by Tab. 3 and Tab. 4 the incorporation of the decoupled scheduler significantly enhances sample quality, as reflected by improvements in FID, HPS, and AeS metrics. Notably, the FID score on CIFAR-I0 10 decreases from 3.82 3.30 and the HPS in lext-to-image tasks is improved from 26.42 to 27.10, demonstrating the superior efficacy of the decoupled scheduler.
Effeet of Annealing Strategy. We also assess the effect of the proposed annealing strategy by removing from the final formulated YOSO. As illustrated in Tab. 3 and Tab. [4] the removal of the annealing strategy degrades FID from 3.05 to 3.30 and reduces HPS from 28.33 to 17.10. Such results demonstrate the effectiveness of the annealing strategy in training YOSO.
Adversarial Divergence Dadv 1p)). The key design of our proposed YOSO smoothing the adversarial di- vergence by using self-generated data instead of real data as ground truth to perform adversarial divergence. We evaluate the variant of using Dadv (pullp') adversarial divergence. As shown in Tab. 33 the performance of the variant is significantly worse than YOSO-Base, i.e., FID degrades from 3.82 to 10.85. Moreover, besides sample quality, we find that the training of our proposed YOSO is much more stable than the variant. As shown in Fig. 7 the discriminator loss of the variant is not stable and is much smaller than ours. We also compare the variant in the text-to-image task. The results are given in Tab. 4 from which we can see that replacing our cooperative adversarial loss with Dadv (Pal (p)) yields serious performance deterioration, i.e. HPS decreases from 26.42 24.88 and AeS decreases from 5.85 to 5.69. This indicates that our approach can effectively smooth the adversarial divergence by narrowing the gap between fake and real distribution, which makes dis- crimination harder and hence increases discriminator loss. Furthermore, in the training of this variant. Note that a similar case also occurs in SD-Turbo, a strong one-step text-to-image model, which performs adversarial divergence directly against real data (See Fig. Overall, these results indicate the superiority of the proposed YOSO.
1050
10000
1030 30003 22.00 10 114000
Figure 7: The discriminator loss curve.
observe mode collapse
Comparison to UFOGen. Similar to us, UFOGen does not directly perform adversarial diver- gence against real data either. Instead, they employ adversarial divergence over corrupted data (i.c., Dady (q(x+-1)11 [(xl)//((x__ q(xx111 Go(xe, (x)()))))(x))))))))) )dx.)). However, we argue that such formulation

---

Published as a conference paper at ICLR 2025
is less effective in constructing one-step generation model as the adversarial divergence it employs only indirectly aligns with one-step generation at the level of clean data. We implement UFOGen on CIFAR-10 10 on our own. Note that we also perform consistency loss in the UFOGen variant to ensure a fair comparison, as the consistency loss has been found useful in YOSO. As shown in Tab. 3 UFOGen only obtains 45.15 FID on CIFAR-10, 10, significantly worse than YOSO with 3.82 FID. This further underscores the superiority of our proposed YOSO. Through comparison with dif- ferent adversarial divergence formulations, we find that our approach not only stabilizes training by smoothing the adversarial divergence but also facilitates effective learning for one-step generation.
6.3 APPLICATION
One promising and attractive property of DMs is that they can be used in multiple downstream tasks. In this section, we show the capability of our proposed YOSO in various downstream tasks, keeping the unique advantages of DMs: 1) Image-to-image Editing: As shown in Fig. 6 our YOSO-LoRA is capable of performing high-quality image-to-image editing (Meng et al. 2022 in one step: 2) Compatibility with ControlNet and Different Base Models: As shown in Fig. our YOSO-LoRA is compatible with ControlNet (Zhang et al. 2023), following the condition well. Our YOSO-LoRA is also compatible with different base models (e.g., Dreamshaper and Toonyou) fine- tuned from SD 1.5, preserving their style well. We find that when applying YOSO-LoR LoRA to new
base models, it fails to produce reasonable samples in one step. This may be due to the capability of LoRA and the distribution shift of training data.
7 RELATED WORK
Few-Step Texa- Generation. LCM (Luo et al. 2023a) adapts consistency distilla- tion (Song et al.] 2023) for stable diffusion (SD) and enables image generation in 4 steps with acceptable quality. InstaFlow (Liu et al. 2023) adapts rectified flows (Liu et al. 2022 Liu 2022) for SD, facilitating the generation of text-to-imaae
mere single step. Despite this, the fidelity of the generated images is still poor. Some works ( (Yin et al. (202)) adapts variation score distil- lation (Wang et al., 20236) and diff-instruce (Luo et al.] (023c) for SD, which requires training an additional DMs for the generator distribution, akin to the discriminator in GANs. We found models trained in this way seem to have serious mode collapse issues (see Fig. 5). Recently, combining DMs and GANs for one-step text-to-image generation has been explored. UFOGen (Xu et al. 2023cp extends DDGANs (Xiao et al., 2022) and SSIDMs (Xu et al. 2023b) to SD by modifying the computation of reconstruction loss from corrupted samples to clean samples. However, it still performs adversarial matching using corrupted samples. ADD (Sauer et al. (023c) introduces a one-step text-to-image generation based on SD. It follows earlier research (Sauer al.) 2023a) by
employing a pre-trained image encoder, DINOv2 (Oquab et al., 2024), as the backbone of the dis- criminator to accelerate the training. However, the discriminator design moves the training from latent space to pixel space, which substantially heightens computational demands. Moreover, they directly perform adversarial matches at clean real data, increasing the challenge of training. This requires the need for better but expensive discriminator design and expensive R1 regularization to stabilize the training. SDXLL-Lightning (Lin et al. (024b) combines progressive distillation and adversarial training in progressive framework. Recently, concurrent work Hyper-SD (Ren al.) 2024) combines consistency distillation and adversarial training in a progressive framework, while our work can be trained end-to-end. However, their work performs adversarial point-level match- ing via consistency loss, and they still perform adversarial training via injecting noise. This results in less effective one-step learning, thus they rely human feedback learning for achieving better performance. In contrast, we perform the adversarial training at the distribution level by replacing real data with self-generated data to smooth the adversarial divergence, which not only can stabilize the training without injecting noise but also form effective one-step generation learning. Moreover, compared with above mentioned scratch to perform one-step generation, which is not demonstrated by them. Additionally, we extend our method not only to SD but also to PixArt-a (Chen et al. (024) which is based on diffusion transformer (Peebles & Xie 2022). This demonstrates the wide application of our proposed YOSO.
cerfusiionGAAl hybrid models, our approaches can be trained from
CONCLUSION
In summary, we present YOSO, a new generative model enabling high-quality one-step generation. Our novel designs combine the diffusion process and GANs, enabling not only one-step generation training from scratch but also one-step text-to- to-iaagg generation tne-tuning from pre-trained models. We will release our model to advance the research of text-to-image synthesis.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Jing Tang's work is partially supported by National Key R&D Program of China under Grant
No. 2023YFF0725100 and No. 102471A1012701, by the National Natural Science Foundation of China (NSFC) under Grant No. 62402410 and No. U22B2060, by Guangdong Provincial Project
(No. 2023QN10X025), by Guangdong Basic and Applied Basic Research Foundation under Grant
No. 2023A1515110111 by Guangzhou Municipal Science and Technology Bureau under Grant
No. 2023A0310667 and No. 2024A04J4454, by Guangzhou Municipal Education Bureau (No. 2024312263), and by Guangzhou Municipality Big Data Intelligence Key Lab (No. 2023A03J0012), Guangzhou Industrial Information and Intelligent Key Laboratory Project (No. 2024A03J0628) and Guangzhou Municipal Key Laboratory of Financial Technology Cutting-Eoh -ddge Research (No. 2024A03J0630).
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint 2xXi::2211.01344 2022.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dmm: analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint arXiv:2201.06503 2022.
Matthias Bauer and Andriy Mnih. Resampled priors for variational autoencoders. In Kama- lika Chaudhuri and Masashi Sugiyama (eds.), The 22nd International Conference on Artifi- cial Intelligecce and Statisties, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, vol- ume 89 of Proceedings of Machine Learning Research, pp. 65-75. PMLR, 2019. URL http: 7/ //proceednggs.. ml www.ssppppe/.eoe.e. / v89 /tt/baaaerlrrr 9a. html
David Berthelot, Arnaud Autef, Jierui Lin, Dian Ang Yap, Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbot, and Eric Gu. Tract: Denoising diffusion models with transitive closure time-distillation. arXiv preprint arXiv:2303.0422. 2023.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rom- bach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-Slaile Saaaph-Sa Fast training of diffusion transformer for photorealistic text-to-image synthesis. In The Twelfth International Conference on Learning Representations, 2024. URL httph:tt///om /ttp///prrrrreeeee revirreviee net
Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang. Haotian Tang, Muyang Li, Yao Lu, and Song Han. Deep compression autoencoder for efficient high-resolution diffusion models, 2025.
URL attp:::://x.i...////////./ http:///////21111.0000
Zhida Feng, Zhenyu Zhang, Xintong Yu, Yewei Fang, Lanxin Li, Xuyi Chen, Yuxiang Lu, Jiaxiang Liu, Weichong Yin, Shikun Feng. Yu Sun, Li Chen, Hao Tian, Hua Wu, and Haifeng Wang- Ernie-vilg 2.0: Improving text-to-image diffusion model with knowledge-enhanced mixture-of- denoising-experts,, 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets, Advances in neural information processing systems, 27, 2014.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prontt image editing with cross attention control. arXiv preprint arXiv:2208.0166 2022.
11

---

Published as a conference paper at ICLR 2025
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-68511 2020.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. arXiv preprint arXiv:2205, 15868, 2022.
Yedid Hoshen, Ke Li, and Jitendra Malik. Non-adversarial image synthesis with generative latent nearest neighbors. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5811-5819, 2019.
Xianxu Hou, Linlin Shen, Ke Sun, and Guoping Qiu. Deep feature consistent variational autoen- coder. 2017 IEEE winter conference on applications of computer vision (WACV), pp. 1133- 1141. IEEE, 2017.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation large language models. In International Con- ference on Learning Representations, 2022. URL https: /ttps:://peeneviite
net / forum?
1d=nzevKeeFYf9
Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Minguk Kang, Richard Zhang, Connelly Barnes, Sylvain Paris, Suha Kwak, Jaesik Park, Eli Shecht- man, Jun-Yan Zhu, and Taesung Park. Distilling Diffusion Models into Conditional GANs. In European Conference on Computer Vision (ECCV), 2024.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Proc. NeurIPS, 2022.
Dongjun Kim, Chich-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning prob- ability flow ODE trajectory of diffusion. In The Twelfth International Conference on Learning Representations, 2024. URL https: http:s::/pppeeevveeeeeee net htt.e/neettttorae111111111133110
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances in neural information processing systems, 34:21696-21 2021.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29:4743 4751, 2016.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprint arXiv:2305.01566. 2023.
Alexej Klushyn, Nutan Chen, Richard Kurle, Botond Cseke, and Patrick van der Smagt. Learn- ing hierarchical priors vaes. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' AlAché-Buc,, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC. Canada, pp. 2875, 2019. URL 7d1 2d16ddddddff wtw.aa422211337700800000 wat.
12

---

Published as a conference paper at ICLR 2025
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 5404-5411, January 2024a.
Shanchuan Lin. Anran Wang, and Xiao Yang. Sixl-nigt Progressive adversarial diffusion distillation, 2024b. UtLpttos::////// http://axivv.... wwwp/////bbaaaeeeee.e /ttpa////./..9///
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland. September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Qiang Liu. Reetified flow: A marginal preserving approach to optimal transport. arXiv preprint arXiv:2209.14577, 2022.
Xingchao Liu, Chengyue Gong, et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2022.
Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, and Qiang Liu. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. arXiv preprint arXiv:2309.06330 2023.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:221..01095, 2022b.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthe- sizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.0437, 2023a.
Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longho Huang. Jian Li, and Hang Zhao. Lem-lora: A universal stable-diffusion acceleration module, 2023b.
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36, 2023c.
Yihong Luo, Siya Qiu, Xingjian Tao, Yujun Cai, and Jing Tang. Energy-calibrated vae with test time free lunch, 2024a. URL hatps:://// http//axivv.../////////////.atvm wwtp://..rrg.b.......aeeees 04071
Yihong Luo, Siya Qiu, Xingjian Tao, Yujun Cai, and Jing Tang. Energy-calibrated vae with test time free lunch. In European Conference on Computer Vision, pp. 326-344. Springer, 2024b.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. International Conference Learning Representations, 2022.
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint 2.XX::2302.08453. 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut. Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features with- out supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL net
6x
13

---

Published as a conference paper at ICLR 2025
Junting Pan, Keqiang Sun, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Journeydb:: A benchmark for generative image understanding, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09778, 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, In ICLR, 2016.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.0612) 1(2):3, 2022.
Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao. Hyper-sd: Trajectory segmented consistency model for efficient image synthesis, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegam-xl: Scaling stylegan to large diverse datasets. volume abs/2201.00273, 2022. URL
abs/gg/aaa//2/221/ 002 73
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. StyleGAN-T: Unlocking the power of GANs for fast large-scale text-to-image synthesis. volume abs/2301.09515, 2023a. URL perttps://arxte org. abs 22t.. 09515
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. arXiv preprint arX1v:2301.095/5 2023b.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis- tillation, 2023c.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 002225777222999 2022.
Jascha Sohl- Sohl-Diccstten, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint w:Xiv:2010.02500 2020.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In The Twelfth International Conference on Learning Representations, 2024. URL https://
http:/iwwwwwete..rr..eeeeeo www.foormminnneerr... wae 9bRDvG
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.
14

---

Published as a conference paper at ICLR 2025
Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion- gan: Training gans with diffusion. In The Eleventh International Conference on Learning Repre- sentations, 2023a.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: figh-fidelity and diverse text-to-3d generation with variational score distillation, arXiv preprint arXiv:2305. 13, 2023b.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis. arXiv preprint arXiv:2306.0934/ 2023.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising difffusion GANs. In International Conference on Learning Representations, 2022. URL
htttps://openrevil wtw.:n/eeooouummo11ooopr000--00-
net/ wow.rorrumieioro
Jianwen Xie, Yang Lu, Ruiqi Gao, and Ying Nian Wu. Cooperative learning of energy-based model and latent variable model via meme teaching. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Jianwen Xie, Zilong Zheng, and Ping Li. Learning energy-based model with variational auto- encoder as amortized sampler. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 10441-10451, 2021.
Jianwen Xie, Yaxuan Zhu, Jun Li, and Ping Li. A tale of two flows: Cooperative learning of langevin flow and normalizing flow toward energy-based based model. In International Conference on Learning Representations, 2022. URL wttp:///ww./ereeee...ee.mm wtt.ee/orrmmmm-m1155555300
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang. and Shalini De Mello. Open-Vocabeee Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models. arXiv preprint 33NV::2303.0800. 2023a.
Jiazheng Xu, Xiao Liu, Yuchen WU, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Yanwu Xu, Mingming Gong, Shaoan Xie, Wei Wei, Matthias Grundmann, Tingbo Hou, et al. Semi- implicit denoising diffusion models (siddms). arXiv preprint 220.::23.6...111,, 2023b.
Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. Ufogen: You forward once large scale text-to-image generation via diffusion gans. ArXiv, abs/2311.092557 2023c. URL https://
api. www.sematnncchhhlaaaam http:hhc.....oroooooo.saaaaaeaa.
Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: text-e generation via large mixture of diffusion paths. arXiv preprint arXiv:2305.18295. 2023.
Hanshu Yan, Xingchao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng. Perflow: Piecewise rectified flow as universal allg-and-ppp accelerator. arXiv preprint 20555.240..07510 2024.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint arXiv:2311. 188288, 2023.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint 1506::5556.035.. 033600 2015.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
15

---

Published as a conference paper at ICLR 2025
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. In CVPR, 2018.
Jianbin Zheng, Minghui Hu, Zhongyi Fan, Chaoyue Wang, Changxing Ding, Dacheng Tao, and Tat-Jen Cham. Trajectory consistency distillation, 2024.
Mingyuan Zhou, Huangjie Zheng, Zhendong Wang. Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In International Conference on Machine Learning, 2024.
16

---

Published as a conference paper at ICLR 2025
Algorithm YOSO training from scratch.
Require: dataset D, learning rate 7, total denoising steps T, total iterations N, distance metric d(s)), and noise scheduler 2(.,;.).
Ensure: optimized models Gg, De-
1: Initialize weights {0.Ø);
2: for to N do
3: Sample noise from standard normal distribution;
4: Sample data from dataset D;
5: Sample Timesteps è from to 'I' uniformly.
6: Obtain noisy samples x, Q(x, e,
7:
Obtain less noisy samples X,-1 - ②(x, e, 1,1-1)
8:
Obtain one-step prediction: x Go(x,, and xt11 -o( 1)
9:
I update discriminaton
Compute Loss La following Eq.
11:
& & & VLCp
12:
# update Generator
13:
Compute Loss La following Eq.
14:
f < VoLo:
15:
end for
A BROADER IMPACTS
This work presents YOSO, a method that accelerates multi-step large-scale text-to-image diffusion models into one-step generator. On the positive side, while this is academic research, we believe the proposed YOSO can be widely applied in industry. and its high efficiency can lead to energy savings and environmental benefits. However, when these rapid generation models lated by malicious actors, they can also simplify and accelerate the creation of harmful information. Although work focuses scientific research, will take actions to reduce the harmful infor- mation, such as filtering out harmful content in the dataset.
manipu-
B LIMITATION
Our model, like most text-to-image diffusion models, may exhibit shortcomings in terms of fairness, as well as handling specific details and accurately controlling the number targets. We plan to explore these unresolved issues in the generation field our future work, in order enhance the model's capabilities in text generation, fairness, detail control, and quantitative control.
C
PROOF
(x)). Since the divergence is non-negative, and the Po is defined pa, there exist global optimal solution (T)
Proof: The cooperative adversaril loss is defined as: E2 DDa ( (sg(x)))l
Pg (x) = pa (T-1) (x) = ... = (x( such that (Dadv (f(( (sq()) (x)) = O. This com-
pletes the proof.
D TRAINING ALGORITHM
present the algorithm for training YOSO from scratch Algorithm
!
E ADDITIONAL RELATED WORK
Text-to-image Diffusion Models. Since Diffusion models have shown stable training process and are well-suited for scaling up generative models, numerous works have been proposed to extend
DMs for text-to-image generation (Ramesh et al. 2022 Balaji et al. 2022 Rombach et al. 2022
17

---

Published as a conference paper at ICLR 2025
(a) w/o quick adaption.
(b) quick adaption.
Figure 8:
One-step generated images by YOSO-PixArt-a trained with only 5k iterations under different configurations from the same initial noise and prompt.
Xue et al. 2023). Latent DMs (Rombach et al.) 2022 Podell et al., 2023) are widely adopted for high-resolution image and video generation due to their computational efficiency.
F THE EFFECT QUICK-ADAPTION
We present qualitative comparison between YOSO w/o quick adaption and YOSO w/ quick adap- tion in Fig. B Note that the YOSO and variants here are trained with only 5k iterations. And for YOSO w/ quick adaption, we incorporate the 1 iterations used for quick adaptation into the total 5k training iterations to ensure fair comparison. As shown in Fig. g YOSO w/ quick adaption demonstrates significantly better image quality compared to YOSO w/o quick adaption. The gener- ated images of YOSO w/o quick adaption exhibit over-smoothing and severe artifacts. This indicates that the convergence of direct transition to v- v-prediction and zero terminal SNR relatively slower. Additionally. it is worth highlighting that although there only 5k training iterations, the one-step generation quality of YOSO is already reasonable, which demonstrates the effectiveness and quick convergence of the proposed quick adaption stage and YOSO.
G EXPERIMENT SETTING DETAILS
G.1 UNCONDITIONAL GENERATION EXPERIMENTS
For the generator, use the Adam optimizer with 3 0.9 and 3/ = 0.999; for the discriminator, use the Adam optimizer with 31 = and B3 0.999. We adopt constant learning rate of 2e-4 for both the discriminator and the generator in training from scratch. We adopt constant learning rate of 2e-5 for both the discriminator and the generator in fine-tuning. We apply EMA with coefficient of 0.9999 for the generator. We let the A SNR(t) and X
SNR(6) "SR(E-1)
G.2 TEXT-TO-IMAGE EXPERIMENTS
the AdamW optimizer with 31 = 0.9 and 83 0.999; for the discriminator, we use the AdamW optimizer with 3 0. and 3. = 0.999. We adopt a constant learning rate of 2e-5 for both the discriminator and the generator. We apply gradient norm clipping with value of 1.0 for the generator only. We use batch size 256. For full fine-tuning, we apply EMA with coefficient of 0. 0.9999 for the generator. For LoRA fine-tuning, apply EMA with coefficient of 0.999 for
For the generator,
18

---

Published as a conference paper at ICLR 2025
PixArt-a 50 NFE, CF67.5 5 25 steps
PixArt-ō (LCM) NFE, NO CFG 4 steps
YOSO NFE, NO CFG steps
PixArt-a 50 NFE, CF67.5 25 steps
PixArt-8 (LCM) 4 NFE. NO CFG steps
YOSO
NFE, NO CFG steps
anthropomorphic cat wearing fancy gentlemon hat and jacket walking in autumn forest
An astronaut sitting a diner, eating fries, cinematic, analog film
The parametric hotel lobby is sleek and modern space with plenty of natural light.
Pirate ship trapped cosmic maelstrom nebula, volumetric lighting, spectacular, ambient lights.
Spectacular Tiny World in the Transparent Jar On the Table, interior of the Great Hall
A small cactus with hoppy face in the Sahara desert
Figure 9: Additional qualitative comparison on PixArt-a backbone at 1024 resolution.
the generator. Generally, the training is done within 30k iterations for full fine-tuning, while within 5k iterations for LoRA fine-tuning. We let the A, SNR(() and Xco
SNRRET
SNKTE m.]
For full fine-tuning, we train YOSO on the JourneyDB dataset (Pan et al. 2023). by resizing to 512 resolution. And we only use the square image. For LoRA fine-tuning, we use an internally collected dataset and the caption of the JourneyDB dataset (Pan et al.) (023) to generate data for training, we only generate one image for one caption. For the evaluation, we evaluate the HPS score on its benchmark, and we evaluate other metrics based on COCO-5k (Lin et al. 2014) datasets.
G.3 ABLATION STUDY
For the ablation study on CIFAR-10, we use the same UNet and discriminator architecture as used DDGANs (Xiao al. 2022). For the generator, we use Adam optimizer with 31 0.9 and 32 00999; for the discriminator, a constant learning rate of 2e-4 for both the discriminator and the generator. We apply EMA with coefficient of 0.9999 for the generator. We let the X SNR(t) and Amae
use the Adam optimizer with 31 = 0. and 83 = 0.999. We adopt
SNR(E) SNI[C-
H ADDITIONAL EXPERIMENTS
Comparison other Diffusion-Gan Hybrid Methods on High-Resolutton Generation. We con- ducted comparative experiments between methods by fine-tuning pre-trained diffusion models. We trained a lightweight diffusion model in the latent space of DC-AE (Chen et al. (2025) for efficiency. Specifically, following DC-AE, we adopted DīT-S as the model backbone, which is a lightweight model with 33M parameters. For a fair comparison, we initialized the generator for YOSO as well as the other methods using the pre-traie pre-trained diffusion model. As shown in Tab. 6 YOSO achieves better performance on FFHQ- 1024 compared to other diffusion-gan ganninnnnggnnno hybrid distillation methods (Xu
al. 2023c Wang al. 2023a Kang al. 2024
19

---

Published as a conference paper at ICLR 2025
Table Training from scratch on ImageNet-64.
Method
NFE
FID (L)
BigGAN-deep StyleGAN-XL XL
4.06
1
1.52
ADM
250
2.07
EDM
511
1.36
CT
1
13.0
iCT
3,25
YOSO (Ours)
1
2.65
Table 6: Comparison of different diffusion-gan hybrid methods on FFHQ-1024.
Method
NFE
FID (L)
Teacher Diffusion (Our reproduced)
250
13.88
Diffusion-GAN
22.16
Diffusion2GAN
18.01
UFOGen YOSO (Ours)
24.37
16.23
Training from scratch on ImageNet-66. To further demonstrate the ability of YOSO in training from scratch, we conducted additional experiments on ImageNet-64 using the EDM architecture, demonstrating YOSO's effectiveness when trained from seratch. As shown in Tab. [5] method achieves a better FID score compared to other methods, indicating both efficiency and improved performance.
Zero-Shot COCO FID. We argue that zero-shot COCO FID not a good metric for evaluating text-to-image models. Existing work has found that zero-shot COCO FID even be negatively correlated with human preferences in certain cases Kirstain et al. 2023 Podell et al. 2023). We report the metric for completeness. We retrain the YOSO by fine-tuning on COYO-700M and our internal dataset. As shown in Tab. 7 YOSO is capable of achieving a low zero-shot COCO FID-30k of 8 8.90, outperforming prior tex-ge--mmage GANs.
Model
Latency 1
FID-30k 4.
CLIP Score-30k
StyleGAN-T
0.10s
13.90
GigaGAN
0.13s
9.09
SD v1.5 (CFG=7.5)
2.59s
13.45
0.32
SD v1.5 (CFG=3)
2.59s
8.78
0.32
LCM (4-step)
0.26s
23.62
0.30
Diffusion2GAN
0.09s
9.29
UFOGen
0.09s
12.78
InstaFlow-0.9B
0.09s
13.27
0.28
DMD
0.09s
14.93
0.32
SwiftBrush
0.09s
16.67
0.29
YOSO (full-finetunin)
0.09s
8.90
0.31
Table 7: Comparison of zero-shot COCO FID
20

---

Published as a conference paper at ICLR 2025
(a) 1-step samples
(b) 4-step samples
Figure 10: Qualitative comparison among different sampling steps using YOSO-LoRA.
I ADDITIONAL QUALITATIVE COMPARISON
We present an Additional Qualitative comparison to PixArt-a and PixArt-ó on 1024 resolution in As can be observed, our method produces significantly better image quality compared to PixArt-ò (LCM) and achieves comparable results to the multi-step teacher model.
Fig. 9
We present an Additional qualitative comparison among different sampling steps using YOSO- As can observed, the 4-step samples have better visual quality than 1-step
LoRA in Fig. I0
samples
We present an Additional qualitative comparison among 2-step examples of LCM by varying the distance metries in Fig. [ verify the effectiveness of the proposed latent perceptual loss. In particular, we mainly vary the feature layer for computing the latent perceptual loss. can be that using the bottleneck layer delivers better visual quality. More importantly, regardless of which layers are used to compute LPL, it is significantly better than MSE. This highlights the elfectiveness of proposed latent perceptual loss.
21

---

Published as a conference paper at ICLR 2025
A dog
A cat
Snowy Mountain
A Tiger
LCM, MSE
LCM, Latent Perceptual Loss, Encoder Features
LCM, Latent Perceptual Loss, Decoder Features
LCM, Latent Perceptual Loss, All Features
LCM, Latent Perceptual Loss, Bottleneck Features
Figure 11: Additional qualitative comparison of 2-step examples varying the distance metric in LCM.
22


Reviews:
[{"review_id": "B1QfcKLemc", "reviewer": "Reviewer_gStt", "summary": "This paper presents YOSO (You Only Sample Once), a novel approach for high-quality, one-step text-to-image generation that enhances training stability and efficiency by combining elements from diffusion and GAN models. YOSO’s design includes Self-Cooperative Learning, where less noisy “teacher” samples guide more noisy “student” samples to stabilize training, along with a Decoupled Scheduler that applies separate optimization schedules to Adversarial Loss and Consistency Loss, improving sample quality. Additionally, Informative Prior Initialization (IPI) provides a more data-informed starting point for faster convergence and reduced artifacts, while Quick Adaptation facilitates smooth adjustments to v-prediction and zero terminal SNR, essential for stable, one-step sampling. Unlike prior models that require multiple steps, YOSO achieves single-step synthesis with refined adversarial learning, using teacher-student distributions within the model to mitigate instability and mode collapse risks. This efficient and stable framework positions YOSO as a significant advancement for rapid, high-fidelity text-to-image generation.", "questions": "Questions are embedded in the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The proposed self-cooperative technique to train a one-step sampler seems novel in the diffusion literature, and the author proved it to be very effective. The fact that the less noisy (teacher) samples guide more noisy (student) samples during training is quite interesting.\n\n2. It is interesting to see the connection between the self-cooperative objective and consistency models.\n\n3. The experimental results are quite strong. They outperform other one-step baseline models on CIFAR-10 and also show improvement in the large-scale text-to-image setting.\n\n4. The mathematical analysis is clear and robust, which supports each of its contributions. The derivations for Self-Cooperative Learning and the teacher-student dynamic seems well-structured.", "weaknesses": "1. Lack of the ablation study on training techniques. The paper introduces training techniques like Informative Prior Initialization (IPI) and v-prediction, which are valuable training methods that could also benefit other models. However, the lack of an ablation study on these components makes it difficult to assess their specific impact and to directly compare YOSO's performance with other models that could potentially leverage these techniques.\n\n2. Hyperparameter sensitivity. The timestep difference between the teacher and student latents, a key factor in the Decoupled Scheduler, is a hyperparameter that directly impacts training stability and eventual performance. Does it need to be tuned for different datasets? Could you show how robust the method is to this hyperparameter? \n\n[minor comments]\n1. Table 2 caption: text-to-to -> text-to\n2. Eq (3): remove ‘x’\n3. In Eq (3): why did you use different lambda for different t?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FGd0AzKL9b", "reviewer": "Reviewer_tJNk", "summary": "This paper combines diffusion models and GANs to propose the YOSO framework for one-step image generation. It employs various training strategies to enhance training stability and has been validated on both unconditional generation and text-to-image tasks.", "questions": "- Why were only SD-LoRA and PixArt-Full experiments conducted in the experimental setup, without including SD-Full and PixArt-LoRA related experiments?\n\n- I am curious about how the zero-shot one-step 1024 resolution is effective. As far as I know, 1024 has a different ar_size compared to 512, and the posemb scale is also different. Wouldn't these factors significantly impact the weight distribution of the 1024 model?\n\n- Without retraining, will increasing the sampling steps, similar to LCM, improve the results? If further enhancements are desired for better generated image quality, what would be the subsequent optimization directions based on the proposed approach?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- In terms of qualitative metrics, YOSO demonstrates certain advantages across various tasks and metrics, and it is capable of transferring to higher resolutions, image editing, and conditional generation tasks.\n\n- The simple and effective training strategies proposed during practical implementation have certain reference significance for generative tasks.\n\n- The paper shows many analyses and easy to follow.", "weaknesses": "- The presentation of the visualization results in this paper is somewhat insufficient, including a comparison with YOSO-PixArt-alpha and other Few-Step models, such as PixArt-delta, and it might be worth attempting to compare with the original models as well.\n\n- The clarity of the narrative in this paper needs improvement; currently, the narrative logic resembles an experimental report. When encountering issues in a new task (unconditional and text-to-image generation / finetuning) or a new model (U-Net or DiT based), a specific strategy is proposed to address the problem. This approach may not be conducive to the coherence expected in a complete paper.\n\n- Regarding writing:\n  - The abbreviation \"NEF\" is not explained.\n  - Should the steps for Hyper-SD-LoRA and YOSO-LoRA in Table 2 be equal to 1?\n  - Figure 5 does not specify the comparative configuration for model inference—does it keep the prompt unchanged while changing the same seed?\n  - The reference formatting in the paper should be kept consistent as possible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E7qWu6nYdm", "reviewer": "Reviewer_RjVy", "summary": "This paper introduces YOSO, a novel generative model designed for rapid, scalable, and high-fidelity one-step image synthesis with high training stability and mode coverage. Specifically, the paper smooths the adversarial divergence by the denoising generator itself, performing self-cooperative learning. This method can serve as a one-step generation model training from scratch with competitive performance. Moreover, the authors extend YOSO to one-step text-to-image generation based on pre-trained models by several effective training techniques (i.e., latent perceptual loss and latent discriminator for efficient training along with the latent DMs; the informative prior initialization (IPI), and the quick adaption stage for fixing the flawed noise scheduler). Experimental results show that YOSO achieves state-of-the-art one-step generation performance even with Low-Rank Adaptation (LoRA) fine-tuning.", "questions": "See Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The article is well written and easy to understand.\n2. The visualization results look very good.", "weaknesses": "1. There is a lack of important references in the Background, such as [1,2,3], which are all about combining Diffusion models and GANs.  The differences between the proposed method and these methods [1,2,3] need to be emphasized and discussed.\n2. In the experimental part, it is also necessary to compare with these methods [1,2,3].\n3. There is no comparison with the current SOTA methods in terms of model complexity (such as the number of network parameters and training inference time).\n4. In the TEXT-TO-IMAGE GENERATION experiment, there is no comparison with the GAN-based methods, such as [4]. It is also worth discussing whether the proposed method is better or more efficient than the GAN-based methods.\n\n[1]Wang, Zhendong, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. \"Diffusion-gan: Training gans with diffusion.\" arXiv preprint arXiv:2206.02262 (2022).\n[2]Kang, Minguk, Richard Zhang, Connelly Barnes, Sylvain Paris, Suha Kwak, Jaesik Park, Eli Shechtman, Jun-Yan Zhu, and Taesung Park. \"Distilling Diffusion Models into Conditional GANs.\" ECCV (2024).\n[3]Xu, Yanwu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. \"Ufogen: You forward once large scale text-to-image generation via diffusion gans.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8196-8206. 2024.\n[4]Kang, Minguk, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. \"Scaling up gans for text-to-image synthesis.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10124-10134. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3g8QhP1Uw7", "reviewer": "Reviewer_SYJU", "summary": "This paper, \"You Only Sample Once: Taming One-Step Text-to-Image Synthesis by Self-Cooperative Diffusion GANs (YOSO)\", proposes a novel generative method that merges diffusion models (DMs) with GANs to perform efficient, high-quality one-step text-to-image synthesis. YOSO leverages a self-cooperative learning framework, where a generator smooths adversarial divergence by referencing self-generated denoised samples, combined with a latent perceptual loss and a decoupled scheduler to stabilize training. The paper positions YOSO as an efficient alternative to multi-step latent diffusion methods (LDMs) with discriminator-based losses, aiming to achieve comparable quality while reducing computational overhead. Experimental results demonstrate competitive performance with state-of-the-art methods, though the model would benefit from clearer differentiation from similar single-step latent diffusion approaches.", "questions": "Please refer to the weakness section", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Creative Approach to Stability in One-Step Synthesis: YOSO’s self-cooperative learning framework is an inventive attempt to stabilize adversarial training by smoothing divergence via self-generated data, a strategy distinct from standard GAN-DM hybrids. This approach may address some instability challenges faced by prior one-step latent diffusion models.\n2. Efficient Resource Utilization: YOSO’s compatibility with LoRA allows it to produce high-resolution images without the significant computational burden typically associated with GAN-based models, making it adaptable for resource-constrained settings.", "weaknesses": "1. Clarity of Key Contribution: This paper lacks clarity on whether YOSO is a new model architecture that can be trained from scratch or a step distillation methodology that should be applied on pre-trained models. Further elaboration, possibly with additional diagrams, would make the paper more accessible and clear.\n\n2. Experimental Scope and Robustness: This paper introduces YOSO both as a new model and a novel distillation techniques for pre-trained models. However, the former part was only evaluated on the CIFAR-10 dataset, additional benchmarks are necessary to substantiate its effectiveness across more diverse and complex datasets.\n\n2. Novelty in Methodology: Besides self-cooperative learning, the adoption of decoupled scheduling, perceptual and consistency loss isn't novel to the field, and while the rationale is sound, these sections could benefit from more detailed comparison with previous works and how it is different from them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2dvPlWlBEJ", "reviewer": "Reviewer_Awjz", "summary": "The paper proposes a novel one-step generative model named YOSO, which offers higher training stability and model coverage compared to existing methods. This approach aims to achieve high-quality generation while maintaining performance metrics in a single step. Experiments are conducted on CIFAR-10 using HPS and AeS evaluation metrics.", "questions": "1. The authors did not provide information about the dataset used for the test metrics in Table 2.  Although coco2014 is mentioned in the appendix, I am not certain if it is the dataset used for these metrics.\n2. The ablation study is unconvincing, as the authors used CIFAR-10 for validation. However, CIFAR-10 has a low resolution and only consists of 10 simple categories, lacking the richness of text prompts. It would be better to conduct the ablation study on the COCO 2014 and COCO 2017 datasets, similar to existing one-step text-to-image generative methods (e.g., InstaFlow, SD-Turbo, SwiftBrush).", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Applying the Latent Discriminator leads to a stable training process with fast convergence.\n2. The paper tackles a relevant application (i.e., ControlNet) with one-step.", "weaknesses": "1. As mentioned in the paper, \"It is hard to extend GANs on large-scale datasets due to training challenges,\" yet there are existing works that successfully train GANs with large-scale datasets (e.g., GigaGAN with 2.7 billion training image-prompt pairs, Diffusion2GAN with 12 million training image-prompt pairs). It is important to compare these methods. While GigaGAN and Diffusion2GAN do not provide source code, the authors can still conduct comparisons using the COCO 2014 dataset, as GigaGAN provides inference results for this dataset.\n\nGigaGAN: Scaling up GANs for Text-to-Image Synthesis (CVPR 2023)\n\nDiffusion2GAN: Distilling Diffusion Models into Conditional GANs.\n\n2. Why use the bottleneck layer of the UNet to compute the latent perceptual loss? Some works (e.g., DIFT, PnP, FasterDiffusion) suggest that encoder features are more important than both the encoder and bottleneck features.\n\nDIFT: Emergent Correspondence from Image Diffusion (NeurIPS'23)\n\nPnP: Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation (CVPR 2023)\n\nFasterDiffusion: Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS'24)\n\n3. It’s better to provide experiments comparing with exiting SOTA one-step generative methods (e.g., SwiftBrush) to prove that YOSO enhances the inference efficiency and high-quality generation.\n\nSwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation (CVPR'24)\n\n4. The effectiveness of IPI should be validated on datasets like COCO 2014 using FID and CLIPscore to avoid cherry-picking. FID and CLIPscore is wildy used in one-step generation methods like SD-Turbo, InstaFlow, SwiftBrush.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)