You Only Sample Once: Taming One-Step Text-to-Image Synthesis by Self-Cooperative Diffusion GANs

Authors: Yihong Luo, Xiaolong Chen, Xinghua Qu, Tianyang Hu, Jing Tang
Primary area: generative models
Keywords: One-step text-to-image generation; Diffusion Models; Generative Adversarial Networks

Abstract:
Recently, some works have tried to combine diffusion and Generative Adversarial Networks (GANs) to alleviate the computational cost of the iterative denoising inference in Diffusion Models (DMs). 
However, existing works in this line suffer from either training instability and mode collapse or subpar one-step generation learning efficiency. 
To address these issues, we introduce YOSO, a novel generative model designed for rapid, scalable, and high-fidelity one-step image synthesis with high training stability and mode coverage. 
Specifically, we smooth the adversarial divergence by the denoising generator itself, performing self-cooperative learning. We show that our method can serve as a one-step generation model training from scratch with competitive performance. 
Moreover, we extend our YOSO to one-step text-to-image generation based on pre-trained models by several effective training techniques (i.e., latent perceptual loss and latent discriminator for efficient training along with the latent DMs; the informative prior initialization (IPI), and the quick adaption stage for fixing the flawed noise scheduler). Experimental results show that YOSO achieves the state-of-the-art one-step generation performance even with Low-Rank Adaptation (LoRA) fine-tuning.
In particular, we show that the YOSO-PixArt-$\alpha$ can generate images in one step trained on 512 resolution, with the capability of adapting to 1024 resolution without extra explicit training, requiring only \textasciitilde10 A800 days for fine-tuning. Our code is available at:  [https://github.com/Luo-Yihong/YOSO](https://github.com/Luo-Yihong/YOSO)

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730040396722

thread_type:
official_review

<thread_object>
creator: reviewer_awjz

<content>
summary:
The paper proposes a novel one-step generative model named YOSO, which offers higher training stability and model coverage compared to existing methods. This approach aims to achieve high-quality generation while maintaining performance metrics in a single step. Experiments are conducted on CIFAR-10 using HPS and AeS evaluation metrics.

strengths:
1. Applying the Latent Discriminator leads to a stable training process with fast convergence.
2. The paper tackles a relevant application (i.e., ControlNet) with one-step.

weaknesses:
1. As mentioned in the paper, "It is hard to extend GANs on large-scale datasets due to training challenges," yet there are existing works that successfully train GANs with large-scale datasets (e.g., GigaGAN with 2.7 billion training image-prompt pairs, Diffusion2GAN with 12 million training image-prompt pairs). It is important to compare these methods. While GigaGAN and Diffusion2GAN do not provide source code, the authors can still conduct comparisons using the COCO 2014 dataset, as GigaGAN provides inference results for this dataset.

GigaGAN: Scaling up GANs for Text-to-Image Synthesis (CVPR 2023)

Diffusion2GAN: Distilling Diffusion Models into Conditional GANs.

2. Why use the bottleneck layer of the UNet to compute the latent perceptual loss? Some works (e.g., DIFT, PnP, FasterDiffusion) suggest that encoder features are more important than both the encoder and bottleneck features.

DIFT: Emergent Correspondence from Image Diffusion (NeurIPS'23)

PnP: Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation (CVPR 2023)

FasterDiffusion: Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS'24)

3. It’s better to provide experiments comparing with exiting SOTA one-step generative methods (e.g., SwiftBrush) to prove that YOSO enhances the inference efficiency and high-quality generation.

SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation (CVPR'24)

4. The effectiveness of IPI should be validated on datasets like COCO 2014 using FID and CLIPscore to avoid cherry-picking. FID and CLIPscore is wildy used in one-step generation methods like SD-Turbo, InstaFlow, SwiftBrush.

questions:
1. The authors did not provide information about the dataset used for the test metrics in Table 2.  Although coco2014 is mentioned in the appendix, I am not certain if it is the dataset used for these metrics.
2. The ablation study is unconvincing, as the authors used CIFAR-10 for validation. However, CIFAR-10 has a low resolution and only consists of 10 simple categories, lacking the richness of text prompts. It would be better to conduct the ablation study on the COCO 2014 and COCO 2017 datasets, similar to existing one-step text-to-image generative methods (e.g., InstaFlow, SD-Turbo, SwiftBrush).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730251085315

thread_type:
official_review

<thread_object>
creator: reviewer_syju

<content>
summary:
This paper, "You Only Sample Once: Taming One-Step Text-to-Image Synthesis by Self-Cooperative Diffusion GANs (YOSO)", proposes a novel generative method that merges diffusion models (DMs) with GANs to perform efficient, high-quality one-step text-to-image synthesis. YOSO leverages a self-cooperative learning framework, where a generator smooths adversarial divergence by referencing self-generated denoised samples, combined with a latent perceptual loss and a decoupled scheduler to stabilize training. The paper positions YOSO as an efficient alternative to multi-step latent diffusion methods (LDMs) with discriminator-based losses, aiming to achieve comparable quality while reducing computational overhead. Experimental results demonstrate competitive performance with state-of-the-art methods, though the model would benefit from clearer differentiation from similar single-step latent diffusion approaches.

strengths:
1. Creative Approach to Stability in One-Step Synthesis: YOSO’s self-cooperative learning framework is an inventive attempt to stabilize adversarial training by smoothing divergence via self-generated data, a strategy distinct from standard GAN-DM hybrids. This approach may address some instability challenges faced by prior one-step latent diffusion models.
2. Efficient Resource Utilization: YOSO’s compatibility with LoRA allows it to produce high-resolution images without the significant computational burden typically associated with GAN-based models, making it adaptable for resource-constrained settings.

weaknesses:
1. Clarity of Key Contribution: This paper lacks clarity on whether YOSO is a new model architecture that can be trained from scratch or a step distillation methodology that should be applied on pre-trained models. Further elaboration, possibly with additional diagrams, would make the paper more accessible and clear.

2. Experimental Scope and Robustness: This paper introduces YOSO both as a new model and a novel distillation techniques for pre-trained models. However, the former part was only evaluated on the CIFAR-10 dataset, additional benchmarks are necessary to substantiate its effectiveness across more diverse and complex datasets.

2. Novelty in Methodology: Besides self-cooperative learning, the adoption of decoupled scheduling, perceptual and consistency loss isn't novel to the field, and while the rationale is sound, these sections could benefit from more detailed comparison with previous works and how it is different from them.

questions:
Please refer to the weakness section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730532752783

thread_type:
official_review

<thread_object>
creator: reviewer_rjvy

<content>
summary:
This paper introduces YOSO, a novel generative model designed for rapid, scalable, and high-fidelity one-step image synthesis with high training stability and mode coverage. Specifically, the paper smooths the adversarial divergence by the denoising generator itself, performing self-cooperative learning. This method can serve as a one-step generation model training from scratch with competitive performance. Moreover, the authors extend YOSO to one-step text-to-image generation based on pre-trained models by several effective training techniques (i.e., latent perceptual loss and latent discriminator for efficient training along with the latent DMs; the informative prior initialization (IPI), and the quick adaption stage for fixing the flawed noise scheduler). Experimental results show that YOSO achieves state-of-the-art one-step generation performance even with Low-Rank Adaptation (LoRA) fine-tuning.

strengths:
1. The article is well written and easy to understand.
2. The visualization results look very good.

weaknesses:
1. There is a lack of important references in the Background, such as [1,2,3], which are all about combining Diffusion models and GANs.  The differences between the proposed method and these methods [1,2,3] need to be emphasized and discussed.
2. In the experimental part, it is also necessary to compare with these methods [1,2,3].
3. There is no comparison with the current SOTA methods in terms of model complexity (such as the number of network parameters and training inference time).
4. In the TEXT-TO-IMAGE GENERATION experiment, there is no comparison with the GAN-based methods, such as [4]. It is also worth discussing whether the proposed method is better or more efficient than the GAN-based methods.

[1]Wang, Zhendong, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. "Diffusion-gan: Training gans with diffusion." arXiv preprint arXiv:2206.02262 (2022).
[2]Kang, Minguk, Richard Zhang, Connelly Barnes, Sylvain Paris, Suha Kwak, Jaesik Park, Eli Shechtman, Jun-Yan Zhu, and Taesung Park. "Distilling Diffusion Models into Conditional GANs." ECCV (2024).
[3]Xu, Yanwu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. "Ufogen: You forward once large scale text-to-image generation via diffusion gans." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8196-8206. 2024.
[4]Kang, Minguk, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. "Scaling up gans for text-to-image synthesis." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10124-10134. 2023.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649677116

thread_type:
official_review

<thread_object>
creator: reviewer_tjnk

<content>
summary:
This paper combines diffusion models and GANs to propose the YOSO framework for one-step image generation. It employs various training strategies to enhance training stability and has been validated on both unconditional generation and text-to-image tasks.

strengths:
- In terms of qualitative metrics, YOSO demonstrates certain advantages across various tasks and metrics, and it is capable of transferring to higher resolutions, image editing, and conditional generation tasks.

- The simple and effective training strategies proposed during practical implementation have certain reference significance for generative tasks.

- The paper shows many analyses and easy to follow.

weaknesses:
- The presentation of the visualization results in this paper is somewhat insufficient, including a comparison with YOSO-PixArt-alpha and other Few-Step models, such as PixArt-delta, and it might be worth attempting to compare with the original models as well.

- The clarity of the narrative in this paper needs improvement; currently, the narrative logic resembles an experimental report. When encountering issues in a new task (unconditional and text-to-image generation / finetuning) or a new model (U-Net or DiT based), a specific strategy is proposed to address the problem. This approach may not be conducive to the coherence expected in a complete paper.

- Regarding writing:
  - The abbreviation "NEF" is not explained.
  - Should the steps for Hyper-SD-LoRA and YOSO-LoRA in Table 2 be equal to 1?
  - Figure 5 does not specify the comparative configuration for model inference—does it keep the prompt unchanged while changing the same seed?
  - The reference formatting in the paper should be kept consistent as possible.

questions:
- Why were only SD-LoRA and PixArt-Full experiments conducted in the experimental setup, without including SD-Full and PixArt-LoRA related experiments?

- I am curious about how the zero-shot one-step 1024 resolution is effective. As far as I know, 1024 has a different ar_size compared to 512, and the posemb scale is also different. Wouldn't these factors significantly impact the weight distribution of the 1024 model?

- Without retraining, will increasing the sampling steps, similar to LCM, improve the results? If further enhancements are desired for better generated image quality, what would be the subsequent optimization directions based on the proposed approach?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708388921

thread_type:
official_review

<thread_object>
creator: reviewer_gstt

<content>
summary:
This paper presents YOSO (You Only Sample Once), a novel approach for high-quality, one-step text-to-image generation that enhances training stability and efficiency by combining elements from diffusion and GAN models. YOSO’s design includes Self-Cooperative Learning, where less noisy “teacher” samples guide more noisy “student” samples to stabilize training, along with a Decoupled Scheduler that applies separate optimization schedules to Adversarial Loss and Consistency Loss, improving sample quality. Additionally, Informative Prior Initialization (IPI) provides a more data-informed starting point for faster convergence and reduced artifacts, while Quick Adaptation facilitates smooth adjustments to v-prediction and zero terminal SNR, essential for stable, one-step sampling. Unlike prior models that require multiple steps, YOSO achieves single-step synthesis with refined adversarial learning, using teacher-student distributions within the model to mitigate instability and mode collapse risks. This efficient and stable framework positions YOSO as a significant advancement for rapid, high-fidelity text-to-image generation.

strengths:
1. The proposed self-cooperative technique to train a one-step sampler seems novel in the diffusion literature, and the author proved it to be very effective. The fact that the less noisy (teacher) samples guide more noisy (student) samples during training is quite interesting.

2. It is interesting to see the connection between the self-cooperative objective and consistency models.

3. The experimental results are quite strong. They outperform other one-step baseline models on CIFAR-10 and also show improvement in the large-scale text-to-image setting.

4. The mathematical analysis is clear and robust, which supports each of its contributions. The derivations for Self-Cooperative Learning and the teacher-student dynamic seems well-structured.

weaknesses:
1. Lack of the ablation study on training techniques. The paper introduces training techniques like Informative Prior Initialization (IPI) and v-prediction, which are valuable training methods that could also benefit other models. However, the lack of an ablation study on these components makes it difficult to assess their specific impact and to directly compare YOSO's performance with other models that could potentially leverage these techniques.

2. Hyperparameter sensitivity. The timestep difference between the teacher and student latents, a key factor in the Decoupled Scheduler, is a hyperparameter that directly impacts training stability and eventual performance. Does it need to be tuned for different datasets? Could you show how robust the method is to this hyperparameter? 

[minor comments]
1. Table 2 caption: text-to-to -> text-to
2. Eq (3): remove ‘x’
3. In Eq (3): why did you use different lambda for different t?

questions:
Questions are embedded in the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732493421165

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank the reviewers for their time and thoughtful feedback on our work. 

In this work, we introduce YOSO, a novel framework for combining diffusion models and GANs via the proposed self-cooperative adversarial learning. Our work demonstrates strong experimental results and wide application. We appreciate the reviewers' acknowledgment. Specifaclly, "The proposed self-cooperative technique to train a one-step sampler seems novel in the diffusion literature, and the author proved it to be very effective" (Reviewer gStt), "YOSO demonstrates certain advantages across various tasks and metrics, and it is capable of transferring to higher resolutions, image editing, and conditional generation tasks." (Reviewer tJNk), "The visualization results look very good" (Reviewer RjVy), "YOSO’s compatibility with LoRA allows it to produce high-resolution images without the significant computational burden typically associated with GAN-based models" (Reviewer SYJU), YOSO "tackles a relevant application (i.e., ControlNet) with one-step" (Reviewer Awjz).

In what follows, following the reviewers' suggestions and comments, we summarize additional experiments we have conducted to address some common concerns, which will be incorporated to the revision.

1. **Ablation Studies and Training Techniques**:  
   As suggested by the reviewers, there is a need for ablation studies to assess the impact of specific training techniques, including Informative Prior Initialization (IPI), v-prediction, the Decoupled Scheduler, and Latent Perceptual Loss. We emphasize that comprehensive ablation studies were conducted and included in the original submission (Table 4) on text-to-image generation. Additionally, we have performed new experiments to further validate the effectiveness of our proposed techniques, such as the quick adaptation to v-prediction and varying timestep differences in the cooperative adversarial loss. These results consistently demonstrate the robustness and performance improvements provided by our methods.

**Table A.** Ablation on timestep difference. The setting is using LoRA for fine-tuning SD 1.5 with 3k iterations. 
| Model | Skip | HPS ↑ | AeS ↑ |
|--------|------|-----|-----|
| YOSO | 150 | 25.94 | 5.89 |
| YOSO | 200 | 26.37 | 6.03 |
| YOSO | 250 | 26.04 | 5.91 |
| YOSO | 300 | 25.88 | 5.87 |

**Table B.** Ablation on the model parametrization. The setting is fine-tuning PixArt-$\alpha$ with 5k iterations. 
|Method |HPS ↑| AeS ↑|
|--------|------|-----|
|$\epsilon$-prediction | 18.03 | 4.93|
|naive  v-prediction | 22.37 | 5.62|
|quick adpation to v-prediction | 25.39 | 6.01|
  
2. **Comparison to advanced Diffusion-GAN hybrid Models** 
We provide an additional comparison to advanced Diffusion-GAN hybrid Models on CIFAR-10. The results indicate the clear advantage of our proposed YOSO.
| Model | FID↓ |
|-------|--------|
| Diffusion-GAN  | 3.19 |
| Diffusion2GAN  | 3.16 |
| YOSO (Ours) | 1.81 |

3. **Training YOSO from scratch on ImageNet-64**  
To further demostrate the ability of YOSO in training from scratch, we conducted additional experiments on ImageNet-64 using the EDM architecture, demonstrating YOSO's effectiveness when trained from scratch.
  
| Method | NFE | FID (↓) |
|--------|---------------|---------|
| BigGAN-deep  | 1 | 4.06 |
| StyleGAN-XL | 1 | 1.52 |
| ADM  | 250 | 2.07 |
| EDM | 511 | 1.36 |
| CT | 1 | 13.0 |
| iCT  | 1 | 3.25 |
| YOSO （Ours）  | 1 | 2.65 |

4. **Zero-shot FID on COCO**:  
   While we argue that zero-shot FID on COCO has limitations as a metric for evaluating modern text-to-image models (Pick-a-Pic [a] found that zero-shot COCO FID can be negatively correlated with human preferences in certain cases), we provided these results for completeness. Our method demonstrates competitive performance compared to existing SOTA models in terms of FID and CLIP Score, further validating its effectiveness.

| Model | Latency ↓ | FID-30k ↓ | CLIP Score-30k ↑ |
|--------|---------|--------|---------|
| **GANs** | | | |
|  StyleGAN-T | 0.10s | 13.90 | - | 
|  GigaGAN | 0.13s | 9.09 | - |
| **Oringal Diffusion** | | | |
| SD v1.5 | 2.59s | 13.45 | 0.32 |
| **Diffusion Distillation** | | | |
| LCM (4-step) | 0.26s | 23.62 | 0.30 |
| Diffusion2GAN | 0.09s | 9.29 | - |
| UFOGen | 0.09s | 12.78 | - |
| InstaFlow-0.9B | 0.09s | 13.27 | 0.28 |
| DMD | 0.09s | 14.93 | 0.32 |
| SwiftBrush | 0.09s | 16.67 | 0.29 |
| YOSO (Ours) | 0.09s | 12.35 | 0.31 | 
| YOSO（Ours, retrained, full fine-tuned, selected by FID）  | 0.09s | 9.61 | 0.31 |

[a] Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation, NeurIPS 2023.

We appreciate the reviewers' constructive comments, which have allowed us to improve the clarity and comprehensiveness of our work. We believe that our additional experiments and clarifications address the concerns raised and further demonstrate the robustness, efficiency, and effectiveness of YOSO. Thank you again for your insightful feedback!
</content>
</thread_object>

</thread>
<thread>
creation date:
1733225707550

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response Summary

comment:
We thank the area chair and all reviewers for your time, insightful suggestions, and valuable comments. Your suggestions have been invaluable in refining our work, and we deeply appreciate the time and effort you dedicated to reviewing our paper. We have carefully addressed all points in our response.

We are encouraged by the reviewers’ positive feedback on various aspects of our work:
* Reviewer gStt: "The proposed self-cooperative technique to train a one-step sampler seems novel in the diffusion literature, and the author proved it to be very effective".
* Reviewer tJNk: "YOSO demonstrates certain advantages across various tasks and metrics, and it is capable of transferring to higher resolutions, image editing, and conditional generation tasks".
* Reviewer RjVy: "The visualization results look very good".
* Reviewer SYJU: "YOSO’s compatibility with LoRA allows it to produce high-resolution images without the significant computational burden typically associated with GAN-based models".
* Reviewer Awjz: YOSO "tackles a relevant application (i.e., ControlNet) with one-step".

We are also pleased that our clarifications and additional experiments have been well-received:
* Reviewer gStt: "I think this paper meets the standard for acceptance in ICLR."
* Reviewer SYJU: "I have no further questions and raised my score."
* Reviewer tJNk: "It has addressed most of my concerns, and I will raise my score."

**Additional Experiments and Improvements**

We also sincerely thank the reviewers for their valuable suggestions, which helped us identify areas for improvement. In response to their feedback, we have made several additional experiments to further strengthen our work:

* **Ablation Studies and Training Techniques** (Reviewer gStt): We performed experiments validating the quick adaptation to v-prediction and varying timestep differences in the cooperative adversarial loss. Results consistently demonstrate the robustness and performance improvements of our methods.
* **Comparison to advanced Diffusion-GAN hybrid Models** (Reviewer RjVy): We provide an additional comparison to advanced Diffusion-GAN hybrid Models on CIFAR-10 and high-resolution dataset FFHQ-1024. The results indicate the clear advantage of our proposed YOSO.
* **Training YOSO from scratch on ImageNet-64** (Reviewer SYJU): To further demonstrate the ability of YOSO in training from scratch, we conducted additional experiments on ImageNet-64. Results show that YOSO achieves comparable performance to state-of-the-art diffusion models and GANs.
* **Super-Resolution (128→512)** (Reviewer RjVy): Experiments demonstrate that YOSO achieves comparable performance to SD upscaler with a ~50x speedup.
* **Zero-shot FID on COCO** (Reviewer Awjz, RjVy): We have provided comparison regarding zero-shot COCO FID. The results show that YOSO achieves competitive performance compared to existing state-of-the-art models. We also observe that YOSO is capable of achieving a lower Zero-shot COCO FID of 8.90 compared to GigaGAN by longer training.
* **Layer Selection for Latent Perceptual Loss (LPL)** (Reviewer Awjz): We follow the reviewers' advice and provide more detailed analysis and experiments, additional ablation studies confirmed that using bottleneck features for LPL consistently outperforms other layer configurations. Moreover, regardless of which layers are used to compute LPL, it is significantly better than MSE. 

We believe our additional experiments and clarifications comprehensively address the remaining concerns raised by Reviewer RjVy (regarding additional high-resolution comparison and zero-shot COCO FID) and Reviewer Awjz (regarding layer selection in LPL). Additionally, we explored accelerating SD upscaler using YOSO, addressing the Reviewer RjVy’s interest in this aspect.

We thank all the reviewers again for their time and effort in reviewing our paper and are committed to addressing every issue raised.

Sincerely,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a framework for efficient and high-quality one-step image generation by combining diffusion models with GANs. The authors propose several techniques, including self-cooperative learning, a decoupled scheduler, and informative prior initialization. The paper demonstrates promising results on multiple benchmark datasets.

Following the authors-reviewers discussion, two reviewers increased their ratings. However, other reviewers leaning toward rejection with concerns about: 1) high-resolution generation on certain datasets, 2) inferior FID results, and 3) insufficient ablations to understand the optimal layers/features used for optimization. Subsequently, the authors provided additional responses addressing these three concerns.

Given the promising results, the paper is recommended for acceptance. However, the authors should include the discussions with reviewers in the final version.