Does Diffusion Beat GAN in Image Super Resolution?

Authors: Denis Kuznedelev, Valerii Startsev, Daniil Shlenskii, Dmitry V. Dylov, Sergey Kastryulin
Primary area: generative models
Keywords: Image Super-Resolution, GANs, Diffusion Models, Generative Models, Deep Learning

Abstract:
There is a prevalent opinion that diffusion-based models outperform GAN-based counterparts in the Image Super Resolution (ISR) problem. However, in most studies, diffusion-based ISR models employ larger networks and are trained longer than the GAN baselines. This raises the question of whether the high performance stems from the superiority of the diffusion paradigm or if it is a consequence of the increased scale and the greater computational resources of the contemporary studies. In our work, we thoroughly compare diffusion-based and GAN-based super resolution models under controlled settings, with both approaches having matched architecture, model and dataset sizes, and computational budget. We show that a GAN-based model can achieve results comparable or superior to a diffusion-based model. Additionally, we explore the impact of popular design choices, such as text conditioning and augmentation on the performance of ISR models, showcasing their effect in several downstream tasks.

Decision: reject
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729751267646

thread_type:
official_review

<thread_object>
creator: reviewer_pcgz

<content>
summary:
This paper explores the challenge of conducting a fair comparison between GAN and diffusion models in image super-resolution. It notes that diffusion models often employ larger networks and longer training times than GANs, raising questions about whether their performance is due to inherent advantages or increased resources. Through controlled comparisons, matching architecture, dataset size, and computational budget, the study observes that GANs can achieve results comparable or superior to diffusion models. It also examines factors like text conditioning and data augmentation, finding that commonly used image captions do not significantly impact performance.

strengths:
1. The authors provide valuable insight into the current research trend, highlighting the need for fair comparisons between diffusion and GAN models. 
2. They conduct several experiments, considering computational budget, dataset size, and notably, text conditioning as input prompts.
3. The authors incorporate human perception to regulate training, ensuring a fair comparison by controlling the training duration.
4. The experiments are sound, with the authors aiming to maintain consistent conditions, such as using DPM-Solver++ to control the inference steps.

weaknesses:
1. The presentation of this paper could be enhanced with more figures to better illustrate the concepts and experiments, especially those related to text-conditioning.
2. The comparison of super-resolved images in the manuscript and supplementary materials lacks numerical metrics, making evaluation challenging.
3. The current conclusion is somewhat vague. While the authors conduct various ablations between GAN and diffusion, it remains unclear under which specific conditions GAN can truly outperform diffusion, thus it needs to be clarified and streamlined.

questions:
1. The authors use Imagen as the architecture for both GAN and diffusion models in their SR experiments. I question whether it is appropriate to use Imagen for the GAN model, given it wasn't designed for GAN-based SR.
2. My major concern is that the experiments aren't entirely equivalent. Although the authors attempt to balance factors, the GAN model requiring an additional discriminator compared to the diffusion model, the training stability and results heavily depend on this discriminator. It's challenging to ensure true equality between GAN-based methods and diffusion models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729976669210

thread_type:
official_review

<thread_object>
creator: reviewer_cal4

<content>
summary:
This paper investigates whether diffusion-based models truly outperform GAN-based models for the task of Image Super Resolution (ISR) in a fair setup. The authors conduct a rigorous comparison between diffusion and GAN-based models under controlled experimental settings, ensuring that both model types have equivalent architectures, dataset sizes, and computational budgets. In contrast with common belief, the primary findings reveal that GAN-based models can match or even surpass diffusion models in terms of quality when appropriately scaled. The paper also finds that text conditioning has little effect on model performance, while augmentations can sometimes hurt the training of diffusion models.

strengths:
1. The main contribution of this paper is a fair comparison between GAN and diffusion-based ISR models, controlling for architecture, dataset size, and computational resources.

2. The authors perform detailed ablation studies, particularly focusing on the effects of pretraining, augmentations, and training with full-resolution images.

3. The paper explores not only the overall performance of the models but also the impact of various design choices such as text conditioning and augmentation, which can be helpful for future work in the field.

weaknesses:
1. The contribution is a bit limited as there is no really new and impactful insights presented in this work. Additionally, I'm not sure ICLR is a suitable venue for submitting this work, because it lean toward to more empirical side. May be MLSys is a better venue ? Again, I'm not sure.

2. Even though the experiment setting is quite fair, GAN-based models actually have one additional pretraining stage whereas diffusion model has to be trained from scratch, which could be a reason why diffusion-based model lacks behind GAN-based ones. 

3. The authors do not report how the GAN-based model perform without L1 pretraining stage, both qualitatively and quantitatively. Also, claiming that training GAN-based models for ISR does not face instability issues is quite a bold claim because without pretraining on L1 loss, I can imagine that it could be really unstable.

4. Though the paper highlights that both GANs and diffusion models benefit from scaling, it does not investigate how these models scale in terms of data. Like, how much data that GAN-based model starts to outperform diffusion ones. This could help to strengthen the work.

questions:
1. Is it possible to do some "pretraining" for diffusion-based models similar to the pretraining on L1 for Gan-based method ? So that it can be more fair in evaluation.

2. As I mentioned above in Weaknesses section (bullet point 4), I would like to see how data scaling affect the performance of both GAN-based and diffusion-based ISR models.

3. I know it's a bit infeasible but showing comparison on other restoration tasks like image dehazing, deblurring, .. to see if the same phenonmena happens can be really valuable and strengthen the work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730049789879

thread_type:
official_review

<thread_object>
creator: reviewer_slas

<content>
summary:
This study challenges the assumption that diffusion-based models inherently outperform GANs in Image Super Resolution (ISR). Noting that diffusion-based ISR models often use larger networks and longer training than GANs, the authors investigate whether these performance gains come from model design or simply increased resources. By controlling for architecture, model size, dataset, and computational budget, they find that GAN-based models can achieve comparable or even superior results. They also examine the influence of factors like text conditioning and augmentation, analyzing their effects on ISR performance and related tasks.

strengths:
+ Conducting a comparison of GAN and Diffusion-based approaches for Super-Resolution with the same computational resource can provide good insight for the community.
+ The finding that given the same model size, GAN matches the performance of quality with the diffusion-based method is interesting.

weaknesses:
**Concerns**
I believe that the paper needs to have a thorough clarification. Specifically:

+ Claiming that GAN-based and Diffusion-based approaches give the comparison if using the same number of parameters might be relatively strong. It needs very careful investigations and evaluation. Because, if they give comparable performance, the community has no reason to use diffusion with much more cost for both training (much longer) and inference (much more sampling steps). A comparison with the same setup on some widely-used common dataset benchmarks at first might provide some insights and support rather than just collecting some custom massive datasets.

+ Conducting experiments on extremely huge data, i.e. 17 million images is a very high cost. The author could provide a comparison from a small to a larger number of data in their collected data to see the differences between GAN-based and Diffusion-based methods. For example, 100k, 1M, 2M, 5M, 17M, etc pickup of some of these settings might be reasonable to see if the results/findings are consistent. In practice and research, the number of images for the study is often not too huge up to 17M.

+ Were the methods in Table 1, and Table 4 "SUPIR, RealESRGAN, DiffBIR, ResShift" trained on the same dataset as the Diff (ours) and GAN (ours)? Also in these tables, it should be better to clearly state which one is GAN-based and diffusion-based would greatly improve readability.

+ Table 1 and Table 8 show that ResShift with about just 1/4 parameters (174M) already outperformed the GAN (ours) and Diffusion (ours) 614 and 630M on PSNR and SSIM. This may raise a question of whether scaling more can bring up the performance or not, which is contradicting as concerned in the paper doubts the performance gain that comes from scaling up model size. 

+ Figure 1 and Figure 4 are almost the same and seem to be redundant with no more information added.

**Other suggestions** 
+ For the whole paper, the current form seems to use all \cite{} making it very messy for all references. I think the use of \citep{} in latex would produce a more correct presentation of citations for many parts of the paper. Using \cite{} for cases where the citing author is subject (S), but \citep{} for other cases when referring to the paper.
+ Text in figures presented in the paper is too small, e.g. Figure 2, figure 3, figure 5.

questions:
Q1. The detail of "we did not encounter any difficulties with optimization" --> What can be the reasons for that nice success? Since many works and practices confirm that training GAN is very unstable and mode collapse is a well-known problem of GAN.

Q2. An experiment for the diffusion-based method in this study took 1 month to get the checkpoint, didn't it? Here many experiments were conducted for diffusion, how much time (months) it is estimated to take to complete all of the reported training? This is to provide some information for reproducibility for the community.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704170126

thread_type:
official_review

<thread_object>
creator: reviewer_euhf

<content>
summary:
This paper systematically compares GANs and diffusion models for image super-resolution (ISR) under controlled, comparable conditions. The findings reveal that GANs, when trained with similar protocols, can match the quality of diffusion models while offering practical advantages, such as faster training and single-step inference. This study highlights key trade-offs between GAN and diffusion approaches, contributing valuable insights into their performance and efficiency in ISR tasks.

strengths:
Overall, this type of work should be appreciated as it probes deeper into what the differences in paradigms are when it comes to the SR task. The authors ensure that the setup for both paradigms is as comparable as possible through the architecture, datasets, etc. In general, the writing is good but some parts were confusing.

weaknesses:
Because their experimental results were sometimes in favor of diffusion, sometimes in favor of GAN, and sometimes no difference was found, I have two suggestions that would significantly improve this paper.  First, the authors should taxonomize and explain to the reader when one paradigm should be preferred and in which scenarios. Otherwise, this type of work does not help us improve actual application performance. Second, for the authors to actually be able to make such claims, they need to use more fine-grained evaluation, e.g. the model outputs (SR) images should be used for a specific task like digit recognition, segmentation, etc. However, just comparing single valued PSNR, SSIM, LPIPS etc does not really tell us where these models are outperforming each other. That is also very evident from the qualitative results where within the same figure, both paradigms *visually outperform* each other. 

----not necessarily weaknesses----
Terminology: In Line 49, the term “fairness” might be misleading in this context. Instead, a term like “controlled conditions” or “standardized experimental setup” could better communicate the need for consistent variables, such as dataset size and model complexity, in comparing results.
-  In the related works section, the authors mention conflicting findings about whether text conditioning improves diffusion-based ISR. However, it’s unclear why these differences exist or what insights the current paper offers on this topic. A more thorough discussion or stance on this issue could add depth and relevance.
- Moving the “variations of SR” subsection earlier in the paper would help readers understand the exact ISR task being investigated, providing important context before diving into the model comparisons.
- In Line 130, “given a reference on training” is unclear.  

- A significant limitation is the use of proprietary models and datasets, making it difficult for others to replicate the experiments. For instance, the use of “internal foundation model” and a “proprietary dataset of 17 million…” lacks important detail.  Will this dataset be released? 

Figure Captions and Clarity:
Figures 1 and 4: These figures would benefit from more descriptive captions, highlighting key differences and the main takeaways.
Figure 4: The meaning of the green and grey indicators should be further clarified, as well as the criteria used to define convergence in the caption. I’m aware it is in the text.
Figures 2 and 5: Why do these not include the original HR?  
Figure 3: This figure is hard to interpret because it’s unclear what exact quantity or metric is being reported. Should be added to the caption. The corresponding section is also difficult to understand.

questions:
“We note that we use semantic image-level captions. Global-level information appears to be not very useful for the SR task” What is the difference between image-level and global-level. Can you give an example? This is unclear.

I have never seen “p-value on SbS comparison” as a way to evaluate using human judgment. Why not just do this quantitatively and threshold the difference between checkpoints? The current way of stopping and evaluating seems incredibly arbitrary and subjective. Can the authors share a few works that use this paradigm?  as I have never encountered it. 
E.g. also in “ We conduct an SbS comparison between text-conditioned and unconditional models for both paradigms at all stages of training”
How exactly are the GAN models conditioned on the text?

One major issue is that because of the nature of the SR problem (ill-posed, many to many mapping etc) the types of errors SR models make can inherently invalidate the correctness of the image, e.g. numbers get blurred. It would make the paper significantly stronger if the authors can dig deeper into the entire performance of the SR models for specific tasks as they pertain to downstream tasks and make the evaluation more human interpretable. Right now it is being collapsed into a single number and some qualitative examples. It’s really challenging to make use of the findings in the paper for downstream research/applications. This could significantly enhance the contributions of this work (e.g. by answering *when should one use one paradigm over the other*)
But really what are the types of error differences between the two? 
E.g. figure 9 in appendix - the digits or letters. 
E.g. figure 12 in appendix - sometimes diffBIR and real esrgan , the results are switched, one does better than the other
We are not really getting conclusive results. Instead mixed findings. It would be great if the authors can taxonimize and dig deeper into when we should use GAN over Diffusion based SR models.
Line 1505 - figure caption “high frequency”
Not obvious to me why in section G, “G SUPER RESOLUTION OF SYNTHETIC IMAGES” the authors used those datasets to test for OOD? Why not use real data not synthetic data? Second, what if the synthetic data generated by diffusion models (e.g. SDXL as mentioned in the paper) may actually produce a distribution that is closer to that of the diffusion based SR model, thereby giving the diffusion based model a sort of advantage? 
Table 1 What is the dataset being used? Authors should mention this in the caption of the table.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work focuses on the comparison of GAN and diffusion model in the task of super-resolution. Specifically, GANs and diffusion models with similar parameters are trained, and their performance is observed. The findings of the paper are that GANs achieve comparable results to diffusion models, but with the merits of faster convergence and single-step inference. Reviewers generally agree that the findings in this work are interesting, and the AC shares the same view.

This paper receives mixed ratings of (5, 6, 6, 6). After reading the paper, review, and rebuttal, the AC has concerns about the comprehensiveness of the analysis. For example:

1. This paper mainly focuses on one design choice for each model, and a rather limited training paradigms. For example, the conclusion could be different for different architecture (e.g., UNet vs DiT) and pre-training scheme (e.g., text-to-image pre-training). While the current findings is somewhat surprising and interesting, more thorough studies should be conducted.

2. The studies of parameter count can be more comprehensive. For example, while it is shown that GANs achieve comparable performance to diffusion models with around 600M - 700M parameters, it is unclear the same conclusion can be reached for say fewer or more parameters. This study is important to know the limits and potential of both models.

As an analytical paper, it is recommended to include more studies and comparison between the two paradigms in the future version. Based on the concerns, the AC would recommend a rejection.