Does Diffusion Beat GAN in Image Super Resolution?

## Abstract

Abstract There is a prevalent opinion 1 1 1 See, for example, [ 33 ] . that diffusion-based models outperform GAN-based counterparts in the Image Super Resolution (ISR) problem. However, in most studies, diffusion-based ISR models employ larger networks and are trained longer than the GAN baselines.
This raises the question of whether the high performance stems from the superiority of the diffusion paradigm or if it is a consequence of the increased scale and the greater computational resources of the contemporary studies. In our work, we thoroughly compare diffusion-based and GAN-based Super Resolution models under controlled settings, with both approaches having matched architecture, model and dataset sizes, and computational budget. We show that a GAN-based model can achieve results comparable or superior to a diffusion-based model.
Additionally, we explore the impact of popular design choices, such as text conditioning and augmentation on the performance of ISR models, showcasing their effect in several downstream tasks. We will release the inference code 2 2 2 https://github.com/yandex-research/gan_vs_diff_sr and weights of our scaled GAN.

## 1 Introduction

In recent years, the field of Image Super Resolution (ISR) has witnessed significant advancements, primarily attributed to improvements in image generation frameworks, mostly Generative Adversarial Networks (GANs) [] and Denoising Diffusion Models (DDMs) [].

GANs introduced the adversarial training framework, which led to their remarkable ability to fit and recover narrow, unimodal distributions while struggling with complex multi-modal data.
This trade-off allowed GANs to dominate the area of ISR for half a decade [, , , ], pushing the boundaries of resulting image quality, especially for high upsampling factors.

In contrast, more contemporary DDMs provide a robust framework for image generation, reliably covering multi-modal data distributions.
Yet, the diffusion ability to yield high-quality and diverse outputs comes with a cost of high computational load during training and inference.

At the moment, the literature suggests that diffusion models have become a dominant source of state-of-the-art methods in generative modeling, including ISR ([, , , ]).
However, a comprehensive comparison between GAN and diffusion models in the context of ISR under controlled conditions still needs to be explored.
In particular, GAN-based methods that modern state-of-the-art methods are compared with are typically represented by Real-ESRGAN [], which has orders of magnitude fewer parameters, training data samples, and computational budget.

This work aims to thoroughly compare GAN and DDM frameworks for Image Super Resolution.
Given the unique strengths of both GANs and diffusion models, understanding their comparative performance in ISR tasks is crucial.
This comparison requires a rigorous and controlled experimental setup to ensure fairness, i.e., standardized variables such as the size of the training dataset, model complexity, and controlled allocation of computational resources.

Furthermore, we explore the influence of various design choices on the performance and generalization of these models.
In particular, we investigate the value of practically useful Real-ESRGAN [] augmentations and validate the necessity of incorporating textual conditioning (e.g., image captions).
The contributions of our work are the following:

- –
We thoroughly compare GAN and DDM in Image Super Resolution problem: if scaled appropriately, GAN-based models surpass quality of diffusion-based ones in classical SR and yield similar quality in SR in the Wild.
- –
We investigate the influence of textual conditioning in the form of non-ISR-specific image captions and find that it has no significant effect on both frameworks.
- –
We explore the pros and cons of a complex augmentations pipeline for each paradigm and reveal that it may introduce some quality benefits but comes with its cost, especially hurting DDM training dynamics.

## 2 Related Work

#### GANs for SR

Shortly after the introduction of the original GAN paper [] for the general image generation problem, C. Ledig et al. presented SRGAN [], which introduced perceptual loss, combining content loss (using a pre-trained VGG network []) with adversarial loss to generate high-resolution images with improved perceptual quality.
The following research updated the initial scheme with improved architecture and feature extraction (ESRGAN [], ESRGAN+ [], LAPGAN [], LSMGAN []), more effective training mechanism (RankSRGAN [], PGGAN []) more robust discriminator (RaGAN []) or multiple discriminator models (MPDGAN []), and sophisticated data pre-processing strategies to increase robustness to in the wild data (BSRGAN [], RealSR [], Real-ESRGAN []).
These methods built a strong foundation for GANs, which have become the de facto standard framework for all kinds of ISR problems.

Recent advances in DDMs resulted in ISR models with stunning restoration quality.
This remarkable progress is at least partially attributed to scaling up models, data, and computational resources.
While novel GAN-based methods make progress in catching up with updating and scaling (GigaGAN []), small-scale Real-ESRGAN remains the most commonly used baseline model in modern ISR research.

#### Diffusion models for SR

Recent developments in training and scaling of DDMs resulted in diffusion surpassing GANs for the general image generation problem [] and several new methods aimed to transfer this success to the ISR problem.
The pioneering ISR DDM approaches trained image-conditioned DDM from scratch, concatenating the input noise with either the upscaled version of a low-resolution image (SR3 []) or its hidden representation produced by a convolutional network (SRDiff []).
Their impressive results relegated ISR GANs to the background and inspired the next-generation methods that propose to directly sample from the distribution of LR-images rather than the conditioned Gaussian noise (I^2SB [], ResShift []).
Another branch of research aims to incorporate generative prior of Latent Diffusion Models [] by freezing their generative backbones and training only lightweight ControlNet-based [] modules (StableSR [], DiffBIR [], SUPIR []).
The latter class of methods typically uses additional condition sources, such as depth maps, segmentation maps, and text.

#### Text-conditioned Super Resolution

Most papers [, ] generally report that using additional conditions in ISR DDMs improves the resulting image quality.
However, the case of text conditioning is more nuanced.
Pixel-based natural images-focused ISR models (SR3 [], SR3+ []) do not use textual conditioning.
At the same time, several ISR models designed for upscaling
inside cascaded diffusion systems, namely, Imagen [], DeepFloyd [], Kandinsky 3.0 [] in the $1024\rightarrow 4096$ upscaler, incorporate
textual information via cross-attention.
In contrast, a more recent YaART [] reports the lack of noticeable quality improvement and removes cross-attention at this stage.
Latent diffusion-based ISR models also differ in this regard.
StableSR [] adopts the null-text prompt, effectively removing the text conditioning, and DiffBIR [] does not have any text conditioning.
However, more recent SUPIR [] uses a dedicated multi-modal large language model to automatically caption input images with non-ISR specific captions, reporting a noticeable performance boost.
SeeSR [] goes further and proposes a method to generate ISR-specific captions to boost the model’s generative ability, claiming its effectiveness in upscaling small objects with ambiguous semantics.

In this paper, we aim to shed light on the importance of textual conditioning for ISR in a practical setup when ISR-specific captions are not accessible, and the model has no intrinsic, generation-based prior to utilizing textual information.

#### Variations of SR task

A general SR problem can be formulated as follows. For a given
high quality image $I^{HR}$, one applies some degradation operation
$\mathcal{D}$ to produce low quality image $I^{LR}$ typically of smaller spatial resolution. This degradation can be a simple bicubic/area downsampling kernel or a more general degradation process.
In the task of Blind Super Resolution (BSR) [], which
has attracted significant attention recently, one aims to reconstruct
low-resolution images exposed to *unknown* and *complex* degradations. Higher-order degradation models introduced
in [, ] emulate possible types of corruptions that occur in the real world.

## 3 Methodology

#### Models

Since our goal is to compare GAN and diffusion Super Resolution under a comparable setup, we use the same architecture for both paradigms considered. Specifically, we adopt a network similar to $256\times 256\rightarrow 1024\times 1024$ SR model from Imagen ([]). It acts in the pixel domain and is based on Efficient U-Net architecture []. This model has $600-700$M trainable parameters, depending on the configuration. Details about architecture are provided in Appendix C.

GAN-based Super Resolution models take bicubic-upscaled low-resolution image and directly predict a high-resolution image, given a reference on training.

Diffusion Super Resolution models predict a noise applied to the high-resolution image, using the bicubic-upscaled low-resolution image as a condition.
Following the prior work, the condition image is channel-wise concatenated with the noisy input.

Both models are almost identical in terms of parameters, but for the slight difference in the diffusion model having slightly more parameters due to the concatenation of noisy input and image condition in the input layer and timestep embeddings.

#### Text conditioning

One of the questions addressed in our work is the impact of text conditioning on the performance of Super Resolution models. In our work, we consider two types of text encoders: a CLIP-like [] proprietary encoder transformer model that we call XL with 1.3B parameters and UMT5 [] encoder-decoder with 3B parameters. The text caption, corresponding to the image, is first processed via the text encoder and then passed to the Super Resolution model via image-text cross-attention at the lowest resolution
of the U-Net encoder, middle block, and decoder. This design choice follows Imagen [] work.

In our work, we study the impact of the prompts with semantic information i. e., the ones that are typically available from large image-text datasets [, ], rather than image-quality aware captions adopted in SUPIR [] and SeeSR []. We believe it is timely to revisit the impact of these non-SR-specific texts due to their prominent availability in recently published image-text datasets [, ] used for training large-scale multi-modal models.

#### Augmentation

In our work, we study the impact of train-time augmentations in two setups: Super Resolution of down-scaled images and in case of a more general and complex degradation process. We train GAN and diffusion models with and without the image degradation model proposed in [] to address both scenarios.

#### Training

Following the standard practice [, , , ], both types of models are trained on image crops.
In the ablation study in Section 5, we also investigate the impact of finetuning on full-resolution images.

For GAN-based SR, we initially pretrain the generator model with $L_{1}$ loss only. This pretraining is essential since training from scratch with adversarial loss yields artifacts in agreement with prior work [, , ]. Training with $L_{1},L_{2}$ or any derivative training objective alone leads to over-smoothed and blurry upscaled images. Adversarial training, introduced in SRGAN [], allows the production of images with sharp edges and distinguished high-frequency details. We adopt non-saturating adversarial loss for training without any additional regularization.

$$ $L_{D,\mathrm{adv}}=\mathbb{E}_{I^{HR}\sim p(I^{HR})}[\log D_{\theta_{D}}(I^{HR })]+\mathbb{E}_{I^{LR}\sim p(I^{LR})}[\log(1-D_{\theta_{D}}(G_{\theta_{G}}(I^{ LR})))]$ (1) $$

$$ $L_{G,\mathrm{adv}}=-\mathbb{E}_{I^{LR}\sim p(I^{LR})}[\log D_{\theta_{D}}(G_{ \theta_{G}}(I^{LR}))]$ (2) $$

Above, $\theta_{G}$ and $\theta_{D}$ are the parameters of generator and discriminator, respectively. $I^{LR}$, $I^{HR}$ are the low-resolution and high-resolution images. We employ a U-Net with spectral normalization introduced in [] as a discriminator with twice as many hidden channels. Discriminator is trained from scratch.

During the $L_{1}$ pretraining stage, the generator model is only trained with $L_{1}$ loss.
After that, we turn the adversarial loss on and reformulate the generator objective as a sum of $L_{1}$ loss and adversarial loss $L_{G,\mathrm{adv}}$ with equal weights:

$$ $L_{G}=L_{1}+L_{G,\mathrm{adv}}$ (3) $$

We note that usually GAN-based SR papers include an additional perceptual loss term, following SRGAN [] work. We experimented with the option of adding loss on VGG19 [] features computed between the predicted and original high resolution image, but observed no improvement.

The diffusion model is trained on $\epsilon$-prediction objective with noise timesteps $t$ sampled uniformly from $[0,1]$:

$$ $L_{\mathrm{diff}}=\mathbb{E}_{I^{HR}\sim p(I^{HR}),I^{LR}\sim p(I^{LR}), \epsilon\sim\mathcal{N}(0,1),t\sim\mathcal{U}[0,1]}\|\epsilon-\epsilon_{\theta }(z_{t},I^{LR},t)\|_{2}^{2},z_{t}=\alpha_{t}I^{HR}+\sigma_{t}\epsilon$ (4) $$

Where $\alpha_{t}$ and $\sigma_{t}$ are determined by the noise schedule.
We adopt standard variance-preserving (VP) schedule [] with linearly increasing variances.

#### Inference

GAN-based upscalers produce high-resolution images in a single forward pass, whereas diffusion models iteratively denoise input noise conditioned on a low-resolution image.
For that, we adopt DPM-Solver++ [] as the one providing an excellent trade-off between generation quality and efficiency. We set the order of the solver to be two and chose the multistep method for sampling. All upscaled images are generated with 13 sampling steps. Our evaluations showed that a further increase in the number of sampling steps does not lead to visually noticeable image quality gains H.1.

## 4 Experiments

### 4.1 Experimental setup

#### Training Dataset

We train our models on a large-scale proprietary dataset of 17 million text-image pairs. The dataset consists of $1024\times 1024$ px images with exceptional image quality, high image-text relevance, and English captions. The high dataset quality is achieved by a rigorous filtering process described in detail in Appendix  F.

We train our models on image crops to ensure training efficiency and sample variety. For that, we produce low-resolution - high-resolution (LR, HR) image pairs via extracting $256\times 256$ px random crops of the original images, which are then downscaled to $64\times 64$ px. For the finetuning stage, we interpolate 1024 px images to 256 px. cv2.INTER_AREA algorithm is adopted for image resizing.

#### Training Hyperparameters

We follow prior work in the choice of hyperparameters [, ] and observe that the training with these choices is stable and converges fast enough.
We fix the batch size for GAN and diffusion to ensure fairness of comparison on the number of samples seen during training and adjust the learning rate accordingly.
Further increasing the learning rate with the goal of speeding up training leads to training instability and poor final results.
For both models, the batch size is set to 512 for training on crops and 128 for full-resolution images.
We adopt a constant linear rate with a linear warmup phase.
The models are trained with Adam  [] optimizer without weight decay.
More details are provided in Appendix D.

While most of the related works adopt a predefined number of training iterations, we set the training duration dynamically and terminate the training only when the visual quality of the images upscaled by adjacent model checkpoints becomes indistinguishable from the humans’ perspective. Since the changes become smaller throughout the training, we use an exponentially spaced grid with checkpoints from the following list of thousands of training iterations: [10, 20, 40, 60, 100, 140, 220, $\ldots$], where we increase the interval between adjacent two checkpoints by a factor of two. Then, we estimate the human annotator preference using the Side-by-Side (SbS) evaluation for the current and previous training steps. If there is no improvement for three subsequent measurements, we conclude that the model performance is saturated and proceed to the next stage. We note that the training loss and full-reference metrics may continue to improve, but the improvement is imperceptible to human observers. In our work, user preferences are used both for early stopping and model evaluation. The SbS comparison setup is described and explained in greater detail in Appendix E.

#### Evaluation Datasets

We evaluate the performance of the SR models under study on the 244-images dataset constructed from the pictures from RealSR [], DRealSR [], and some additional high-quality samples from the web. LR 256 px images are obtained from HR images with cv2.INTER_AREA. Captions for images required as input for text-conditional models are generated with the LLaVA model [].

To estimate the quality of image restoration under more complex and severe corruptions, we use the same dataset but with LR inputs obtained through Real-ESRGAN [] degradation pipeline.

In addition, we investigate the robustness of ISR models on out-of-distribution synthetic data in Appendix G. For that, we provide a qualitative comparison between GAN and diffusion models together with the baselines on the Super Resolution of downscaled images produced by the YaART [] and SDXL [] models using prompts from DrawBench [] and YaBasket [] prompt sets.

#### Baselines

Our work’s primary goal is to compare GAN and diffusion Super Resolution rather than establishing a new state-of-the-art in the domain. However, to demonstrate that our models yield quality competitive to the current state-of-the-art, we provide both qualitative and quantitative comparison with SUPIR [], DiffBIR [], RealESRGAN [], ResShift [].

### 4.2 Training Dynamics

#### GANs converge faster than diffusion

As discussed above, we train the model until the human annotators do not show a preference for the current training step over the previous one for three evaluations in a row. To have enough reference points, we train the models such that they cover at least six milestones—[10k, 20k, 40k, 60k, 100k, 140k] iterations.

We observe that GAN Super Resolution converges very quickly in all stages. After 40k steps of $L_{1}$ pretraining, the changes in the output of the SR model become imperceptible. Adversarial training converges relatively fast as well. After pretraining on crops for 40k steps, the performance of the Super Resolution model almost stabilizes. Prior works [, , ] conduct training for 500k-1500k iterations with batch size $\sim 10$ times smaller than the one used in our work. Therefore, the total number of samples seen is of the same order of magnitude as the referenced papers.

We note that in our experiments, in addition to worsening computational efficiency, pretraining on full-resolution images resulted in worse Super Resolution quality starting from the first thousands of training iterations. After several attempts to adjust training hyperparameters, we stopped this line of experimentation.

GANs are claimed to be prone to mode collapse [] and notoriously hard to optimize, requiring careful tuning and regularization for successful training. However, we did not encounter any difficulties with optimization. After several hundreds of iterations, our GAN models achieved equilibrium between the generator and the discriminator and continued to improve steadily until convergence.

Conversely, diffusion models exhibit slower convergence, requiring up to 620k iterations of $L_{2}$ pretraining on cropped images for the model to fully converge.

Evaluation results are presented in Figure 1. We treat models as equal if $p$-value
on SbS comparison $>0.05$ following common practice. More details are in Appendix  E.

Figure: Figure 1: SbS comparison between two subsequent checkpoints showcases faster convergence of GAN models. Green corresponds to statistical improvement on the current step, grey to equality. Three evaluations without improvement in a row indicate convergence.
Refer to caption: x1.png

### 4.3 Performance comparison

#### Evaluation

To assess the performance of SR models, we report classic full-reference metrics PSNR, SSIM [], LPIPS [] and recent no-reference CLIP-IQA [] metric in Table 1.
Figure 2 presents several representative visual samples. Additionally, we conduct side-by-side (SbS) comparisons between different models and present user preference results in Table 2 and Table 5. We calculate the number of wins of one model over another plus the number of draws divided by two, which we denote as the *win rate*. The win rates are color-coded: Green if the first model is statistically better than the second, red if the first model is statistically worse, and black otherwise. Details about the statistical significance criteria are provided in Appendix E.

#### Results

GAN-based SR model is equal or better than diffusion with respect
to the majority of evaluation metrics. Yet both approaches produce visually aesthetic samples with sharp edges, small details, and apparent textures.
Our SR models are competitive to the baselines from the literature.
According to the user preferences (Table 2)
GAN-based model outperforms any of the other models considered.
We note that full-reference metrics do not always agree with each other
and SbS comparison results.

**Table 1: Quantitative comparison between GAN, diffusion SR and current state-of-the-art on $\times 4$ image Super Resolution. Red - best, blue - second best result.**
| Metrics | Diff (ours) | GAN (ours) | SUPIR | RealESRGAN | DiffBIR | ResShift |
| --- | --- | --- | --- | --- | --- | --- |
| PSNR $\uparrow$ | 26.655 | 26.006 | 24.270 | 24.435 | 24.809 | 27.830 |
| SSIM $\uparrow$ | 0.748 | 0.770 | 0.679 | 0.721 | 0.682 | 0.786 |
| LPIPS $\downarrow$ | 0.253 | 0.208 | 0.310 | 0.316 | 0.337 | 0.238 |
| CLIP-IQA $\uparrow$ | 0.719 | 0.826 | 0.757 | 0.660 | 0.850 | 0.692 |

**Table 2: SbS comparison between GAN-based and diffusion-based SR with each other and current state-of-the-art. The values in the table are win rates of Model 1 over Model 2. Green corresponds to statistical advantage, red to statistical disadvantage, black to statistical indifference between two models.**
| Model 1<br>Model 2 | Diff (ours) | GAN (ours) | SUPIR | RealESRGAN | DiffBIR | ResShift |
| --- | --- | --- | --- | --- | --- | --- |
| Diff (ours) | x | 9.0 | 37.7 | 85.2 | 51.0 | 46.3 |
| GAN (ours) | 91.0 | x | 62.7 | 94.6 | 69.0 | 86.0 |

Figure: Figure 2: Visual comparison between GAN and diffusion SR model from our work and the baselines on SR($\times 4$). Zoom in for the best view.
Refer to caption: x2.png

### 4.4 Impact of Text Conditioning

We train text-conditioned emodels with the same training setup used for unconditional models. Our experiments cover two text encoders of different nature: our internal CLIP-like model called XL and UMT5 Text-conditioned models have slightly more parameters than unconditional models due to additional transformer blocks. We conduct an SbS comparison between text-conditioned and unconditional models for both paradigms at all stages of training. According to
the results in Figure 3 human annotators show preference neither for text-conditional models nor for unconditional.

Figure: Figure 3: SbS comparison between text-conditional (based on a proprietary model called XL and UMT5) and unconditional SR models. Bar plots show that additional text-conditioning does not noticeably improve perceived image quality
Refer to caption: x3.png

We note that we use semantic image-level captions. Global-level information appears to be not very useful for the SR task. Quality-aware prompts could be beneficial, especially for stronger corruptions ([, ]). We leave verification of the usefulness of various SR-specific image captioning techniques outside the scope of this work.

### 4.5 ISR for a Complex Degradation Model and the Impact of Training Augmentations

Above, all methods were applied to the SR$(\times 4)$ task, which is essentially an inversion of a fixed downscaling. However, many contemporary works study the task of Blind Super Resolution (BSR), which assumes a more complex degradation model involving multiple, typically unknown corruptions. Real-ESRGAN []
is a popular degradation model emulating various degradations that occur in low-quality images. Therefore, following the standard practice, we produce LR inputs via the Real-ESRGAN pipeline.

In our work, we study the impact of augmentations for performance on
a simple SR$(\times 4)$ task and the problem of more general degradation removal. We train both GAN and diffusion models with
Real-ESRGAN augmentations in the same way as for the vanilla SR task.

We observe that $L_{1}$ pretraining and training with the adversarial loss for GAN models converge with the same number of training steps
as for training without augmentations (see Figure  4). However, with Real-ESRGAN augmentations, the diffusion SR model requires even more extended training, and its performance saturates only after 940k iterations. The augmentations
appear to noticeably slow down the convergence of the diffusion model.

Figure: Figure 4: SbS comparison between two subsequent checkpoints. Green corresponds to statistical improvement, grey to equality. Three evaluations without improvement in a row indicate convergence.
Refer to caption: x5.png

#### Results

Pretraining with augmentations is detrimental to the
performance on vanilla SR task. The inversion of a non-deterministic, complex degradation process is an inherently more challenging task since the network has to account for multiple degradations of varying magnitude. Therefore, it is natural to expect that models specialized in inverting downscaling without additional degradations better represent sharp edges and fine details. This intuition agrees with the experimental results in Table 3 for SR($\times 4$).

At the same time, pretraining with augmentations appears to be necessary for the SR task with additional corruptions. According to the evaluation results in Table 3, pretraining with augmentations improves robustness to corruptions for both GAN and diffusion SR models. However, metrics are still far below the ones for the simple upscaling.

**Table 3: Impact of augmentations. Red - best, blue - second best result.**
| Dataset | Metrics | Diff (no aug) | Diff (w aug) | GAN (no aug) | GAN (w aug) |
| --- | --- | --- | --- | --- | --- |
| SR($\times$4) | PSNR | 26.655 | 24.633 | 26.006 | 24.228 |
| SSIM | 0.748 | 0.674 | 0.770 | 0.761 |  |
| LPIPS | 0.253 | 0.330 | 0.208 | 0.294 |  |
| CLIP-IQA | 0.719 | 0.773 | 0.706 | 0.751 |  |
| SR($\times$4)<br>+ degradations | PSNR | 22.012 | 22.387 | 22.075 | 22.459 |
| SSIM | 0.560 | 0.518 | 0.570 | 0.607 |  |
| LPIPS | 0.576 | 0.502 | 0.557 | 0.427 |  |
| CLIP-IQA | 0.159 | 0.513 | 0.144 | 0.730 |  |

We show a couple of samples for the dataset with Real-ESRGAN-augmented pipeline in Figure 5. GAN-based and diffusion-based SR models yield samples of similar quality.
GAN-based SR from our study appears to be the best in terms of full-reference metrics. At the same time, human annotators show preference
for SR models with diffusion priors (DiffBIR, SUPIR). We speculate that
presence of rich semantic information is more beneficial in the presence of degradations.

**Table 4: Quantitative comparison between GAN, diffusion SR and current state-of-the-art on SR($\times$4) with Real-ESRGAN degradations. Red - best, blue - second best result.**
| Metrics | Diff (ours) | GAN (ours) | SUPIR | RealESRGAN | DiffBIR | ResShift |
| --- | --- | --- | --- | --- | --- | --- |
| PSNR $\uparrow$ | 22.387 | 22.459 | 19.714 | 21.071 | 21.996 | 21.309 |
| SSIM $\uparrow$ | 0.518 | 0.607 | 0.491 | 0.543 | 0.544 | 0.472 |
| LPIPS $\downarrow$ | 0.502 | 0.427 | 0.479 | 0.472 | 0.457 | 0.535 |
| CLIP-IQA $\uparrow$ | 0.513 | 0.730 | 0.747 | 0.656 | 0.781 | 0.585 |

**Table 5: SbS comparison between GAN-based and diffusion-based SR with each other and current state-of-the-art. The values in the table are win rates of Model 1 over Model 2. Green corresponds to statistical advantage, red to statistical disadvantage, black to statistical indifference between two models.**
| Model 1<br>Model 2 | Diff (ours) | GAN (ours) | SUPIR | RealESRGAN | DiffBIR | ResShift |
| --- | --- | --- | --- | --- | --- | --- |
| Diff (ours) | x | 34.0 | 18.4 | 71.7 | 29.3 | 72.1 |
| GAN (ours) | 66.0 | x | 30.0 | 66.2 | 30.6 | 80.2 |

Figure: Figure 5: Visual comparison between GAN and diffusion SR model from our work and the baselines on SR($\times 4$) + degradations. Zoom in for the best view.
Refer to caption: x6.png

### 4.6 Artifacts

Both SR paradigms suffer from specific artifacts. GAN-based SR models are prone to oversharpening and generating unnatural textures. On the other hand, diffusion-based SR is usually inferior at generating high-frequency details. In addition, diffusion-based SR models tend to generate pale, dim images when compared to the original. We provide visualizations of these common artifacts in Figure 13 and Figure 14.

## 5 Ablations

#### Training with Adversarial Loss

In agreement with the literature, we observe that training with adversarial loss is essential for single-step SR to generate sharp details and realistic textures. Human annotators prefer the GAN model when compared to $L_{1}$ pretrain. According to the results of the SbS comparison
a model trained with adversarial loss is preferred in
$\sim$95% cases.

#### Finetuning on Full-Resolution Inputs

Conventionally, papers on ISR adopt image crops for training since they allow the utilization of larger batches and faster training. However, the inference network accepts full-resolution input rather than crops.
Awareness of the overall content could provide additional hints to the network.
Therefore, it is questionable whether finetuning on full-resolution inputs may improve quality.

In our experiments, we do not observe any improvement from finetuning on full-resolution images for both SR paradigms. We have tried several choices of learning rate, and for a small enough learning rate, quality remains unchanged according to human preferences according to Table 6, whereas larger learning rates drive both models from good optima and result in degradation of models’ performance.

**Table 6: Human preferences for GAN SR models without finetuning on full resolution images and with finetuning.**
| w/o full resolution finetuning | Equal | w full resolution finetuning |
| --- | --- | --- |
| 4.2% | 90.4% | 5.4% |

#### Impact of Perceptual Loss

SRGAN [] introduced perceptual loss for GAN training.
It is defined as $L_{2}$ distance between the features of upscaled and the high-resolution image processed via some feature extractor, for instance, VGG19 []. Typically, the addition of perceptual loss to the pixel and adversarial losses is reported to improve the end quality of ISR models. To revisit the impact of perceptual loss, we follow [] and use the {conv1, ...conv5} feature maps (with weights {0.1, 0.1, 1, 1, 1}) before activations.

Intriguingly, we observe no significant improvement from using the perceptual loss term. According to the SbS comparison between models trained with and without perceptual loss in Table 7 human annotators do not show preference to model trained with perceptual loss. Based on that, we conclude that perceptual loss has little, if any, impact on the end quality of trained models.

**Table 7: Human preferences for GAN SR models trained with and w/o LPIPS loss.**
| w LPIPS | Equal | no LPIPS |
| --- | --- | --- |
| 5.0% | 94.2% | 0.8% |

## 6 Efficiency

Another important aspect from a practical perspective is the efficiency of the Super Resolution pipeline. While one approach may offer better Super Resolution quality compared to another, this advantage could be outweighed by a much larger inference cost and memory overhead. As one can observe from Table 8, models differ by orders of magnitude in terms of the total number of parameters involved in all stages of the inference workflow. Some approaches can be easily deployed on mobile devices, while others necessitate a compute accelerator with a large enough amount of memory.

We carried out inference measurements on an NVIDIA A100 GPU for upscaling a single image from $256\rightarrow 1024$ (batch size is 1) and report the latency in Table 8. GAN-based methods produce a high-resolution image given a low-resolution input in a fraction of a second, whereas diffusion-based SR with generative priors requires dozens of seconds.

**Table 8: Runtime and number of parameters (during inference) of compared models. Some of the models are consisted of several stages with parameters distributed among them. The benchmarking and parameter-counting protocol are explained in greater detail in Appendix H.5.**
|  | Diff (ours) | GAN (ours) | SUPIR | RealESRGAN | DiffBIR | ResShift |
| --- | --- | --- | --- | --- | --- | --- |
| Runtime, (s) | $3.20\pm 0.21$ | $0.24\pm 0.02$ | $18.11\pm 0.85$ | $0.065\pm 0.003$ | $24.69\pm 0.03$ | $1.19\pm 0.03$ |
| # Parameters, (M) | 631 | 614 | 17846 | 17 | 1683 | 174 |

## 7 Limitations

This work considers the standard SR task and SR in the presence of corruptions from a predefined degradation pipeline. The evaluation
on the real-world low-quality images, such as the ones from RealLR200 [] and RealPhoto60 [] datasets is left for future work.

We also do not consider cases when strong generative capabilities and the ability to sample diverse outputs for a given input are required. We hypothesize that in this setup mode collapse can be an issue for GAN-based models. It is also possible that, in these cases, textual information could be necessary for accurate reconstructions. However, we do not consider these facts or the exact turning point when such model capabilities become required.

Another interesting question not covered in our investigation is the study of scaling behavior for both GAN and diffusion SR paradigms with respect to the model size and amount of training data and compute. In preliminary experiments we observed that the training parameters adopted in our main training setup do not transfer immediately to other model sizes and have to be tuned for specific experimental setup.

## 8 Conclusion

In our work, we performed a systematic comparison between GAN
and diffusion ISR under a fair comparison setup when both approaches are matched in model size and training data available. Our results suggest that GANs can achieve the quality level of modern diffusion models if trained with a similar protocol. At the same time, GAN-based upscalers offer several practical advantages - faster training and single-step inference instead of an iterative denoising procedure.

We hope that our work encourages researchers to conduct more
careful examination when introducing new methods. Since the amount of available resources and data is constantly growing, more recent works have the advantage of having more compute and higher-quality data relative to the prior art. Therefore, it is vital to identify whether the improvement comes from one approach’s superiority over the other or could be attributed merely to the increase of scale.

## Appendix A Background on generative paradigms

In this section, we provide a mathematical background for the generative models used in our study.

### A.1 Generative Adversarial Networks

To produce samples that resemble a target distribution, one can train an auxiliary critic network, known as a discriminator, to distinguish generated data from real data. The discriminator, denoted as $D_{\theta}$, is optimized alongside the generator, denoted as $G_{\theta}$, to solve the following adversarial min-max problem:

$$ $\min_{\theta_{G}}\max_{\theta_{D}}\mathbb{E}_{I^{out}\sim p(I^{out})}[\log D_{ \theta_{D}}(I^{out})]+\mathbb{E}_{I^{in}\sim p(I^{in})}[\log(1-D_{\theta_{D}}( G_{\theta_{G}}(I^{in})))]$ (5) $$

where $I^{in}$ and $I^{out}$ are the input and output spaces respectively. For instance, $I^{in}$ can be the space of random noise or low-resolution images (for the Super Resolution problem) and $I^{out}$ may correspond to the space of natural images. In the case of successful training, the generator learns to produce samples that are difficult to classify by $D_{\theta_{D}}$ and resemble the target distribution.

This approach was introduced in [] and subsequently became a leading generative paradigm for an extended period. GANs have demonstrated the capability to generate high-quality, photorealistic samples [, , ]. However, they are known to suffer from issues such as unstable training and mode collapse [].

### A.2 Diffusion Models

Diffusion models ([]) have emerged as a promising technique in the landscape of generative modeling. These models are primarily inspired by concepts from nonequilibrium thermodynamics, leveraging the idea of diffusion processes to gradually transform simple distributions into complex data distributions and vice versa.

The forward process in diffusion models is a mechanism that systematically transforms structured data into noise through a series of discrete steps. Starting with an original data point $z_{0}$, this process involves a Markov chain where each step $z_{t}$ given $z_{t-1}$ is defined by a Gaussian distribution

$$ $q(z_{t}|z_{t-1})=\mathcal{N}(z_{t};\sqrt{\alpha_{t}}z_{t-1},(1-\alpha_{t})I),$ (6) $$

with $\alpha_{t}$ controlling the noise level. As the process progresses, the data is incrementally corrupted, ensuring the distribution of $z_{t}$ given $z_{0}$ remains Gaussian,

$$ $q(z_{t}|z_{0})=\mathcal{N}(z_{t};\sqrt{\gamma_{t}}z_{0},(1-\gamma_{t})I),$ (7) $$

where $\gamma_{t}=\prod_{i=1}^{t}\alpha_{i}$. This results in $z_{t}$ becoming increasingly dominated by noise, converging to a standard Gaussian distribution as $t$ approaches $T$. The formulation allows for the derivation of the posterior $q(z_{t-1}|z_{t},z_{0})=\mathcal{N}(z_{t-1};\mu_{t},\sigma_{t}^{2}I)$, providing a basis for the reverse process that aims to denoise the data back to its original form.

The backward diffusion process, or reverse process, in diffusion models, is essential for generating structured data from noise by reversing the forward diffusion steps. Starting from a noise sample $z_{T}\sim\mathcal{N}(0,I)$, this process iteratively denoises the sample through a sequence of steps $z_{T},z_{T-1},\ldots,z_{0}$, transforming it back into structured data $z_{0}$. Each reverse step $p_{\theta}(z_{t-1}|z_{t})$ is modeled as a Gaussian distribution:

$$ $p_{\theta}(z_{t-1}|z_{t})=\mathcal{N}(z_{t-1};\mu_{\theta}(z_{t},t),\sigma_{ \theta}(z_{t},t)I),$ (8) $$

where $\mu_{\theta}$ and $\sigma_{\theta}$ are parameterized by a neural network, trained to approximate the reverse of the forward noise addition. The training objective typically maximizes a variational lower bound on the data log-likelihood, encouraging the learned reverse process to closely match the true reverse process. The posterior distribution from the forward process $q(z_{t-1}|z_{t},z_{0})=\mathcal{N}(z_{t-1};\mu_{t},\sigma_{t}^{2}I)$ guides the reverse process’s learning, ensuring that each step effectively reduces the noise incrementally added during the forward process.

This objective can be reduced ([]) to a noise prediction one. Given the original data point $z_{0}$ and its noisy version $z_{t}$ at step $t$, produced by the forward process $q(z_{t}|z_{0})=\mathcal{N}(z_{t};\sqrt{\gamma_{t}}z_{0},(1-\gamma_{t})I)$, the model is tasked with predicting the noise $\epsilon$ that was added. The neural network, parameterized by $\theta$, outputs the predicted noise $\epsilon_{\theta}(z_{t},t)$, aiming to match the actual noise added during the diffusion step. The training objective minimizes the mean squared error between the predicted noise and the true noise:

$$ $L_{\text{diff}}=\mathbb{E}_{z_{0},\epsilon\sim\mathcal{N}(0,I),t\sim\mathcal{U }[0,1]}\left[\|\epsilon-\epsilon_{\theta}(z_{t},t)\|^{2}\right],$ (9) $$

where $z_{t}=\sqrt{\gamma_{t}}z_{0}+\sqrt{1-\gamma_{t}}\epsilon$ and $\epsilon\sim\mathcal{N}(0,I)$.

Finally, the diffusion framework can be extended to conditional modeling via appropriate conditioning mechanisms. By incorporating conditional information $c$, such as class labels, text prompts or images, the model is guided to generate data that is contextually relevant and aligned with the specified conditions.

## Appendix B Qualitative Comparison

Below in Figures 6 and 7, we show additional qualitative comparisons between GAN and SR models from our work. Images are chosen at random from validation datasets.

Figure: Figure 6: Qualitative comparison between GAN and SR models from our work on the original dataset. Zoom in for the best view.
Refer to caption: x7.png

Figure: Figure 7: Qualitative comparison between GAN and SR models from our work on a test set with Real-ESRGAN degradations. Zoom in for the best view.
Refer to caption: x8.png

## Appendix C Architecture Details

We employ Efficient U-Net architecture introduced in Imagen [] for GAN and diffusion SR models.
The number of channels and residual blocks for each resolution is the same as in the $256\times 256\rightarrow 1024\times 1024$ Super Resolution model. The exact model configuration is presented in the listing below.

Figure: Listing 1: UNet model configuration

The core difference between GAN and diffusion SR models is the absence
of timestep embedding and concatenation of the input noise for the GAN SR model.

For text-conditional models, image caption is first processed via text encoder, and the condition processing module outputs both the tokenized prompt (sequence of embeddings) and the pooled embedding. Text condition is forwarded to the SR model via scale-shift modulation in the residual blocks and cross-attention in transformer blocks. Cross-attention acts only at the lowest resolution in encoder, decoder, and middle blocks. There is no self-attention between image patches.

The model with the smallest number of trainable parameters is a text-unconditional GAN generator with 614M trainable parameters, and the largest one is diffusion U-Net, conditioned on UMT5 embeddings with 696M trainable parameters. Therefore, it is fair to say that the models considered are of the same scale.

For training with adversarial loss, we adopt the discriminator from
Real-ESRGAN [] with the only difference that the number of channels is multiplied by $2$ relative to the network used in the original work. The generator model used in our work is significantly larger than the RRDBNet [] from RealESRGAN. Therefore, we have decided to increase the capacity of the discriminator. GAN is trained from scratch and for finetuning on full resolution images we start from the last discriminator checkpoint from training on crops.

## Appendix D Training Details

We use a linear warmup schedule over 1000 training steps to train the diffusion model on image crops, followed by a constant learning rate of 3e-5. For full-resolution inputs, we decrease the learning rate to 1e-6 and increase the duration of the warmup phase to 10k training steps.

For $L_{1}$ pretrain of GAN SR we use the same learning rate schedule as for diffusion but with a higher learning rate - 2e-4. When training with adversarial loss we adopt the same learning rate 1e-4 both for generator and discriminator. Finetuning on high-resolution images is conducted with a learning rate of 1e-5.

In both cases, models are trained with Adam  [] optimizer without weight decay with $\beta_{1},\beta_{2}=(0.90,0.99)$. For more stable training, we follow common practice and adopt an exponential moving average (EMA) with a factor of $0.999$.

Our experiments on crops were performed on two nodes with 8 NVIDIA A100 with 80Gb of VRAM. Distributed communication is performed via Open MPI []. The optimal batch size for model training is 16, and we use two gradient accumulation steps to have a total batch size of 512.

140k iterations for pretraining and
training with the adversarial loss for the GAN-based SR model take approximately three days. Training of diffusion model takes about a month with the same resources.

We adopt Fully Sharded Data Parallel [] for parameter and optimizer state sharding to reduce memory consumption and allow working with larger batch sizes.

## Appendix E Human Evaluation

Our main tool for training early stopping is side-by-side (SbS) comparisons of Super Resolution results between two subsequent checkpoints of the same model.
For that, we use an internal crowdsourcing platform with non-expert assessors.
Before labeling, each candidate assessor must pass an exam and achieve at least an 80% accuracy rate among a pre-defined set of 20 assignments.
After that, we ask assessors to select one of the two high-resolution images shown Side-by-Side based on the following evaluation criteria placed in order of their importance:

- •
The presence of color distortions;
- •
The presence of added artifacts such as exploded pixels and unnatural textures;
- •
Image blurriness/detalization;
- •
Color saturation;
- •
The level of noise.

The assessed high-resolution images are generated from the same low-resolution image.
We also provide the initial low-resolution image as a condition to assess potential color distortions.
Figure 8 showcases the main labeling interface.
Users can also click on any images to open them in full-screen mode and zoom in to better consider the finer details.
After that, it becomes possible to toggle between two high-resolution images using hotkeys to assess minor differences in an almost pixel-wise manner.

Figure: Figure 8: An example of a user interface for Human Evaluation with Side-by-Side comparisons.
Refer to caption: extracted/5954781/resources/sbs_interface.png

From a mathematical point of view, human evaluation is a statistical hypothesis test.
In particular, we are using a two-sided binomial test and its implementation from scipy [] library to test the null hypothesis of whether the two given models are equal in ISR (accessors follow a special set of instructions when making their decisions). More precisely,

$$ $p\mathrm{-value}=\mathrm{binom_test}((n_{\mathrm{wins},1}+n_{\mathrm{eq}}//2, n_{\mathrm{wins},2}+n_{\mathrm{eq}}//2),\;p=0.5),$ (10) $$

where $n_wins_{i}$ - is the number of generations produced by model $i$ and preferred by accessors, $n_eq$ - is the number of generations whose quality is indistinguishable according to our guideline. We reject the null hypothesis if $p_value$ is less than $0.05$, i.e., at the 5% significance level.

## Appendix F Dataset Preparation

We start from a proprietary pool of several billion image-text pairs, initially collected to train image-text models. The data pool originally contained samples with various quality of images, texts, and image-text relevance. The following filtering procedure aims to select high-quality samples suitable for training competitive Super Resolution models.

First, we select samples with images with exactly 1024 px of height or width and not less than 1024 px in the other direction. For non-square images, we perform a center crop of 1024 $\times$ 1024 px. This resolution allows us to achieve a reasonable trade-off between data quality and quantity. After that, we pre-calculate a set of scores produced by multiple learned predictors for each image-text pair. These predictors estimate the quality of images, texts, and image-text relevance.

Image quality predictors are fully connected classifiers, trained on top of image and text features, extracted by our proprietary visual foundation model. In this work, we use estimators of image quality trained on PIPAL [], KonIQ-10k [] and KADID-10k [] datasets, watermark and NSFW content detectors trained on internal data.

We use OpenCLIP [] ViT-G/14 to obtain image-text relevance scores.

We train quality predictor and English language detector for text based on our internal foundation model and data.

After that, we filter the data based on the predictors mentioned above so that the resulting dataset size does not contain obviously bad samples according to the predictor models. We also aim for the resulting dataset size sufficient to train any of our models for up to several dozen epochs. The resulting dataset consists of 17 million 1024 $\times$ 1024 px images and corresponding captions.

## Appendix G Super Resolution of synthetic images

To test the robustness of Super Resolution models on out-of-distribution (OOD) data, we apply both models to upscaling
of 4$\times$ downscaled 1024px images generated by YaART [] and SDXL [] models as two examples of modern text-conditional generative models.
The former is a cascaded pixel diffusion model, while the latter operates in latent space and transforms the generated sample to pixel space using a decoder network.

We generate samples using prompts from DrawBench [] and
YaBasket [] with both models. LR images are obtained by downscaling 1024px generations to 256px via cv2.INTER_AREA.Even though both models produce photorealistic samples of high quality, generated images could still be different from the data observed during training. The discrepancy between training and evaluation data may negatively affect the performance of SR models and cause artifacts.

#### Results

Below, we provide only qualitative evaluation since there is no actual ground truth for this setting, as generative models produce outputs that only approximate real-world images. Hence, the calculation of no-reference and full-reference metrics is not very meaningful. Representative examples are shown in Figures
9,10, 11, 12. Models trained without augmentations appeared to be robust enough for evaluation on synthetic images while providing sharper edges and better representing high-frequency details.

Figure: Figure 9: Qualitative comparison between SR models on images generated with SDXL on DrawBench prompts. Zoom in for the best view.
Refer to caption: x9.png

Figure: Figure 10: Qualitative comparison between SR models on images generated with YaART on DrawBench prompts. Zoom in for the best view.
Refer to caption: x10.png

Figure: Figure 11: Qualitative comparison between SR models on images generated with SDXL on YaBasket prompts. Zoom in for the best view.
Refer to caption: x11.png

Figure: Figure 12: Qualitative comparison between SR models on images generated with YaART on YaBasket prompts. Zoom in for the best view.
Refer to caption: x12.png

## Appendix H Additional Details

### H.1 Sampler Hyperparameters Setting for diffusion Model

In our experiments, we evaluate the diffusion model using DPM-Solver++(2M) [] with 13 sampling steps. This setup demonstrated robust outcomes with a relatively modest number of steps, effectively balancing speed and output quality. To ensure this, we evaluated our diffusion model with different samplers (DDIM [], UNIPC []) and a varying number of sampling steps. The results are presented in Table 9. It is evident from the results that increasing the number of steps beyond those used in our DPM-Solver++ baseline (13 steps) does not enhance performance, whereas reducing the number of steps compromises visual quality.

**Table 9: Comparison of different sampler settings to the one used in the experiments (DPM-Solver++, 13). Side-by-side results and basic metrics are introduced. The setting used in this work is optimal in terms of speed and output quality.**
| Sampler, #steps | Wins | Loses | Ties | p-value | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | CLIP-IQA $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DPM-Solver++, 13 | - | - | - | - | 26.66 | 0.748 | 0.253 | 0.719 |
| DPM-Solver++, 6 | 0.192 | 0.479 | 0.329 | 0.0 | 27.77 | 0.793 | 0.229 | 0.666 |
| DPM-Solver++, 64 | 0.287 | 0.283 | 0.430 | 1.0 | 26.28 | 0.730 | 0.275 | 0.737 |
| UniPC, 6 | 0.138 | 0.354 | 0.508 | 0.001 | 27.58 | 0.786 | 0.232 | 0.657 |
| UniPC, 13 | 0.367 | 0.338 | 0.295 | 0.70 | 26.50 | 0.740 | 0.259 | 0.728 |
| UniPC, 32 | 0.267 | 0.254 | 0.479 | 0.85 | 26.28 | 0.730 | 0.274 | 0.740 |
| UniPC, 64 | 0.425 | 0.425 | 0.150 | 1.0 | 26.26 | 0.730 | 0.277 | 0.742 |
| DDIM, 100 | 0.358 | 0.354 | 0.288 | 1.0 | 26.60 | 0.745 | 0.256 | 0.730 |
| DDIM, 500 | 0.417 | 0.400 | 0.183 | 0.85 | 26.35 | 0.731 | 0.269 | 0.741 |
| DDIM, 1000 | 0.421 | 0.383 | 0.183 | 0.61 | 26.32 | 0.729 | 0.272 | 0.735 |

### H.2 Fine-tuning on Full-Resolution Inputs

Our experiments show that fine-tuning on full-resolution inputs brings no benefit compared to training only on crops. Here, we present metrics computed based on the best checkpoints trained on crops only and full-resolution images (Table 10), as well as a side-by-side comparison (Table 11). This can be explained by the fact that both full-resolution and cropped inputs contain the same low-level information, and the global-level semantic information appears to be irrelevant for the SR task.

**Table 10: Comparison of only-crops traing vs fine-tuning on full-resolution images based on basic metrics.**
| Models | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | CLIP-IQA $\uparrow$ |
| --- | --- | --- | --- | --- |
| GAN (crops) | 26.00 | 0.770 | 0.208 | 0.806 |
| GAN (full-res) | 28.27 | 0.811 | 0.182 | 0.790 |
| Diffusion (crops) | 26.66 | 0.748 | 0.253 | 0.719 |
| Diffusion (full-res) | 26.83 | 0.761 | 0.236 | 0.715 |

**Table 11: Side-by-side comparison of only-crops training vs fine-tuning on full-resolution images.**
|  | Wins | Loses | Ties | p-value |
| --- | --- | --- | --- | --- |
| GAN (crops) vs GAN (full-res) | 0.425 | 0.441 | 0.133 | 0.847 |
| Diff (crops) vs Diff (full-res) | 0.213 | 0.320 | 0.467 | 0.109 |

We experimented with different learning rates for the full-resolution fine-tune: 1e-4, 1e-5 for GAN, and 3e-5, 1e-6 for diffusion. While training models with the rates used for training on crops (1e-4 for GAN and 3e-5 for diffusion) degrades generation quality (color-shifting and blurred generation), fine-tuning both models even with the smallest rates for 300k iterations leads to no statistical improvements. We provide the results of human evaluations against crop-trained baselines for all mentioned setups in Table 12.

**Table 12: Side-by-side comparison of only-crops training vs fine-tuning on full-resolution images with different learning rates.**
|  | Wins | Loses | Ties | p-value |
| --- | --- | --- | --- | --- |
| GAN (crops) vs GAN (full-res, 1e-5) | 0.425 | 0.441 | 0.133 | 0.847 |
| GAN (crops) vs GAN (full-res, 1e-4) | 0.658 | 0.175 | 0.167 | 0.000 |
| Diff (crops) vs Diff (full-res, 1e-6) | 0.213 | 0.320 | 0.467 | 0.109 |
| Diff (crops) vs Diff (full-res, 3e-5) | 0.212 | 0.155 | 0.633 | 0.410 |

### H.3 Early Stopping with Side-by-side Comparison

In our experiments, we adopted a side-by-side comparison to determine the optimal time for early stopping, when further changes in the output of the SR model become imperceptible to human observers (see Appendix E for more detail). However, automated full-reference metrics are more common in the SR literature, so it is interesting to track their evolution during training. We present the measurement of some full-reference metrics in Table 13. One can observe that during pretraining, PSNR and SSIM continue to slightly improve, whereas for GAN training, the metrics stabilize around a specific value after a certain number of iterations. This indicates that above a certain threshold, further improvement in PSNR/SSIM is hardly discernible.

**Table 13: Evolution throughout training of PSNR and SSIM.**
|  | L1-pretrain | GAN | Diff |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Evaluation Step | PNSR $\uparrow$ | SSIM $\uparrow$ | PNSR $\uparrow$ | SSIM $\uparrow$ | PNSR $\uparrow$ | SSIM $\uparrow$ |
| 20k | 28.21 | 0.834 | 25.84 | 0.767 | 20.57 | 0.659 |
| 40k | 28.48 | 0.840 | 26.01 | 0.770 | 20.89 | 0.656 |
| 60k | 28.66 | 0.843 | 26.01 | 0.770 | 21.33 | 0.663 |
| 100k | 28.92 | 0.846 | 26.00 | 0.770 | 22.25 | 0.675 |
| 140k | 29.03 | 0.847 | 26.00 | 0.770 | 22.89 | 0.681 |
| 220k | - | - | - | - | 23.92 | 0.701 |
| 300k | - | - | - | - | 25.66 | 0.733 |
| 460k | - | - | - | - | 26.03 | 0.741 |
| 620k | - | - | - | - | 26.66 | 0.748 |

### H.4 Super Resolution Artifacts

We visualize artifacts typical for GAN-based SR and diffusion-based SR
in Figure 13 and 14.

Figure: Figure 13: GAN’s oversharpening leads to unnatural textures (e.g. fur or grass) and artifacts.
Refer to caption: x13.png

Figure: Figure 14: Diffusion is usually worse at high-frequent details (1 row). Additionally, even after big number of training iterations may generate pale, dim images (rows 2-5).
Refer to caption: x14.png

### H.5 Efficiency considerations

#### Model Size

Different SR approaches may vary drastically in terms of the inference pipeline structure and the sizes of the models involved. The superior quality of one method over another could be outweighed in real-world applications by the amount of memory needed to run inference. When considering the size of SR models with diffusion priors (SUPIR, DiffBIR), we include the size of the UNet, text encoders, and VAE decoder.

- •
Diff (631 M) = Denoiser (631 M)
- •
GAN (631 M) = Generator (614 M) + Discriminator (17 M)
- •
SUPIR (17846 M) = SDXL
(3469 M) + ControlNet (1332 M) + LLaVA (13045 M)
- •
RealESRGAN (21 M) = Generator (17 M) + Discriminator (4 M)
- •
DiffBIR (1683 M) = SDv2.1 (1303 M) + Restoration Module (17 M) + ControlNet (363 M)
- •
ResShift (174 M) = Denoiser (119 M) + AutoEncoder (55 M)

One can see that different approaches may differ by orders of magnitude in model scale.

#### Evaluation Settings

We conducted all inference measurements on 1 NVIDIA Tesla A100 GPU with 80GB of VRAM, batch size 1, PyTorch 2.4.0 [], and CUDA 12.4. We evaluated performance on a $256\text{px}\rightarrow 1024\text{px}$ image super-resolution task.

As mentioned earlier, for our diffusion model, we used 13 steps to generate upscales, while for other diffusion baselines we adopted the default settings from the corresponding papers and repositories: 4 sampling steps for ResShift, 100 sampling steps for SUPIR, and 50 sampling steps for DiffBIR. Note that for these baselines, the reported time spent to produce one image also includes inference of some extra networks such as VAE, LLaVa, BSRNet, etc.

Both our GAN and RealESRGAN models require 1 network evaluation to generate final images.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: “We note that we use semantic image-level captions. Global-level information appears to be not very useful for the SR task” What is the difference between image-level and global-level. Can you give an example? This is unclear.

I have never seen “p-value on SbS comparison” as a way to evaluate using human judgment. Why not just do this quantitatively and threshold the difference between checkpoints? The current way of stopping and evaluating seems incredibly arbitrary and subjective. Can the authors share a few works that use this paradigm?  as I have never encountered it. 
E.g. also in “ We conduct an SbS comparison between text-conditioned and unconditional models for both paradigms at all stages of training”
How exactly are the GAN models conditioned on the text?

One major issue is that because of the nature of the SR problem (ill-posed, many to many mapping etc) the types of errors SR models make can inherently invalidate the correctness of the image, e.g. numbers get blurred. It would make the paper significantly stronger if the authors can dig deeper into the entire performance of the SR models for specific tasks as they pertain to downstream tasks and make the evaluation more human interpretable. Right now it is being collapsed into a single number and some qualitative examples. It’s really challenging to make use of the findings in the paper for downstream research/applications. This could significantly enhance the contributions of this work (e.g. by answering *when should one use one paradigm over the other*)
But really what are the types of error differences between the two? 
E.g. figure 9 in appendix - the digits or letters. 
E.g. figure 12 in appendix - sometimes diffBIR and real esrgan , the results are switched, one does better than the other
We are not really getting conclusive results. Instead mixed findings. It would be great if the authors can taxonimize and dig deeper into when we should use GAN over Diffusion based SR models.
Line 1505 - figure caption “high frequency”
Not obvious to me why in section G, “G SUPER RESOLUTION OF SYNTHETIC IMAGES” the authors used those datasets to test for OOD? Why not use real data not synthetic data? Second, what if the synthetic data generated by diffusion models (e.g. SDXL as mentioned in the paper) may actually produce a distribution that is closer to that of the diffusion based SR model, thereby giving the diffusion based model a sort of advantage? 
Table 1 What is the dataset being used? Authors should mention this in the caption of the table.
soundness: 3
strengths: Overall, this type of work should be appreciated as it probes deeper into what the differences in paradigms are when it comes to the SR task. The authors ensure that the setup for both paradigms is as comparable as possible through the architecture, datasets, etc. In general, the writing is good but some parts were confusing.
summary: This paper systematically compares GANs and diffusion models for image super-resolution (ISR) under controlled, comparable conditions. The findings reveal that GANs, when trained with similar protocols, can match the quality of diffusion models while offering practical advantages, such as faster training and single-step inference. This study highlights key trade-offs between GAN and diffusion approaches, contributing valuable insights into their performance and efficiency in ISR tasks.
weaknesses: Because their experimental results were sometimes in favor of diffusion, sometimes in favor of GAN, and sometimes no difference was found, I have two suggestions that would significantly improve this paper.  First, the authors should taxonomize and explain to the reader when one paradigm should be preferred and in which scenarios. Otherwise, this type of work does not help us improve actual application performance. Second, for the authors to actually be able to make such claims, they need to use more fine-grained evaluation, e.g. the model outputs (SR) images should be used for a specific task like digit recognition, segmentation, etc. However, just comparing single valued PSNR, SSIM, LPIPS etc does not really tell us where these models are outperforming each other. That is also very evident from the qualitative results where within the same figure, both paradigms *visually outperform* each other. 

----not necessarily weaknesses----
Terminology: In Line 49, the term “fairness” might be misleading in this context. Instead, a term like “controlled conditions” or “standardized experimental setup” could better communicate the need for consistent variables, such as dataset size and model complexity, in comparing results.
-  In the related works section, the authors mention conflicting findings about whether text conditioning improves diffusion-based ISR. However, it’s unclear why these differences exist or what insights the current paper offers on this topic. A more thorough discussion or stance on this issue could add depth and relevance.
- Moving the “variations of SR” subsection earlier in the paper would help readers understand the exact ISR task being investigated, providing important context before diving into the model comparisons.
- In Line 130, “given a reference on training” is unclear.  

- A significant limitation is the use of proprietary models and datasets, making it difficult for others to replicate the experiments. For instance, the use of “internal foundation model” and a “proprietary dataset of 17 million…” lacks important detail.  Will this dataset be released? 

Figure Captions and Clarity:
Figures 1 and 4: These figures would benefit from more descriptive captions, highlighting key differences and the main takeaways.
Figure 4: The meaning of the green and grey indicators should be further clarified, as well as the criteria used to define convergence in the caption. I’m aware it is in the text.
Figures 2 and 5: Why do these not include the original HR?  
Figure 3: This figure is hard to interpret because it’s unclear what exact quantity or metric is being reported. Should be added to the caption. The corresponding section is also difficult to understand.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: Q1. The detail of "we did not encounter any difficulties with optimization" --> What can be the reasons for that nice success? Since many works and practices confirm that training GAN is very unstable and mode collapse is a well-known problem of GAN.

Q2. An experiment for the diffusion-based method in this study took 1 month to get the checkpoint, didn't it? Here many experiments were conducted for diffusion, how much time (months) it is estimated to take to complete all of the reported training? This is to provide some information for reproducibility for the community.
soundness: 2
strengths: + Conducting a comparison of GAN and Diffusion-based approaches for Super-Resolution with the same computational resource can provide good insight for the community.
+ The finding that given the same model size, GAN matches the performance of quality with the diffusion-based method is interesting.
summary: This study challenges the assumption that diffusion-based models inherently outperform GANs in Image Super Resolution (ISR). Noting that diffusion-based ISR models often use larger networks and longer training than GANs, the authors investigate whether these performance gains come from model design or simply increased resources. By controlling for architecture, model size, dataset, and computational budget, they find that GAN-based models can achieve comparable or even superior results. They also examine the influence of factors like text conditioning and augmentation, analyzing their effects on ISR performance and related tasks.
weaknesses: **Concerns**
I believe that the paper needs to have a thorough clarification. Specifically:

+ Claiming that GAN-based and Diffusion-based approaches give the comparison if using the same number of parameters might be relatively strong. It needs very careful investigations and evaluation. Because, if they give comparable performance, the community has no reason to use diffusion with much more cost for both training (much longer) and inference (much more sampling steps). A comparison with the same setup on some widely-used common dataset benchmarks at first might provide some insights and support rather than just collecting some custom massive datasets.

+ Conducting experiments on extremely huge data, i.e. 17 million images is a very high cost. The author could provide a comparison from a small to a larger number of data in their collected data to see the differences between GAN-based and Diffusion-based methods. For example, 100k, 1M, 2M, 5M, 17M, etc pickup of some of these settings might be reasonable to see if the results/findings are consistent. In practice and research, the number of images for the study is often not too huge up to 17M.

+ Were the methods in Table 1, and Table 4 "SUPIR, RealESRGAN, DiffBIR, ResShift" trained on the same dataset as the Diff (ours) and GAN (ours)? Also in these tables, it should be better to clearly state which one is GAN-based and diffusion-based would greatly improve readability.

+ Table 1 and Table 8 show that ResShift with about just 1/4 parameters (174M) already outperformed the GAN (ours) and Diffusion (ours) 614 and 630M on PSNR and SSIM. This may raise a question of whether scaling more can bring up the performance or not, which is contradicting as concerned in the paper doubts the performance gain that comes from scaling up model size. 

+ Figure 1 and Figure 4 are almost the same and seem to be redundant with no more information added.

**Other suggestions** 
+ For the whole paper, the current form seems to use all \cite{} making it very messy for all references. I think the use of \citep{} in latex would produce a more correct presentation of citations for many parts of the paper. Using \cite{} for cases where the citing author is subject (S), but \citep{} for other cases when referring to the paper.
+ Text in figures presented in the paper is too small, e.g. Figure 2, figure 3, figure 5.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Is it possible to do some "pretraining" for diffusion-based models similar to the pretraining on L1 for Gan-based method ? So that it can be more fair in evaluation.

2. As I mentioned above in Weaknesses section (bullet point 4), I would like to see how data scaling affect the performance of both GAN-based and diffusion-based ISR models.

3. I know it's a bit infeasible but showing comparison on other restoration tasks like image dehazing, deblurring, .. to see if the same phenonmena happens can be really valuable and strengthen the work.
soundness: 3
strengths: 1. The main contribution of this paper is a fair comparison between GAN and diffusion-based ISR models, controlling for architecture, dataset size, and computational resources.

2. The authors perform detailed ablation studies, particularly focusing on the effects of pretraining, augmentations, and training with full-resolution images.

3. The paper explores not only the overall performance of the models but also the impact of various design choices such as text conditioning and augmentation, which can be helpful for future work in the field.
summary: This paper investigates whether diffusion-based models truly outperform GAN-based models for the task of Image Super Resolution (ISR) in a fair setup. The authors conduct a rigorous comparison between diffusion and GAN-based models under controlled experimental settings, ensuring that both model types have equivalent architectures, dataset sizes, and computational budgets. In contrast with common belief, the primary findings reveal that GAN-based models can match or even surpass diffusion models in terms of quality when appropriately scaled. The paper also finds that text conditioning has little effect on model performance, while augmentations can sometimes hurt the training of diffusion models.
weaknesses: 1. The contribution is a bit limited as there is no really new and impactful insights presented in this work. Additionally, I'm not sure ICLR is a suitable venue for submitting this work, because it lean toward to more empirical side. May be MLSys is a better venue ? Again, I'm not sure.

2. Even though the experiment setting is quite fair, GAN-based models actually have one additional pretraining stage whereas diffusion model has to be trained from scratch, which could be a reason why diffusion-based model lacks behind GAN-based ones. 

3. The authors do not report how the GAN-based model perform without L1 pretraining stage, both qualitatively and quantitatively. Also, claiming that training GAN-based models for ISR does not face instability issues is quite a bold claim because without pretraining on L1 loss, I can imagine that it could be really unstable.

4. Though the paper highlights that both GANs and diffusion models benefit from scaling, it does not investigate how these models scale in terms of data. Like, how much data that GAN-based model starts to outperform diffusion ones. This could help to strengthen the work.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. The authors use Imagen as the architecture for both GAN and diffusion models in their SR experiments. I question whether it is appropriate to use Imagen for the GAN model, given it wasn't designed for GAN-based SR.
2. My major concern is that the experiments aren't entirely equivalent. Although the authors attempt to balance factors, the GAN model requiring an additional discriminator compared to the diffusion model, the training stability and results heavily depend on this discriminator. It's challenging to ensure true equality between GAN-based methods and diffusion models.
soundness: 3
strengths: 1. The authors provide valuable insight into the current research trend, highlighting the need for fair comparisons between diffusion and GAN models. 
2. They conduct several experiments, considering computational budget, dataset size, and notably, text conditioning as input prompts.
3. The authors incorporate human perception to regulate training, ensuring a fair comparison by controlling the training duration.
4. The experiments are sound, with the authors aiming to maintain consistent conditions, such as using DPM-Solver++ to control the inference steps.
summary: This paper explores the challenge of conducting a fair comparison between GAN and diffusion models in image super-resolution. It notes that diffusion models often employ larger networks and longer training times than GANs, raising questions about whether their performance is due to inherent advantages or increased resources. Through controlled comparisons, matching architecture, dataset size, and computational budget, the study observes that GANs can achieve results comparable or superior to diffusion models. It also examines factors like text conditioning and data augmentation, finding that commonly used image captions do not significantly impact performance.
weaknesses: 1. The presentation of this paper could be enhanced with more figures to better illustrate the concepts and experiments, especially those related to text-conditioning.
2. The comparison of super-resolved images in the manuscript and supplementary materials lacks numerical metrics, making evaluation challenging.
3. The current conclusion is somewhat vague. While the authors conduct various ablations between GAN and diffusion, it remains unclear under which specific conditions GAN can truly outperform diffusion, thus it needs to be clarified and streamlined.

Meta-review: This work focuses on the comparison of GAN and diffusion model in the task of super-resolution. Specifically, GANs and diffusion models with similar parameters are trained, and their performance is observed. The findings of the paper are that GANs achieve comparable results to diffusion models, but with the merits of faster convergence and single-step inference. Reviewers generally agree that the findings in this work are interesting, and the AC shares the same view.

This paper receives mixed ratings of (5, 6, 6, 6). After reading the paper, review, and rebuttal, the AC has concerns about the comprehensiveness of the analysis. For example:

1. This paper mainly focuses on one design choice for each model, and a rather limited training paradigms. For example, the conclusion could be different for different architecture (e.g., UNet vs DiT) and pre-training scheme (e.g., text-to-image pre-training). While the current findings is somewhat surprising and interesting, more thorough studies should be conducted.

2. The studies of parameter count can be more comprehensive. For example, while it is shown that GANs achieve comparable performance to diffusion models with around 600M - 700M parameters, it is unclear the same conclusion can be reached for say fewer or more parameters. This study is important to know the limits and potential of both models.

As an analytical paper, it is recommended to include more studies and comparison between the two paradigms in the future version. Based on the concerns, the AC would recommend a rejection.

Decision: Reject