Towards Realistic Data Generation for Real-World Super-Resolution

## Abstract

Abstract Existing image super-resolution (SR) techniques often fail to generalize effectively in complex real-world settings due to the significant divergence between training data and practical scenarios. To address this challenge, previous efforts have either manually simulated intricate physical-based degradations or utilized learning-based techniques, yet these approaches remain inadequate for producing large-scale, realistic, and diverse data simultaneously. In this paper, we introduce a novel Realistic Decoupled Data Generator (RealDGen), an unsupervised learning data generation framework designed for real-world super-resolution. We meticulously develop content and degradation extraction strategies, which are integrated into a novel content-degradation decoupled diffusion model to create realistic low-resolution images from unpaired real LR and HR images. Extensive experiments demonstrate that RealDGen excels in generating large-scale, high-quality paired data that mirrors real-world degradations, significantly advancing the performance of popular SR models on various real-world benchmarks.

## 1 Introduction

Real-world image Super-Resolution (Real SR) is a fundamental problem in image processing, aiming to enhance the resolution and quality of images in real-world scenarios [, , , , , , , , , , , , , , , , , , , , , ]. It has a wide range of applications across various fields, including photography [] and medical imaging [], which enhance human visual perception and the robustness of vision systems [, , , ]. However, traditional bicubic-interpolation-based Real SR methods have proven less effective in complex real-world scenarios due to the significant discrepancy between the bicubic pattern and real degradation [, , , , ]. Consequently, substantial efforts have been directed towards developing methods for generating more realistic data to improve the generalization ability of Real SR models [, , , , , , , , ].

To explore what kind of data contributes most to the SR model’s capability, we synthesize different sets of training data using [] and evaluate the performance of FSRCNN []. As shown in the top left of Figure 1, the red triangles represent test samples of the target domain, while rectangular boxes indicate training sets with different blur kernels and noise levels. The results clearly demonstrate that the closer the training data distribution is to the test data, the better the model’s performance. This underscores the importance of designing a method that can adaptively generate accurate data for different target domains. Therefore, an ideal data generation system for Real SR should meet the following criteria: I) Large-scale, to satisfy the extensive data requirements for training deep learning models [, , , ]; II) Realistic, to enable Real SR models to accurately learn the characteristics of real-world degradation  [, , ]; and III) Adaptive, to flexibly generate data with arbitrary given degradation patterns, improving generalization in target domains [, ].

Existing data generation methods for Real SR [, , , , , , , , , ], as shown in the bottom left of Figure 1, can be broadly categorized into the following: a) Manual Collection via Focal Length Adjustment: This approach involves using digital single-lens reflex cameras (DSLRs) with varying focal lengths to capture images, followed by alignment [, ]. While it can produce realistic paired data, it is labor-intensive and often results in scene monotony and image misalignment, failing to meet the large-scale data requirement. b) Hand-Crafted Physical-Based Degradation Modeling: This method employs various degradation models (e.g., noise, blur, bicubic, JPEG) applied in single-order or higher-order combinations [, ]. Although efficient in generating large data quantities, the synthetic data often fails to accurately reflect the complex degradation patterns of real-world images, and its lack of adaptability to specific domains limits effectiveness. c) Learning-Based Methods: Techniques involving Generative Adversarial Networks (GANs) [, ] and diffusion models [] are proposed to simulate realistic real-world degradation for low-resolution (LR) images. While these methods produce more realistic data compared to hand-crafted approaches, they often struggle to generalize to new and diverse domains, limiting their applicability in real-world scenarios. In summary, existing data generation methods face challenges in achieving both realism and adaptability: tailoring models to specific target domains may hinder their adaptability to new domains, and vice versa. Overcoming these challenges is crucial for advancing the field of Real SR.

In this paper, we introduce the unsupervised learning framework Realistic Decoupled Data Generator (RealDGen) to meet the large-scale, realistic, and adaptive data generation criteria. RealDGen enhances degradation realism and content fidelity by separately modeling content and degradation through unsupervised learning and integrating them into a diffusion model to generate paired data. The training involves two steps: first, pre-training degradation and content extractors using contrastive and reconstruction learning strategies to improve representation robustness; second, using these pre-trained extractors to condition the diffusion model with real LR degradation and content representations. To improve generalization to unknown LR distributions, the partial parameters of the extractors are fine-tuned. During data generation, unpaired HR and real LR images are used to extract content and degradation representations, which are then combined in the diffusion model. This process produces data that marries HR content with arbitrary LR degradation, improving adaptability to new domains. Extensive experiments show that RealDGen outperforms previous methods in generating realistic paired data and enhancing the performance of SR models in real-world scenarios.

The contributions of this paper can be summarized as follows:

Figure: Figure 1: The top left and bottom left are SR performance on different train data and degradation distribution of different methods. The right is our unsupervised data generation framework RealDGen.
Refer to caption: extracted/5942525/Styles/Fig/fig1.png

- •
We propose a novel unsupervised Realistic Decoupled Data Generator (RealDGen) to adaptively generate large-scale, realistic, and diverse data for real-world super-resolution.
- •
We introduce well-designed content and degradation extraction strategies and a novel content-degradation decoupled diffusion model to generate realistic LR with arbitrary unpaired LR and HR conditions.
- •
Compared with previous methods, our method significantly advances the generalization ability of popular SR models, achieving the best performance on real-world benchmarks.

Figure: Figure 2: An overview of our unsupervised training pipeline. We first train on the content and degradation extractors, then train Decoupled DDPM while fine-tuning the partial parameters of the extractors. RealDGen adaptively generates realistic LR images with arbitrary given real LR and unpaired HR conditions.
Refer to caption: extracted/5942525/Styles/Fig/framework.png

## 2 Method

In this paper, we introduce a novel Realistic Decoupled Data Generator (RealDGen) for real-world super-resolution to adaptively generate large-scale, realistic, and diverse real paired data. In particular, well-designed content and degradation extractor learning strategies are proposed to capture robust content and degradation representations in the real world. A novel content-degradation decoupled diffusion model is proposed to generate realistic LR with arbitrary unpaired LR and HR conditions. The training process is divided into two distinct phases: (a) Content and Degradation Extractor Pre-training and (b) Decoupled DDPM Training and Extractor Fine-tuning, as shown in Figure 2.

### 2.1 Content and Degradation Extractor Pre-training

To capture content and degradation representations, we propose dedicated degradation and content extractors, denoted as $E_{deg}$ and $E_{cont}$, respectively. We employ reconstruction learning for training $E_{cont}$, as shown in Figure 2 (a). Specifically, for a given high-resolution (HR) image $\mathcal{X}\in\mathbb{R}^{C\times H\times W}$, we degrade it to a low-resolution (LR) counterpart $\mathcal{X}_{lr}\in\mathbb{R}^{C\times h\times w}$ by Real-RSRGAN [] degradation model $\mathcal{D}$ with diverse synthetic degradations. Subsequently, $E_{cont}$ is engaged to extract the content representation $F_{{cont}}$ from $\mathcal{X}_{lr}$. Thereafter, a HR reconstruction network $Rec_{hr}$ is harnessed to reconstruct the HR image $\hat{\mathcal{X}}\in\mathbb{R}^{C\times H\times W}$ from $F_{{cont}}$, as follows:

$$ $F_{{cont}}=E_{cont}\left(\mathcal{X}_{{lr}}\right),\hat{\mathcal{X}}=Rec_{hr} \left(F_{cont}\right)\,.$ (1) $$

The objective is to minimize the reconstruction loss $\mathcal{L}_{{rh}}$ between the reconstructed image $\hat{\mathcal{X}}$ and the original high-resolution image $\mathcal{X}$, as follows:

$$ $\mathcal{L}_{{rh}}=\frac{1}{N}\sum_{i=1}^{N}\left(\hat{\mathcal{X}}_{i}- \mathcal{X}_{i}\right)^{2}\,,$ (2) $$

where $N$ denotes the batch size, which we set to 64. After training, $E_{cont}$ learns a robust content representation under diverse synthetic degradations.

Considering the variability of degradation in diverse scenarios and imaging devices in the real world, we advocate for contrastive learning to curate positive and negative samples for training $E_{deg}$, which guarantees the uniqueness of the degradation representations. Specifically, for a HR image $\mathcal{X}\in\mathbb{R}^{C\times H\times W}$, we generate an LR image $\mathcal{X}_{{lr}}\in\mathbb{R}^{C\times h\times w}$ by $\mathcal{D}$ with parameter $\theta$ as the anchor $\mathcal{X}_{{anc}}$. We further obtain a set of negative samples $\mathcal{X}_{{neg}_{i}}\in\mathbb{R}^{C\times h\times w}$ by applying $\mathcal{D}$ with different parameters $\theta_{i}$ to $\mathcal{X}$, and a set of positive samples $\mathcal{X}_{{pos}_{i}}\in\mathbb{R}^{C\times h\times w}$ by applying the degradation $\mathcal{D}$ with same parameter $\theta$ to different HR images $\mathcal{X}^{\prime}_{i}\in\mathbb{R}^{C\times H\times W}$, as follows:

$$ $\begin{array}[]{c}\mathcal{X}_{{neg}}=\{\mathcal{D}\left(\mathcal{X},\theta_{1 }\right),\mathcal{D}\left(\mathcal{X},\theta_{2}\right),\ldots,\mathcal{D} \left(\mathcal{X},\theta_{n}\right)\}\\ \mathcal{X}_{{pos}}=\{\mathcal{D}\left(\mathcal{X}^{\prime}_{1},\theta\right), \mathcal{D}\left(\mathcal{X}^{\prime}_{2},\theta\right),\ldots,\mathcal{D} \left(\mathcal{X}^{\prime}_{n},\theta\right))\}\,.\end{array}$ (3) $$

The objective is to minimize the contrastive loss $\mathcal{L}_{{cl}}$ to drive $E_{deg}$ learn the uniqueness of the degradation representations in LR images, suppressing the interruption of content, as follows:

$$ $\small\mathcal{L}_{{cl}}=\sum_{i=1}^{n}\max\left(0,d\left(E_{deg}\left( \mathcal{X}_{{anc}}\right),E_{deg}\left(\mathcal{X}_{{pos}_{i}}\right)\right)- d\left(E_{deg}\left(\mathcal{X}_{{anc}}\right),E_{deg}\left(\mathcal{X}_{{neg} _{i}}\right)\right)+{margin}\right)\,,$ (4) $$

where $d$ symbolizes the L2 distance, $n$ is the number of samples, and we empirically set $n$ and $margin$ to $3$ and $0.01$, respectively. Furthermore, to drive $E_{deg}$ to learn the complete degradation representations, we utilize the reconstruction learning strategy as aforementioned for supervising $E_{deg}$. Specifically, we utilize the pre-trained $E_{cont}$ to learn the content representation of $\mathcal{X}_{{hr}}$ and $E_{deg}$ to learn the degradation representations of $\mathcal{X}_{{lr}}$. We then employ a low-resolution reconstruction network, $Rec_{lr}$, to combine these representations and reconstruct the LR image, $\hat{\mathcal{X}}_{{lr}}$, as follows:

$$ $\hat{\mathcal{X}}_{lr}=Rec_{lr}\left(E_{deg}\left(\mathcal{X}_{{lr}}\right),E_ {cont}\left(\mathcal{X}_{{hr}}\right)\right)\,.$ (5) $$

The objective is to minimize the reconstruction loss $\mathcal{L}_{{rl}}$ to drive $E_{deg}$ learn the completeness of the degradation representations as follows:

$$ $\mathcal{L}_{{rl}}=\frac{1}{N}\sum_{i=1}^{N}\left(\hat{\mathcal{X}}_{{lr}_{i}} -\mathcal{X}_{{lr}_{i}}\right)^{2}\,.$ (6) $$

After training with well-designed learning strategies, the proposed encoders effectively capture robust content and degradation representations. More details are provided in Section A.1 and Section A.2 of the appendix.

Figure: Algorithm 1 Decoupled DDPM Training

### 2.2 Decoupled DDPM Training and Extractor Fine-tuning

We introduce a content-degradation Decoupled Diffusion Probabilistic Model (Decoupled DDPM) to generate real LR images. In detail, given real LR images ${\mathcal{X}}_{{lr}}$ from real-world $q$, we extract their content representation $F_{cont}$ and degradation representation $F_{deg}$ by pre-trained $E_{cont}$ and $E_{deg}$, respectively. To enhance the generalization of $E_{deg}$ and $E_{cont}$ on unseen real distributions, partial parameters are fine-tuned, as shown in Figure 2. A modulation block $\mathcal{M}$ is introduced to adequately incorporate degradation representation into the content, formulated as:

$$ $\mathbf{c}=\mathcal{M}\left(E_{deg}\left(\mathbf{x}_{lr}\right),E_{cont}\left( \mathbf{x}_{hr}\right)\right)\,.$ (7) $$

Then, this fused image representation is utilized as a condition $\mathbf{c}$ for training our Decouple DDPM to generate LR images. To make it clear, we illustrate the detailed training procedure of the Decouple DDPM, as shown in Algorithm 1. A detailed presentation of $\mathcal{M}$ and fine-tuning is provided in Sections A.3, A.4 and  A.5 of the appendix.

### 2.3 Data Generation

We propose a novel strategy to generate realistic LR images using unpaired LR and HR images by decoupling content and degradation. First, we extract the degradation representation from a real-world LR image and the content representation from an HR image. These representations are combined in the modulation module $\mathcal{M}$ to serve as the condition $\mathbf{c}$ for the diffusion model to generate LR images. The generated LR images retain the content of the HR image and the degradation of the real LR image, as shown on the right of Figure 1. To enhance fidelity, following [], we denoise from an initial LR image $\mathbf{x}_{t}$ with $t$ steps of noise rather than from pure Gaussian noise. This initial LR image is degraded by ${\mathcal{D}}$. In Sec.3.4, we analyze the step number $T$. Details of our data generation pipeline are in Algorithm 2.

Although using content and degradation conditions improves the controllability and fidelity of Decoupled DDPM, the inherent stochasticity of the diffusion model [, ] can still introduce tiny artifacts and content distortion. To mitigate this, we propose a filtering mechanism. For each generated LR image, we re-extract content and degradation representations, then calculate the degradation error with the real LR image and the content error with the HR image. By selecting samples with the smallest errors, we reduce diffusion stochasticity and produce higher-fidelity LR images. More details will be presented in Section A.6 of the appendix.

## 3 Experiments and Analysis

### 3.1 Experiments Settings

Training Details. We have collected about 152,000 real low-resolution images from both public datasets [, , ] and those captured using smartphones. We train our extractors and Decouple DDPM on 16 NVIDIA V100 GPUs and set the learning rate and batch size to $1\times 10^{-4}$ and $16$, respectively. More details of the training setting are presented in Section  A.1 and  A.3 of the appendix.

To ensure a fair comparison, we compare our approach with existing methods by using the widely-used DIV2K dataset [] as content reference and real LR images as degradation reference to create the paired training data for training various popular SR models. To maintain a rigorous and fair comparison, we have maintained consistency in all experimental settings and environments, with the exception of data generation. We utilize the public BasicSR (^1^11https://github.com/XPixelGroup/BasicSR) for training Real-SR methods with 16 NVIDIA V100 GPUs.

Compared Data Generation Methods. We compare our methods with some state-of-the-art real-world data generation methods, including Hand-Crafted Physical-Based Degradation Models (BSRGAN [] and Real-ESRGAN []) and learning-based degradation diffusion models proposed by [], denoted as SynDiff for convenience.

Real-SR Models.
To comprehensively validate the effectiveness of the generated data, we carefully select four classic and representative backbone architectures for evaluation, including representative CNN-based model RRDB [], representative transformer-based model SwinIR [] and HAT [], representative lightweight model SwinIR-L []. To be fair, we conduct comparative evaluations under consistent experimental conditions and settings. We utilize L1Loss [, ] and perception loss [, , ] for training PNSR-oriented and Perception-oriented real SR models, respectively.

Metrics. For the PNSR-oriented Real SR model, we adopt PSNR [] and SSIM [] to quantitatively evaluate the performance. For the perception-oriented Real-SR model, we adopt LPIPS [] and FID [] to quantitatively evaluate the performance. DISTS [] and CLIP-Score [] are further introduced to evaluate the accuracy of generated data. Note that the higher the PSNR, SSIM, and CLIP-Score, the better, and the lower the LPIPS, FID, and DISTS, the better.

Evaluation. We utilize two public benchmarks to evaluate the performance of real-world image super-resolution methods, including the real-world dataset RealSR [] and DRealSR [] captured by digital single-lens reflex cameras (DSLRs). To further improve the diversity and quantity of real degradation scenarios, we have collected 891 pairs of data captured by smartphones for evaluation, denoted as SmartPhone.

**Table 1: Quantitative comparisons of PSNR-oriented and Perceptual-oriented training SR models on three real-world image super-resolution benchmarks. The best results are highlighted in bold.**
|  | RealSR [] | DRealSR [] | SmartPhone |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| PSNR-oriented Training | PSNR $\uparrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ |
| SwinIR (Real-ESRGAN []) | 24.395 | 0.7760 | 26.944 | 0.8308 | 27.395 | 0.8338 |
| SwinIR (BSRGAN [])) | 25.852 | 0.7808 | 27.985 | 0.8308 | 28.049 | 0.8407 |
| SwinIR (SynDiff []) | 25.589 | 0.7687 | 28.301 | 0.8309 | 28.566 | 0.8453 |
| SwinIR (Ours) | 26.094 | 0.7822 | 28.721 | 0.8341 | 28.737 | 0.8489 |
| RRDB (Real-ESRGAN []) | 24.579 | 0.7614 | 27.131 | 0.8193 | 27.841 | 0.8378 |
| RRDB (BSRGAN [])) | 25.406 | 0.7685 | 27.523 | 0.8017 | 28.029 | 0.8278 |
| RRDB (SynDiff []) | 25.488 | 0.7691 | 28.078 | 0.8257 | 28.303 | 0.8426 |
| RRDB (Ours) | 26.238 | 0.7747 | 28.727 | 0.8340 | 28.754 | 0.8507 |
| HAT (Real-ESRGAN []) | 24.893 | 0.7726 | 27.339 | 0.8215 | 27.781 | 0.8336 |
| HAT (BSRGAN [])) | 25.997 | 0.7816 | 28.135 | 0.8273 | 28.137 | 0.8369 |
| HAT (SynDiff []) | 25.790 | 0.7584 | 28.506 | 0.8286 | 28.508 | 0.8471 |
| HAT (Ours) | 26.140 | 0.7832 | 28.802 | 0.8345 | 28.767 | 0.8489 |
| SwinIR-L (Real-ESRGAN []) | 24.367 | 0.7723 | 27.018 | 0.8244 | 27.581 | 0.8409 |
| SwinIR-L (BSRGAN [])) | 25.651 | 0.7800 | 27.813 | 0.8301 | 28.118 | 0.8437 |
| SwinIR-L (SynDiff []) | 25.281 | 0.7516 | 28.170 | 0.8244 | 28.474 | 0.8487 |
| SwinIR-L (Ours) | 26.025 | 0.7810 | 28.869 | 0.8328 | 28.868 | 0.8522 |
|  | RealSR[] | DRealSR[] | SmartPhone |  |  |  |
| Perceptual-oriented Training | LPIPS $\downarrow$ | FID $\downarrow$ | LPIPS $\downarrow$ | FID $\downarrow$ | LPIPS $\downarrow$ | FID $\downarrow$ |
| SwinIR (Real-ESRGAN []) | 0.3037 | 69.965 | 0.3219 | 39.175 | 0.4053 | 78.242 |
| SwinIR (BSRGAN [])) | 0.2945 | 79.833 | 0.3023 | 38.541 | 0.3043 | 76.871 |
| SwinIR (SynDiff []) | 0.3835 | 103.179 | 0.3801 | 54.588 | 0.3129 | 83.485 |
| SwinIR (Ours) | 0.2536 | 69.736 | 0.2660 | 38.257 | 0.2964 | 74.778 |
| RRDB (Real-ESRGAN []) | 0.3480 | 82.056 | 0.3551 | 39.310 | 0.3480 | 77.573 |
| RRDB (BSRGAN [])) | 0.3041 | 77.412 | 0.3127 | 36.528 | 0.3381 | 78.812 |
| RRDB (SynDiff []) | 0.4004 | 98.798 | 0.4017 | 56.573 | 0.3511 | 88.431 |
| RRDB (Ours) | 0.2972 | 76.973 | 0.3077 | 36.259 | 0.3125 | 76.723 |
| HAT (Real-ESRGAN []) | 0.3066 | 79.209 | 0.3219 | 41.862 | 0.4022 | 87.950 |
| HAT (BSRGAN [])) | 0.2852 | 80.192 | 0.2835 | 41.723 | 0.3049 | 81.247 |
| HAT (SynDiff []) | 0.3332 | 93.763 | 0.3465 | 50.808 | 0.3171 | 85.587 |
| HAT (Ours) | 0.2457 | 67.573 | 6.2587 | 41.319 | 0.2816 | 76.873 |
| SwinIR-L (Real-ESRGAN []) | 0.3108 | 79.491 | 0.3234 | 42.986 | 0.4021 | 87.531 |
| SwinIR-L (BSRGAN [])) | 0.3013 | 84.195 | 0.2978 | 43.246 | 0.3146 | 83.444 |
| SwinIR-L (SynDiff) | 0.3793 | 98.646 | 0.3748 | 52.464 | 0.3190 | 80.708 |
| SwinIR-L (Ours) | 0.2795 | 75.779 | 0.2862 | 42.542 | 0.3047 | 78.632 |

**Table 2: Quantitative comparisons of the accuracy of generating real LR images on three real-world benchmarks under six distinct similarity metrics.**
| Benchmark | Methods | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | FID $\downarrow$ | DISTS $\downarrow$ | CLIP-Score $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Real-ESRGAN [] | 24.3974 | 0.6798 | 0.3881 | 157.5237 | 0.2618 | 0.7552 |
|  | BSRGAN [] | 23.2665 | 0.6043 | 0.5093 | 177.9996 | 0.3048 | 0.6783 |
|  | SynDiff [] | 25.2461 | 0.7588 | 0.2371 | 119.9950 | 0.1944 | 0.7984 |
| RealSR [] | Ours | 26.1615 | 0.7940 | 0.2226 | 107.2375 | 0.1865 | 0.8238 |
|  | RealESRGAN [] | 26.3510 | 0.6935 | 0.3842 | 60.2665 | 0.2437 | 0.7710 |
|  | BSRGAN [] | 26.1970 | 0.6869 | 0.4319 | 62.9189 | 0.2646 | 0.7137 |
|  | SynDiff [] | 28.0125 | 0.8136 | 0.1739 | 31.0855 | 0.1673 | 0.8462 |
| DRealSR [] | Ours | 28.6629 | 0.8360 | 0.1497 | 30.0078 | 0.1567 | 0.8577 |
|  | RealESRGAN [] | 26.0680 | 0.5981 | 0.5202 | 106.3958 | 0.2855 | 0.7231 |
|  | BSRGAN [] | 27.6215 | 0.7179 | 0.4399 | 105.3289 | 0.2760 | 0.6771 |
|  | SynDiff [] | 28.5020 | 0.8075 | 0.2423 | 70.6379 | 0.2044 | 0.8213 |
| SmartPhone | Ours | 29.0054 | 0.8292 | 0.2121 | 69.1162 | 0.1964 | 0.8334 |

Figure: Figure 3: Visual comparison of generated LR. Our method achieves the best visual results with realistic degradation and high fidelity. Please zoom in for better visualization.
Refer to caption: extracted/5942525/Styles/Fig/syn_compared_3x6.png

Figure: Figure 4: Visual comparison of Real SR based on different data generation methods. Real SR training using our data achieves the best visual results. Please zoom in for better visualization.
Refer to caption: extracted/5942525/Styles/Fig/SR_compared.png

**Table 3: Ablation study on $T$ in our proposed Decoupled DDPM during inferencing.**
| $T$ | 0-200 | 0-300 | 0-400 | 0-500 |
| --- | --- | --- | --- | --- |
| PSNR $\uparrow$ | 28.762 | 28.868 | 28.542 | 26.713 |
| SSIM $\uparrow$ | 0.8501 | 0.8522 | 0.8435 | 0.7931 |

### 3.2 Quantitative Results

Generalization Ability of Real SR Models. We compare our method with the existing data generation methods on both PSNR-oriented and Perceptual-oriented Real SR models to validate the superiority of our method in boosting the generalization capabilities for real-world image super-resolution, and the results are shown in Tab. 1. We can observe that our method comprehensively improves the performance of PSNR-oriented and Perceptual-oriented SR models across three benchmarks. It’s worth noting that our approach achieves a significant performance improvement, including 0.75 dB in RRDB [] on the PSNR of the RealSR benchmark; 0.296dB and 0.699 dB in SwinIR [] and light-weight SwinIR-L on the PSNR of the RealSR benchmark. Furthermore, our methods also significantly improve LPIPS and FID in four SR models, including 0.0395 and 12.619 in HAT [] on the LPIPS and FID of RealSR benchmark. This demonstrates that our method effectively improves the generalization capacity of popular SR models.

Accuracy of Generated Real LR. To validate our method’s superiority in generating accurately realistic and matched real LR, we conduct comparisons with existing methods in terms of the accuracy of the generated data on six metrics for evaluation. Specifically, we utilize HR images in the three real-world datasets as the content reference and employ the existing methods to synthesize generated LR images, while our method is further able to utilize the degradation reference from real LR. As shown in Tab. 2, we can observe that our method achieves the best performance on three datasets under six metrics, including distribution distance metric: FID, image structure and texture similarity metric: DISTS, and perceptual metric: LPIPS and CLIP-score, etc. It demonstrates the superiority of our Decoupled DDPM and verifies that our generation method is closer to target real LR images.

Generalization Ability on Out-of-distribution Data. Although we have collected about 152,000 real LR images for training, unseen scenarios often arise in real-world applications. To further explore our generalization capabilities on out-of-distribution data, we randomly extract 200 images from the unseen real-world video SR data [] for evaluation, denoted as RealVSR. Then, we capture the degradation representing those images to generate the training data and compare it with existing methods on the SwinIR-L model, and the results are shown in Tab. 5. We can observe that our method still achieves the best performance on the out-of-distribution RealVSR benchmark, surpassing the existing state-of-the-art methods by 0.2214 and 0.0055 in PSNR and SSIM, respectively. This demonstrates the ability of our method to generalize for unseen scenarios in the real world.

### 3.3 Qualitative Results

We visualize the results of the generated LR images, as shown in Figure 3. It is clear that the Real-ESRGAN and BSRGAN, which are unable to perceive real degradation, exhibit a significant gap compared to real LR images. The SynDiff also struggles to adaptively capture the real degradation and content representation, resulting in unreal degradation distribution and low fidelity, as shown in the third row in Figure 3. However, benefiting from our adaptive perception ability of the degradation of real LR image and content of HR image, our RealDGen achieves the best visual effect in terms of both the realism of degradation and the fidelity of content. Furthermore, to demonstrate the visual effects of Real SR model, we conduct visualizations of the SwinIR model in the RealSR and DRealSR benchmarks, as shown in Figure 4. It is clear that models based on SynDiff, BSRGAN, and Real-ESRGAN exhibit artifacts and blur. However, our method achieves the best visual quality without artifacts. The textual scene in the second row of Figure 4 also demonstrates the superiority of our method for text texture. This showcases the superior visual effects of our method in real-world scenarios. More visual comparison results are presented in Section A.7 of the appendix.

### 3.4 Ablation Study

Degradation and Content Extractor. Our core idea is utilizing the degradation extractor $E_{deg}$ and content extractor $E_{cont}$ to extract degradation and content representation to control Decoupled DDPM in generating realistic LR. To verify this, we eliminate components $E_{deg}$ and $E_{cont}$, subsequently training the decoupled DDPM and fine-tuning the remaining extractor. For quick evaluation, we employ SwinIR-L as the baseline and evaluate on the Smartphone benchmark. The results in Tab. 4 demonstrate that the absence of $E_{deg}$ and $E_{cont}$ will cause mismatch and unreal degradation and content distortion problems and result in performance degradation, thereby affirming the indispensability of $E_{deg}$ and $E_{cont}$ in our method.

$T$ in Data Generation. As illustrated in Sec. 2.3, in inferencing, we denoise from an initial LR image $\mathbf{x}_{t}$ with $t$ step noise added and perform t steps of denoising to improve the fidelity of Decouple DDPM, where $t$ is selected from the range of 0 to $T$. To explore it, we set $T$ to 200, 300, 400, and 500 and conduct ablation experiments, as presented in Tab. 4. It is observed that the optimal $T$ is 300. A lower $T$ will result in insufficient generation of degradation, while a high $T$ will lead to a decrease in image fidelity. Moreover, we posit that the value of $T$ should not be a manual set but rather a variable that correlates with the degradation process. In future work, we will consider proposing the auto-selection mechanism of $T$ that is specific to the degradation characteristics of given real LR and HR images, enhancing the controllability and fidelity of the model.

**Figure 5: User study of generated real LR.**
| RealVSR | PSNR $\uparrow$ | SSIM $\uparrow$ |
| --- | --- | --- |
| Real-ESRGAN [] | 22.7402 | 0.6939 |
| BSRGAN [] | 22.6564 | 0.6779 |
| SynDiff [] | 22.4607 | 0.6704 |
| Ours | 22.9616 | 0.6994 |

### 3.5 User study

To demonstrate the superiority of RealDGen in generating accurate and realistic low-resolution images, we conduct a user study involving 10 real-world LR images randomly chosen from existing benchmarks. 10 volunteers are asked to rate each scene individually (0: not similar at all; 2: not very similar; 4: slightly similar; 6: moderately similar; 8: similar; 10: extremely similar). Then, we aggregate the scores from all volunteers, and the results are shown in Figure 5. We can observe that the previous methods are unable to adaptively and accurately perceive real degradation, resulting in low fidelity in the generated real LR data, which leads to a general perception among users that there is a significant gap compared to real LR images. However, our RealDGen can adaptively capture the real degradation to accurately generate realistic LR images, resulting in an average score of 8.46 from human evaluators, surpassing previous approaches and thereby highlighting the improved visual quality that our method offers.

## 4 Conclusion

In this paper, we introduce a novel RealDGen to adaptively generate large-scale, high-quality paired data with arbitrary real LR as degradation reference and unpaired HR as content reference. Well-designed extractors and strategies are proposed to facilitate the extraction of robust content and degradation representations. A content-degradation decoupled diffusion model is proposed to adaptively generate realistic LR with given unpaired LR and HR conditions. Extensive experiments demonstrate that RealDGen not only achieves the best performance in generating realistic and accurate real LR images but also comprehensively improves the generalization ability of various popular SR models on real-world benchmarks. In addition, benefiting from the unsupervised learning of our method and the convenience of real LR image collection, it is easy to collect more real LR images with various real degradations to enhance generalization capability further.

Limitation: Due to the high stochasticity in the diffusion model, RealDGen sometimes finds it difficult to preserve tiny textures. In future work, we will incorporate perceptual loss for training and propose the auto-selection mechanism of the denoise step to perceive different given degradation images for further improving controllability and fidelity.

## Appendix A Appendix

### A.1 Training detail of Content and Degradation Extractor

we initially contemplate leveraging the auto-regressive architecture of VQGAN [] as our Content Extractor $E_{cont}$ to capture content representations. However, our experimental endeavors reveal that the Generative Adversarial Network (GAN) in VQGAN impedes fine-tuning and the extraction of realistic content representations. Consequently, we elect to employ the encoder component of VQVAE [] as $E_{cont}$. Given that these extractors have not been trained on degradation scenarios, they are not immediately suitable for our method. To surmount this challenge, we have meticulously crafted a fine-tuning strategy predicated on reconstruction learning. Specifically, we adopt Real-ESRGAN to degrade high-resolution (HR) images to generate paired datasets for training. During training, we maintain the decoder of VQVAE in a frozen state while fine-tuning the encoder, using HR images for supervised learning, with the objective loss function being the L2 Loss. Our fine-tuning regimen is conducted with a learning rate of $1\times 10^{-5}$, utilizing 8 NVIDIA V100 GPUs, which spent approximately one week for training. The reconstructed samples rendered by our fine-tuned Content Extractor are delineated in Fig. 6. It is evident that after fine-tuning, VQVAE adeptly captures content representations in degraded scenarios and adeptly reconstructs high-resolution images.

Figure: Figure 6: Visutal comparison with Input LR image, VQVAE, our pretrained VQVAE, and GT. VQVAE, after training on our designed learning strategy, can better capture the robust content representations to reconstruct high-quilty HR.
Refer to caption: extracted/5942525/Styles/Fig/VQVAE_rec.png

We introduce a novel Degradation Extractor denoted as $E_{deg}$, which is comprised of a feature extraction layer integrated with 16 residual blocks and a mapping function facilitated by adaptive polling and a 4-layer convolutional structure, as shown in Fig. 7. Specifically, given the input of a low-resolution image into the Degradation Extractor, it engenders the output degradation representation $F_{deg}\in\mathbb{R}^{1\times 1\times 2048}$. To ensure the extraction of comprehensive and unique degradation representations in the LR image, we employ both reconstruction learning and contrastive learning methodologies in training our network. The training is executed on 8 NVIDIA V100 GPUs over the course of approximately seven days, with the learning rate configured at $1e-4$.

Figure: Figure 7: Illustration of our proposed degradation extractor.
Refer to caption: extracted/5942525/Styles/Fig/deg.png

### A.2 Detail of HR and LR Reconstruction Network

To better adapt to the content extractor, that is, the encoder of the VQVAE, the decoder of VQVAE is adopted as HR reconstruction network $Rec_{hr}$ tasked with the reconstruction of high-resolution images. Given that the LR reconstruction network is tasked with reconstructing the LR image from content and degradation representations, we have adopted the aforementioned modulation network as our LR reconstruction network $Rec_{lr}$.

### A.3 Training Details of Decouple DDPM

During the training of the Decoupled DDPM, we configure the maximum diffusion step to be 500, span the training over 100 epochs, and utilize a learning rate of $1e-4$ for the decoupled diffusion model. For the fine-tuning of the extractors, we apply a more refined learning rate of $1e-6$. The batch size is defined as 8, and the entire training regimen is on 8 NVIDIA V100 GPUs, which typically consume around 14 days to complete.

We have amassed a collection of approximately 152,000 real low-resolution images sourced from publicly available datasets such as [, , ] and those captured using smartphones. To elaborate, we have extracted all low-resolution images from these datasets and have cropped each to a uniform size of
$256\times 256$, resulting in a total of 110,000 images. Subsequently, to enhance the diversity of real-world degradation distribution, we have additionally procured 42,000 low-resolution images, each of the same
$256\times 256$ size. Collectively, this corpus of 152,000 low-resolution images serves as the training data for our Decoupled DDPM.

### A.4 Detail of Modulation Block

As detailed in Section 2, we commence with real low-resolution images, denoted as $\mathcal{X}_{\text{lr}}$, which are derived from the real-world degradation distribution $q$. Utilizing our pre-trained extractors $E_{cont}$ and $E_{deg}$, we meticulously extract the respective content and degradation representations, $F_{{cont}}$ and $F_{{deg}}$. To authentically emulate the intricacies of the real imaging process, we introduce a modulation block, denoted as $\mathcal{M}$, which seamlessly integrates the degradation representation into the content representation. The integration is mathematically articulated as follows:

$$ $\mathbf{c}=\mathcal{M}\left(E_{{deg}}\left(\mathbf{x}_{{lr}}\right),E_{{cont}} \left(\mathbf{x}_{{hr}}\right)\right)$ (8) $$

In this section, we proceed to elucidate the intricate mechanisms underpinning the modulation block $\mathcal{M}$. $\mathcal{M}$ is meticulously constructed from a series of four modulation layers, each comprising a convolutional layer, an activation function, and a modulation unit, as shown in Fig. 8. The inputs to $\mathcal{M}$ encompass both the degradation and content representations. Within each modulation unit, the degradation representation is subjected to a sophisticated fusion process. Ultimately, the block culminates in its output through a final convolutional operation, synthesizing the enhanced representation.

Figure: Figure 8: llustration of our modulation block.
Refer to caption: extracted/5942525/Styles/Fig/modulation.png

### A.5 Analysis and Details of Fine-Tuning Extractor

During the second phase of training, we froze the majority of parameters in the content extractor, allowing only the first block within the VQVAE to undergo fine-tuning. Simultaneously, we applied a similar approach to the degradation extractor, restricting fine-tuning to the final convolutional layer. To validate the effectiveness of fine-tuning, we conduct an ablation experiment. In the second phase, we completely freeze the extractor and train only the Decoupled DDPM. We find that the realism of the generated degradation decreases because the extractor is trained exclusively on synthetic datasets, which results in its inability to extract real degradation representations and content. Consequently, the subsequent DDPM also has difficulty fitting the real LR accurately. Specifically, we test on the RealSR dataset using SwinIR-L, resulting in a 0.25dB performance degradation in PSNR with fine-tuning.

### A.6 Stochasticity of Diffusion

As discussed in Sec. 4, to mitigate the stochasticity of diffusion, we propose a filtering mechanism to eliminate outlier data. To validate this, we conduct experiments on the RealSR dataset, calculating the similarity and data distribution compared to the real LR images in RealSR. Additionally, we verify the effectiveness using the downstream SR network SwinIR-L, with results presented in Tables 7 and 7. It can be observed that the data generated without the filtering mechanism exhibits lower similarity and further divergence from the real LR images, as well as reduced performance in the downstream SR network. This confirms the effectiveness and practicality of our proposed filtering mechanism in real-world scenarios.

**Table 6: The similarity results of ablation study on proposed filtering mechanic.**
| RealSR | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | FID $\downarrow$ |
| --- | --- | --- | --- | --- |
| w/o Filtering | 25.8965 | 0.7886 | 0.2316 | 116.8970 |
| Ours | 26.1615 | 0.7940 | 0.2226 | 107.2375 |

### A.7 More Visual Comparison

Here, we present more visual results on the DRealSR benchmark to demonstrate the superiority of our method in adaptively generating accurate and realistic LR images, as shown in Fig. 9 and 10.

Figure: Figure 9: Visual comparison of generated LR on DRealSR.
Refer to caption: extracted/5942525/Styles/Fig/DRealSR_data_gen.png

Figure: Figure 10: Visual comparison of generated LR on DRealSR.
Refer to caption: extracted/5942525/Styles/Fig/DRealSR_data_gen2.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please refer to the Weaknesses section.
soundness: 2
strengths: - The paper is well-structured, making the methodology and findings easy to understand.
- The paper achieves competitive SR results under an unpaired setting, which is practical and advantageous in real-world applications.
summary: This paper introduces an unsupervised learning framework designed to generate realistic LR images for real-world SR. This paper utilizes a decoupled approach for content and degradation extraction, incorporating them into a diffusion model to produce paired data that closely represents real-world degradations. Experimental results indicate that the proposed method improves the generalization and performance of SR models across various real-world benchmarks.
weaknesses: - The paper argues that content and degradation can be decoupled by content and degradation extractor. 
However, it is unclear how the content extractor’s encoder is guaranteed to capture only pure content information.

- Equation (5) utilizes X_{hr}, but no explanation or justification is provided for its use in the main manuscript.

- The paper would benefit from a comparison of the total parameters of RealDGen with those of other methods, as this would provide additional context on scalability.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 2
questions: How far is the performance of the model trained using the simulated data from that of the model trained using real collected data?
soundness: 3
strengths: 1. The paper identifies the challenges in SR, particularly the gap between the synthetic degradations used in training data and the degradations in real-world images. By positioning the problem within the existing literature, the authors provide a strong motivation for their approach.
2. The core idea of the paper lies in decoupling content and degradation in real-world images to generate realistic data for SR, which is a novel approach for this task. The proposed approach, RealDGen, provides a scalable and adaptive solution for generating large-scale, realistic datasets. 
3. The authors conduct comprehensive evaluation of RealDGen across multiple SR models on different datasets. The results show consistent improvements in both PSNR-oriented and perceptual-oriented SR models, demonstrates the robustness of the proposed approach across a variety of settings.
summary: The paper presents a novel approach for generating realistic data for real-world image super-resolution (SR) tasks, addressing the limitations in the generalization of SR models due to the divergence between training data and real-world degradations. The authors propose a new framework called Realistic Decoupled Data Generator (RealDGen), which leverages unsupervised learning to decouple content and degradation from real low-resolution (LR) and high-resolution (HR) images. This decoupling is integrated into a diffusion model, allowing the generation of paired training data that more accurately reflects real-world conditions. The paper claims that RealDGen significantly improves the generalization ability of SR models across various backbone architectures.
weaknesses: 1. The paper does not clearly define Real-world LR. Through experiments, we can see that the work is more focused on the SR problem in real-world photography, but the authors do not clearly define or explain it in the paper. This makes Figure 2 (b) somewhat difficult to understand. 
2. The overall presentation of this paper shoude be improved. The symbols in the formula do not correspond well to the figures. Eq.5 is not reflected in Figure 2. The colors used in Figure 1 (b) is somewhat confusion. The representation of 'G', 'S' in Figure 1 (a) should be 'N', 'K'. X_hr in Eq.5 is not explained. Line 179, 'for an HR image'.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: What is the technical contribution of the proposed method RealDGen?
soundness: 2
strengths: 1. The proposed RealDGen framework effectively separates content and degradation features using a diffusion model, facilitating the generation of realistic low-resolution images that more accurately replicate real-world degradation patterns.
2. By utilizing a well-designed contrastive learning approach for degradation extraction and a reconstruction-based method for content extraction, the proposed RealDGen framework effectively decouples degradation and content features, thereby enhancing data realism and adaptability across different models.
3. Extensive experiments on multiple real-world SR benchmarks show that RealDGen consistently enhances the performance of various SR models, highlighting its practical effectiveness for real-world applications.
summary: The authors present a new framework, Realistic Decoupled Data Generator (RealDGen), to generate realistic, large-scale data for real-world image super-resolution. The framework includes a contrastive learning-based degradation extractor and a reconstruction-based content extractor, enabling the effective capture of authentic degradation patterns and content features from real-world data. Utilizing the pre-trained extractors, the authors decouple the degradation and content features of given real low-resolution (LR) images, which are subsequently used as conditioning inputs for a Decoupled Diffusion Probabilistic Model (DDPM) to reconstruct the given images. Finally, by leveraging unpaired LR and HR datasets, the trained DDPM is able to generate realistic LR images, which exhibit degradation patterns closely resembling those found in real-world data. Extensive experiments across various SR models demonstrate that RealDGen consistently enhances SR performance on multiple real-world benchmarks. Detailed comments are listed below.
weaknesses: 1. What is the technical contribution of the proposed method RealDGen? It seems that the effectiveness of the proposed methods mainly comes from the powerful diffusion model, and a similar idea of separating the degradation and content features has been investigated by previous methods [A].
2. During the training phase of DDPM, the authors finetune partial parameters of the extractor. However, it is unclear the motivation and the effect of finetuning partial parameters. Please provide more discussions.
3. The proposed method is tested on several SR benchmarks, but more experiments on diverse data with various types of degradation (e.g., motion or defocus blur [B, C]) would better showcase the framework’s generalization ability and robustness in a wider range of real-world scenarios.
4. The proposed RealDGen relies on a large real LR dataset, which may be a potential limitation for scenarios where collecting such data is difficult or infeasible. It would be useful to investigate the effect of different amounts of real LR data on data generation.
5. The authors use a contrastive learning approach for the degradation extractor but do not provide enough detail on how different contrastive learning configurations (e.g., different negative sampling strategies) could affect the performance of data generation. It would be better to conduct more additional ablations to investigate the effect of different contrastive learning methods, such as [D].
6. Since the proposed method requires training a DDPM from scratch, the data generation cost is substantially high. Please provide a cost comparison with other existing methods to facilitate a clearer understanding of the computational demands and efficiency of the proposed approach.

[A] Learning Many-to-Many Mapping for Unpaired Real-World Image Super-resolution and Downscaling, TPAMI 2024.  
[B] Benchmarking Neural Network Robustness to Common Corruptions and Perturbations, ICLR 2019.  
[C] Efficient Test-Time Adaptation for Super-Resolution with Second-Order Degradation and Reconstruction, NeurIPS 2023.  
[D] Unsupervised Degradation Representation Learning for Blind Super-Resolution, CVPR 2021.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1.	The proposed method seems to share several ideas with Syndiff (Yang et al.). Could you please clarify the key contributions and differences between the two approaches?
2.	To train the E_deg and E_cont modules, the paper utilizes paired synthetic images generated by RealESRGAN. However, real-world low-resolution (LR) images can vary significantly from these synthetic images, which may hinder the accurate separation of degradation and content, potentially leading to suboptimal super-resolution (SR) performance on external datasets, as observed in Table 6. Could you clarify whether the proposed method is limited by the degradation distribution of RealESRGAN?
3.	Additionally, E_deg and E_cont are fine-tuned during the training of ReadDGen. As a result, real-world LR images presented at test time could bypass these modules and be directly passed through to the output. It would be useful for the authors to explain how these LR images differ from the ground truth (GT) LR images, and to demonstrate that the proposed generator can synthesize diverse LR samples. Moreover, more results before and after fine-tunign would be also beneficial.
4.	What would happen if a single, non-pretrained encoder were used instead of the E_deg and E_cont modules? Could you discuss the impact of this change to see whether separation of content and degradation is necessary?
5.	The SR results presented in the tables are somewhat lower than expected. Were these SR networks trained for 4x super-resolution (SR)? Please specify the scale factor. Additionally, it would be helpful to include more SR results using bicubic LR images for the baseline, as well as SR results trained on real-world SR datasets as oracles.
6.	While the adaptability of the proposed method is emphasized in the paper, the evidence provided to support this claim seems insufficient. It would be beneficial for the authors to include additional analysis and results to further substantiate this claim.
soundness: 3
strengths: This work introduces a novel generative model that eliminates the need for paired high-resolution and low-resolution images captured in real-world scenarios. Instead, by utilizing a synthetically generated training dataset from RealESRGAN, the proposed model effectively trains two key modules, E_deg and E_cont, which are designed to separate degradation and content in the given low-resolution (LR) images. This allows the model to generate realistic LR images using only a single LR input image, without the need for a corresponding high-resolution ground truth, setting it apart from conventional approaches. Given that collecting a large volume of real-world paired datasets is both time-consuming and costly, the proposed method, which bypasses the use of such paired real-world datasets, offers a significant advantage over traditional methods. Moreover, experimental results quantitatively and qualitatively demonstrate the superior performance of the generative model, further validating its efficacy compared to traditional approaches.
summary: In this approach, a generative model is introduced to synthesize realistic low-resolution images, which are subsequently used as a training dataset to improve the performance of conventional super-resolution networks. In particular, unlike traditional methods, the proposed generative model is trained in an unsupervised manner using an unpaired training dataset, thus eliminating the dependency on high-quality ground truth images during the inference phase-a novel aspect of this approach. Furthermore, experimental results show that the generated low-resolution images exhibit improved accuracy compared to those produced by conventional methods, leading to superior performance in downstream tasks such as super-resolution.
weaknesses: Although the proposed method removes the reliance on paired real-world datasets for both the training and inference phases, it still has limitations in fully demonstrating its performance. Specifically, the work lacks comprehensive experiments and detailed analysis. For further clarification, please refer to the questions outlined below.

Minor point: 
In line 87, the method by Park et al. is incorrectly described as a GAN-based approach. It actually utilizes normalizing flows for generation, and this should be revised accordingly.

Meta-review: This paper proposes a contrastive learning method to generate data for realistic image super-resolution in an unpaired setting. The image content and the degradation are decoupled to generate realistic and diverse LR images at scale.  The decoupling is guided by the learning objective, and demonstrated by ablation study. 

The reviewers find the idea to be novel and the experimental results to be impressive. 

The authors are advised to add the analysis provided in the rebuttal in the manuscript to reflect the feedback from the reviewers for better exposition.

Decision: Accept (Poster)