DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation
Authors: Yuang Ai
Xiaoqiang Zhou
Huaibo Huang
Xiaotian Han
Zhengyu Chen
Quanzeng You
Hongxia Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: Image restoration
Dataset Curation
Diffusion transformer
Primary area: machine_vision
TL;DR: A high-capacity image restoration model trained on large-scale, privacy-safe high-quality image dataset

Abstract: Image restoration (IR) in real-world scenarios presents significant challenges due to the lack of high-capacity models and comprehensive datasets.
To tackle these issues, we present a dual strategy: GenIR, an innovative data curation pipeline, and DreamClear, a cutting-edge Diffusion Transformer (DiT)-based image restoration model.
**GenIR**, our pioneering contribution, is a dual-prompt learning pipeline that overcomes the limitations of existing datasets, which typically comprise only a few thousand images and thus offer limited generalizability for larger models. 
GenIR streamlines the process into three stages: image-text pair construction, dual-prompt based fine-tuning, and data generation \& filtering. This approach circumvents the laborious data crawling process, ensuring copyright compliance and providing a cost-effective, privacy-safe solution for IR dataset construction. The result is a large-scale dataset of one million high-quality images.
Our second contribution, **DreamClear**, is a DiT-based image restoration model. It utilizes the generative priors of text-to-image (T2I) diffusion models and the robust perceptual capabilities of multi-modal large language models (MLLMs) to achieve photorealistic restoration. To boost the model's adaptability to diverse real-world degradations, we introduce the Mixture of Adaptive Modulator (MoAM). It employs token-wise degradation priors to dynamically integrate various restoration experts, thereby expanding the range of degradations the model can address.
Our exhaustive experiments confirm DreamClear's superior performance, underlining the efficacy of our dual strategy for real-world image restoration. Code and pre-trained models are available at: https://github.com/shallowdream204/DreamClear.

DreamClear: High-Capacity Real-Worke Image Restoration with Privacy-Safe Dataset Curation
Yuang Ai Xiaoqiang Zhou Huaíbo Huang ,,00
Xiaotian Han
Zhengyu Chen Quanzeng You Hongxia Yang*
MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences ByteDance, Inc University Science and Technology of China shallovdream555@gall com, hualbo hww.guur ipac. aa.aacccoo Code and models: https: //gtt/bb.tttb...//. com/
Restoration Real-World Samples
Quality Input
DreamClsär Output
Quality Input
UreamCloar Outbut
Low Quality Input
Output
(b) Image Restoration Degradations
Sovero
Low Quality Input
Real-ESRGAN
SeeSR
SUPIR
DreamClear (Ours)
We present DreamClear, high-capacity image restoration model that delivers photorealistic restora- tion of real-world LQ images, outperforming SOTA diffusion-baase models in handling diverse degradations.
Figure 1:
Abstract
Image restoration (IR) in real-world scenarios presents significant challenges due the lack of high-capacity models and comprehensive datasets. To tackle these issues, we present a dual strategy: GenlR, an innovative data curation pipeline, and DreamClear, cutting-edge Diffusion Transformer (DiT))based image restoration model. GenlR, our pioneering contribution, a dual-prompt learning pipeline that overcomes the limitations of existing datasets, which typically comprise only few thousand images and thus offer limited generalizability for larger models. GenIR streamlines the process into three stages: image-text pair construction, dual-prompt based fine-tuning, and data generation & filtering. This approach circumvents the laborious data crawling process, ensuring copyright compliance and providing a cost-effective, privacy-safe solution for IR dataset construction. The result is a
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

large-scale dataset of one million high-quality images. Our second contribution, DreamClear, a DiT-based image restoration model. It utilizes the generative pr- ors of text-to-image (T2I) diffusion models and the robust perceptual capabilities of multi-modal large language models (MILMM)) to achieve photorealistic restoration. To boost the model's s adaptability to diverse real-world degradations, we introduce the Mixture of Adaptive Modulator (MoAM). It employs token-wise degradation priors to dynamically integrate various restoration experts, thereby expanding the range of degradations the model can address. Our exhaustive experiments confirm DreamClear's 's superior performance, underlining the efficacy of our dual strategy for weal-world image restoration.
Introduction
Image restoration (IR), vital field in computer vision, targets transforming degraded low-quality (LQ) images into high-quality (HQ) counterparts. While IR has achieved significant advancements under predefined conditions, such super-resolution [92, 10] and denoising [12, 40] tasks, real- world IR remains a formidable challenge due to the diversity and complexity of degradation types. The disconnect between training data and real-world scenarios is substantial, as existing datasets inadequately encapsulate the intricacies of real-world degradations. Efforts to bridge this gap include domain adaptation [5. 74, 23, 81], dataset collection [68, 8, 11. 91], and degradation simulation [64, 85, 49, 71j. However, in contrast to the leaps in Natural Language Processing (NLP) [1] and AI- Generated Content (AIGC) [59] enabled by aarge-scale models and extensive data, IR's progress not as pronounced. Real-world challenges persist, and the potential of large-scale data and high-capacity models remains largely untapped. This leads us to two critical questions: how can we obtain large-scale dataset that accurately represents real-world IR, and based on this, how can we construct powerful models tailored for real-world IR scenarios?
Addressing the first question, considerable efforts have been made to curate IR datasets. Given the challenge of collecting real-world paired IR data, these datasets are typically constructed by acquiring HQ images and then simulating degradations to generate corresponding LQ images. While many works [64, 85, 49, 71] have refined the degradation simulation process, this paper focuses on the acquisition of HQ images and the associated challenges of copyright and privacy protection. The predominant method for obtaining HQ images is web scraping. Current open-source IR datasets, such as DIV2K [44] and Flickr2K [2], contain only few thousand images, insufficient for covering a broad spectrum of real-worke scenarios. Larger collections like SUPIR [80], with 20 million images, highlight the labor-intensio
nature of large-scale dataset curation. Moreover, images sourced from the internet often involve copyright issues and privacy concerns, particularly with identifiable human faces. To advance the IR field effectively, there is an urgent need for a dataset curation method that is privacy-safe and cost-effective.
In response, we present an under-explored approach in the image restoration (IR) field: creating high-quality, non-existent images enhance dataset curation efficiency, while evading copyright and privacy issues. We unveil GenlR, repurposes the generative prior pretrained text-to-image (T2I) models for IR tasks, and uses multimodal large language models (MLLMs) to generate text prompts, thereby improving data synthesis quality. GenlR operates three stages: (1) image-text pairs construction, (2) dual-promppt fine-tuning, and (3) data generation & filtering Initially, GenlR utilizes existing IR datasets and the advanced MILM,, demini-1.5-Prr [62], to create image-text pairs, while generating negative samples via an image-to-image pipeline [50]. Subsequently, we apply a dual-prompt learning strategy to adapt pretrained T2I models to the IR task, generating suitable prompts for data synthesis. In the final stage, MLLMs create various scene descriptions and synthesize images using the adapted image prior, with a focus on ensuring no identifiable individuals are included. MLLMs also assess and filter the synthesized data based on quality, producing high-quality images that are privacy-safe and copyright-free, Through GenIR, we generate dataset of one million high-quality images, proving its efficacy in training robust real-world IR model.
automated data curation pipeline that
Armed with a large-scale, high-quality image dataset, our focus shifts to the construction of a high-capacity IR model that can robustly generalize to real-world scenarios. Recent sfate-of-the- art approaches [77, 70, 80] employ the generative priors in pretrained Stable Diffusion [59] (SD) for realistic image restoration, underlining the power of rich generative prior in SD. As Fig.1
2

---

illustrates, SD-based methods outperform GAN-based ones. However, these strategies often neglect the degradation priors in input low-quality images, a critical element in blind IR [72]. This insight leads us to investigate the integration of degradation prior into diffusion-basee IR models, and how to optimize its synergy with large models.
In this paper, we introduce DreamClear, a high-capacity real-world image restoration model, grounded on a large dataset. DreamClear based on Diffusion Transformer (DiT) [53], the cor- nerstone of modern diffusion-based systems (e.g., Sora [7], Stable Diffusion [18]). Our model employs a dual-branch framework with textual guidance from multi-modal large language models (MLLMs) for photorealistic restoration. DreamClear first processes the low-quality image through lightweight network to produce reference image. We propose ControlFormer to enhance the control over DiT-based T2I models, thereby better utilizing the low-quality and reference images to guide the content of the generated image. To further improve the model's generalization across diverse and complex degradations, we incorporate implicit prior degradation information to refine the solution space. Specifically, we suggest a Mixture of Adaptive Modulator (MoAM), which extracts token- wise degradation representations and dynamically integrates various restoration experts for each token based on the Mixture-of-Expert (MoE) [61] structure, thereby enhancing the model's adaptability to different degradation severities (See Fig. 1).
The main contributions of this work can be summarized as follows:
We propose GenlR, a pioneering automated data curation pipeline for image restoration. It addresses the urgent need for privacy-safe and cost-effective methods in image restoration, yielding a dataset of one million high-quality images.
We present DreamClear, a robust, high-capacity IR model that incorporates degradation pri- ors into diffusion-based frameworks. This model improves control over content generation, adapts to various degradations, and generalizes well across diverse real-world scenarios.
Extensive experiments across both low-level (synthetic & real-world) and high-level (de- tection & segmentation) benchmarks have demonstrated DreamClear's state-of-ther of-the-art performance handling intricate real-world scenarios.
2
Related Work
Image Restoration. Image Restoration aims restoring a high-quality image from the low-quality input image. Over the past decade, different approaches have been proposed for image super- resolution [95, 31, 65, 96], denoising [87, 84. 67], deblurring [57, 69, 56], deraining [14, 34, 32, 33], inpainting [97, 31. etc. Recently, researchers have increasingly focused on enhancing the generaliza- tion ability to diverse degradations in real-world applications [74, 68, 63]. The degradation simulation is improved from simple degradations to complex degradation processes, such as BSRGAN [85], Real-ESRGAN [64] and AnimeSR [71]. With the improved degradation simulation process, many recent methods can deal with diverse degradation types and achieve promising performance in real- world scenarios [43, 8]. With the paired data, training a randomly initialized restoration model from scratch is one way to improve generalization ability [41]. The other way is to exploit the generative prior in the pre-trained generative model, such as GAN or diffusion models [52, 63, 4, 37, 66, 9]. In this work, we propose data synthesis pipeline and introduce real-world image restoration model with high generalizability.
Generative Prior. Generative models learn the image synthesis process and embed the image prior in the model weights. The image prior high-quality image generator, such as StyleGAN [36] and Stable Diffusion [59], can be adapted to other visual restoration tasks [43, 8, 38, 20J. To use the image prior in GANs, an additional encoder is often applied to convert the input image to the latent space [76, 48]. For the diffusion models, the forward process adds noise to the image gradually and finally converts the image to the latent noise space [15, 50]. By manipulating feature space, the input image is integrated into the generation process as a conditional input, and the synthesis process exploits the image prior in the pre-trained models. The generative prior the pre-trained models can also serve as good initialization for downstream synthesis tasks [63, 78, 35). We exploit the generative prior the pre-trained diffusion models to synthesize datasets for image restoration tasks and train restoration model for real-world applications.
the latent
3

---

Longuage instruction
Gemini MILM
Panlive Promel Leanning
Cuiddors.... lowwr
Longuage Instruction
Gemini UIM
"outdeers, sunsny italian style, wrben,, Text Prompts
Text Encoder
Diverse Prompt
"outddors, WBRVES,
image to amage
Postiiv Samples
Attention Bicck
Fendbroi Bl0ck
Fonttread
2. "Lnaloors, ... coffie, table."
singeg
"cartsons deformaation. qualiity over Smooth""
Bioo
Swddewatter willllife"
Pre tratned 721 Diffuion Model
| |w/PorBBley Pronce
Text Encoder
Oiffuston
bertzon, Low qualitt dirty imagee" Tort Prompts
CRS Sampling
Negarive Prompt Loorning
Troistbic Modile Frowe Madiv Offiusion
Quality Classifier Gemini
Negosive Somples
Altoriig
(a) Image Text Pairs Construction
Dual-Prompt Bared Fine- Tuning
Dota Generation & Filtering
Figure 2: An overview of the three-stage GenIR pipeline, which includes (a) Image-Tesh Text Pairs Construction, (b) Dual-Prompt Based Fine-Tuning, and (c) Data Generation & Filtering.
Synthetic Dataset. Data size and data quality widely recognized essential for many vision tasks. A large-scale high-quality dataset can facilitate the large model training and improve the model ability greatly [83, 22. 21, 24, 26]. Existing large-scale datasets often manually collected with laborious human efforts [16, 39]. More importantly, the data crawled from the internet may leak privacy information [58, 24], raising concerns related Al security. The synthesized datasets only reduce the laborious human efforts, but also avoid the privacy information leakage. High-quality synthesized datasets are verified be effective in many vision tasks [28, 25, 6]. Our work the first to explore the dataset synthesis in the image restoration field.
3 Privacy-Safe Dataset Curation Pipeline
Traditionally, IR datasets are created by scraping web images and simulating degradations to generate low-quality (LQ) counterparts. This process is labor-intensive and rife with copyright and privacy issues, especially with identifiable human faces. To advance the IR field, privacy-saff and cost- effective dataset curation method is needed. Drawing inspiration from the success of text-to-image (T2I) models in synthesizing high-quality images, we introduce the GenlR pipeline. This novel approach leverages the generative priors of pre-trained T21 models construct extensive, privacy- safe datasets. However, the efficacy of T21 models is contingent upon carefully crafted prompts for generating high-quality images fitting for IR tasks.
To tackle this, GenlR, as illustrated in Fig. 2, employs streamlined three-stage process. Initially, we construct positive and negative samples, each paired with corresponding text prompts. Subsequently, a dual-prompt based finetuning strategy concurrently learns both positive and negative prompts. Finally, we utilize LLMs to generate a diverse array of text prompts, leading to the creation and filtering of data. Throughout this process, we maintain stringent privacy standards, ensuring no specific personal information is embedded in the text prompts or the generated images.
Textt-TText Pairs Construction. We high-resolution, texture-rich images in existing IR datasets [44, 2, 22, 39] as positive samples. Given the unavailability of corresponding text prompts, we employ the sophisticated MILM, Gemini-1.5-Pro [62], to generate necessary prompts via language instructions. Moreover, to identify and eliminate unwanted content during the T2I model's 's fine-tuning and enhance image quality, generate negative samples representing undesirable outcomes using the T21 model. As depicted in Fig. 2 (a), we adopt the image-to-image pipeline proposed in [50], using the T2I model and manually designed prompts such "cartoon, painting, ... over-smooth, dirty", to directly generate negative samples.
Dual-Prome Prompt based Fine-Tuning. Rather than relying on complex, labor-intensive prompts with limited applicability, we propose an innovative dual-prompt based fine-tuning approach to refine the T21 model for our data needs. As illustrated in Fig. (b), we employ positive and negative samples to learn their corresponding prompts. Specifically, use M positive tokens {(p1). ... (psr) and N

---

negative tokens {(m1). ..... (nN)} to represent desired and undesired attributes, respectively, and subsequently learn their embeddings. We initialize these new positive and negative tokens using frequently used positive (e.g.. "4k, highly detailed, professional... ...") and negative text prompts (e.g.. "deformation, low quality, over-smooth ..."). As the text condition is integrated into the diffusion model via cross-attention, we also refine the attention block to better comprehend these new tokens.
After fine-tuning the T2I model with our curated image-text pairs, we can efficiently employ the learned prompts and refined diffusion model to readily generate the needed data.
Data Generation & Filtering. In addition to the quality of images, the diversity of scenes within the IR dataset is of paramount importance. To address this, leverage Gemini to generate one million text prompts, describing varied scenes under carefully curated language instructions. These instructions explicitly proscribe the inclusion of personal or sensitive information, thereby ensuring privacy. As depicted in Fig. 2 (c), we employ the fine-tuned T2I model in conjunction with the learned positive and negative prompts to generate HQ images.
Classifier-free guidance (CFG) [30] provides a mechanism to effectively utilize negative prompts, thereby mitigating the generation of undesired content. During the sampling phase, the denoising model Eo anticipates two outcomes: one associated with the positive prompt pos and the other with the negative prompt neg. The final CFG prediction is formulated as
eo (2,, pos, neg) = w x (Z t, pos) + (1 w) x Eœ(Ze, f, neg),
where w denotes the CFG guidance scale. Post-sampling, the generated images are evaluated by a quality classifier, which decides whether to retain the images based on the predicted probabilities. This binary classifier is trained on positive and negative samples. Gemini is subsequently used to ascertain whether the images exhibit blatant semantic errors or inappropriate content.
Contrasted with direct web crawling, our GenlR provides more cost-effective and approach to data acquisition. It circumvents the potential infringement of personal privacy information prevalent on the web, thereby ensuring our research both ethical and secure a crucial aspect the current artificial intelligence landscape characterized by extensive data usage. Ultimately, we gather one million high-resolution (2040 x 1356) images, each of superior quality.
High-Capacity Image Restoration Model
The complex and varied degradation of applicability of restoration models. We introduce DreamClear, a high-capacity image restoration model that dynamically integrates various restoration experts, guided by prior degradation information. DreamClear is built upon on PixArt-a [13], a pre-trained T21 diffusion model based on the Diffusion Transformer (DiT) [53] architecture, which has proven its powerful generative capabilities [93].
images presents major challenge to the practical
Architecture Overview. Fig. 3 illustrates that DreamClear features a dual-branch architecture, encompassing an LQ Branch and Reference Branch. LQ images J,, are processed by SwinIR [41], a lightweight degradation remover, resulting in smoother, albeit less detailed, reference images Irej- Considering potential detail loss in Iref, f. we employ both Itg and Lre/ direct the diffusion model. Moreover, we utilize the open-source MLLM, LLaVA [47], to generate detailed captions for training images using the prompt "Describe this image and its style in a very detailed manner". supporting the T21 diffusion model in attaining more realistic restoration.
CortrolFormer. ControlNet [88], a prevalent structure for managing diffusion models, tailored for the U-Net [60] structure in SD. It is unsuitable for DiT. stemming from the architecture difference. To address this, we present CortrolFormer, which inherits ControlNet's core features (trainable copy blocks and zero-initialized layers) but is adapted for the DiT-based T21 model, as shown in Fig. 3. ControlFormer, duplicating all DiT Blocks from Pix-rt-o, employs the MoAM block to combine LQ features Zi, and reference features f- This DiT-optimized ControlFormer maintains ControlNet's essential components, providing effective spatial control within DIT.
Mixture-of -ixture-ofAAdappive--Mood Adaptive-Modulatoo To enhance our model's robustness to real-world degradations, (MoAM) for effective IQQ and
we propose degradation vixture-of-Adaapiie---nn reference feature fusion. As shown in Fig. 3, MoAM consists of adaptive modulators (AM),
5

---

Overall Architecture of DreamClear
ControlFormer Suf Pateh- Rmbeddinng
VAE Enceder
ControFFarmer
Detniled Test Prompe
Text
DiT Buckbent
The image depectr a scrone landscope feotuinng a small salngg aeched nissae The wilage consets severol nceses with varying archtectura stylis..
DIT Black 2M
VAE Deceder
Highteeiight Begratiation
MeAM
Zz-1
DIT Block (tranable copy)
Sive
Reference Branch
DIT Black 2 (traisable
Branch
BQ
Coge
Firey
Adaptative (AM)
Adaptiie Modulator (MoAM)) Block
Micter EtEurrs
Inpet
Expert
DIT Bleck (trainable cops)
Cond
Linar
Liscar
Cresi-Atta
10
Zaro Litcar
AM
Expert AM
Zero Litear
Xi
Expert
Figure 3: Architecture of the proposed DreamClear. DreamClear adopts dual-branch structure, using Mixture of Adaptive Modulator to merge LQ features and Reference features. We utilize MLLM to generate detailed text prompt as the guidance for T2I model.
cross-attention layer, and a router block. AM employs AdaLN [54] to learn dimension-wei -wise scale ? and shift parameters, embedding conditional information into input features.
MoAM operates three steps: 1) For DiT features Zins we calculate the cross-attention output Tattn E IR NxC between B2g E IR NxC and Fref NxC where N and C denote the number of
visual tokens and hidden size. Fin is then modulated using Pattn followed by a zero linear layer.
A token-wise degradation map D E NC is generated through the linear mapping of Tattn 2) Features are further modulated using AM, with Tyef as the AM condition to extract clean features. 3) We adopt a mixture of degradation-awan experts to adapt to diverse degradations, detailed below.
Given varying degradations real-world images, our method dynamically processes tokens using degradation priors. Each MoAM block consists of K restoration experts (i.e., AM) (...... Ex each specialized for specific degradation scenarios. routing network R(-) dynamically merges expert guidance for tokens, based on D. The routing network, two-layer MILP followed by softmax, yields token-wer wise expert weights R(D) R The dynamic mixture restoration experts is formulated as
K
7(0)
Σ
k) x [et
8(i) = Σ w(i, k) Net
(2)
Tou (1 (1+ am. + 8
(3)
where and & index tokens and experts respectively, Net and Net map within an expert to 7 and B, and @ denotes element-wise multiplication. MoAM dynamically fuses expert knowledge, leveraging degradation priors to tackle complex degradations.
Experiments
5.1 Experimental Setup
We adopt a combination of DIV2K [44], Flickr2K [2], LSDIR [39], DIV8K [22], and our generated dataset to train DreamClear. We employ the Real-ESRGAN [64] degradation pipeline to generate LQ images, using the same degradation settings as SeeSR [70] to ensure fair comparison. All experiments conducted on scaling factor x4.
Datasets.
For testing datasets, following previous works [63, 80, 70], we evaluate our method on both synthetic and real-world benchmarks. For synthetic benchmarks, randomly crop 3,000 patches from the

---

Table 1: Quantitative comparison with stale-of-the-art real-world IR methods on both synthetic and real-world benchmarks. Best and second best performance are highlighted in red and blue, respectively.
Tatssts
Matnics
BSRCIAN (85)
Real- [64)
ISRGAN
GAN [43]
SwislR-
DASR (+1)
Suni-SR (63)
DiiRIR (06]
ReeShin [62]
SinSR (65)
Se:5R [70]
SUPIR (40)
DreamClea
PSNK 7
19.58
19.92
19.66
19:73
19:73
19:M5
15.80
19:37
19.59
13.05
15:09
S5IM
0,5137
0.5334
0,5353
0,3122
0.5039
0.4957
0.4985
0.4513
0.5045
0.4664
0.4766
LMPS
0.493%
0.9881
0.989
0.450
0.445
015356
0.4450
014363
0L3862
0.3976
0.1657
DISTS
0,2134
0,2253
0,2606
0,2162
012316
0.2383
0.1580
0.1637
DIV3X-16
RTD
49.17
36.62
29.64
57.00
NIUE
X748
MANIQA
0.3514
0.354
0.110
0.2422
0.426
0,3782
0.5251
0.491
044200
MUSIO
90.66
58.90
64.77
62.67
72/04
CLIPROA
0,5589
0.5004
015882
0.5565
9.590
01027
0.695
11066
0.7181
0.6035
0,6663
PSNK
18.27
18.13
17.9
18.15
18.11
1843
18.24
17.44
18003
16.95
17.01
SSIM
0.4673
0.4867
0.4783
0.4679
04506
0.4618
0.4579
0,4302
0.454
0.4000
0.4266
LPPS
04577
0.300
01000)
044505
0.4159
(1 01:009
0.4534
04523
0.3759
B:1119
0.9616
DISTS
0,2539
0.2278
0.2253
0.3615
0,2159
0.3499
0,2435
D2236
0.1966
0.1838
0.1656
RED
53.25
46.46
4531
60,60
31.36
35.01
43.25
36.1
25.11
30.03
22,06
NIUE
3,885
3.0779
33715
36432
4.011
4.3750
54240
MANIQA
0.3991
0.3315
0.3098
0.4551
0,3995
MUSIO
59.57
CLIPIOA
0.5648
1AG218
0,993
0.5081
0.5990 1
0.0592
0.6501
(UGANN
0.7261
0.6174
9,6114
PSNR
24.21
34
25:51
34.00
34.77
24,04
26.47
24
32.67
21.56
SSIM
0.7422
002001
0.7543
6.7526
0.7357
0,2175
1LOT30
U16067
0.0046
LPIPS
0.1855
0.2501
0.3680
0.3201
0.2736
0.3436
0.2864
0.4206
022597
0.3665
0.3654
DASTS
0.1957
1899
IL1334
0:3055
8.1761
112194
0.2467
02453
B 03300
0.1112
ResiRR
FED
84.49
90.10
50.17
91.16
85.89
69.94
59.91
7033
71.91
71.65
65.37
NIQE
4.9261
5,0099
4.9455
5:5639
5.6134
6.6044
6:4622
29102
45366
4,4361
MANIUA
033050
0.2819
0.366
1.000N
(L5189
6.4337
MUSIO
50.97
50.94
61.07
61.74
69.35
CLIPIOA
0.4548
0.3819
0.1339
0,6202
SSIM
0.7812
0.7155
0.5051
0.78M
0.7100
0.7388
0.7566
0.0008
LPIPS
0.278)
0.3165
6.2710
0,9920
04400
0,4500
03117
0.3344
0.3073
DASTS
0.1956
081857
IL1787
0.2072
0.17557
0.4443
0,2583
112500
02101
B.264
0.3145
DiaiSR
FTD 1
84.26
$3.79
$4.22
94.96
90.23
76.89
91.32
55.25
75,07
90.75
74.78
NIQE
5,5866
5,7422
S.5140
6.0663
6.1699
6 54533
7.5615
6,8770
5 576966
5.1115
4.6265
MANIJA 7
0,3420
0.4423
0.326
0.3754
0.3171
113501
0,3350
(13510)
0.8974
0.1174
0,1676
MUSIO
61.22
58.37
57.33
46.49
$6.43
55.14
55.27
58.50
67.42
64.53
59.63
CLJPQQA
0,5585
0.4347
0.4319
0,1878
0.5344
0.605
0.5788
0.6734
0L2822
0.5800
0.6620
NIQE
4.5229
1091
4.0912
4.3866
4.6399
4.8000
59727
5.718
4:41
3633
3,5556
RvalE2250
MANIQA
MUSIO
63.63
0,2782
$3.30
0,2939
$7.11
62.15
CLIPIQA
0,533
0:4071
0.520
016420
0.7104
Table 2: Evaluation on COCO val2017 (object detection & instance segmentation) and ADE20K (semantic segmentation).
Metries
Z.omed
BSRGAN
Real- ESRGAN
SwinlR- GAN
DASR
StilleRR
DATBER
ResShift
SinSR
Se:5R
SUPIR
DreamsClear
Object Detationn (AP*)
49.0
7.4
11.0
13.8
11.8
10.5
16.9
18.7
15.6
13.8
18.2
16.5
19.1
Object Detection (AP
70.6
17.6
20.7
18.9
17.0
26.7
29.9
25.0
22.3
29.1
27.2
30.8
Object Detection (AP%)
53.8
12.1
10.7
17.6
19.4
14.2
18.9
17.0
19.8
Instance Segmenction LAP")
439
6.4
9.5
11.3
10.2
9.3
14.6
16.2
1366
12.0
15.9
14.1
16.7
Instanee Segmenatiinn (APg)
67.7
11.2
16.4
19.3
17.5
15.9
24.6
27.5
2333
20.6
26.6
24.5
25.2
Inttacee Segmestation (A7%)
10.2
9.4
14.0
16.8
Semantic Segmentation (mloU)
14.3
30.4
23.6
19.6
26,9
27.7
31.9
validation sets of DIV2K and LSDIR, and degrade them using the same settings as training. We name these two benchmarks as DIV2K- Val and LSDIR- Val respectively. For real-world benchmarks, we conduct experiments on commonly used RealSR [8] and DRealSR [68] datasets. Besides, we establish another real-world benchmark, called RealLQ250, which includes total of 250 LQ images of size 256 x 256 used in previous works [70, 42, 64, 86, 80] or sourced from the Internet, without corresponding GT images. For all testing datasets with GT images, the resolution of the HQ-LQ image pairs is 1024 x 1024 and 256 x 256, respectively.
Metrics. Following SeeSR [70]. we adopt PSNR and SSIM (calculated on the Y channel of transformed YCbCr space) as reference-basee distortion metries, LPIPS [90] and DISTS [17] as reference-based perceptual metries, NIQE [89], MANIQA [75], MUSIQ [75] and CLIPIQA [75] as no-reference metrics. FID [29] is used to evaluate the image quality. These metries can achieve a comprehensive evaluation of the restoration effects.
Implementation Details For training GenlR and DreamClear, we both use the original latent diffusion loss [59]. The proposed GenlR framework is built on SDXL [55] and trained over 5 days using 16 NVIDIA A100 GPUs. The training is conducted 1024 x 1024 resolution images with
7

---

LQ Ingur
Real ESRGAN [64]
DINBIR [46]
SinSK [65]
SeeSK [79]
SUPIR [80]
DreamCleur (Ous)
Figure Qualitative comparisons on both synthetic (the first row) and real-world (the last two rows) benchmarks.
Please zoom in for better view.
a batch-size of 256. For data generation, we use 256 NVIDIA V100 GPUs and spend 5 days to generate the large-scale dataset. Our DreamClear is built upon PixArt-a [13] and LLaVA [47]. The SwinIR model in DiffBIR [46] is used as the lightweight degradation remover. We use the AdamW optimizer with learning rate of 5e to train our model. The training is conducted on 1024 x 1024 resolution images, running for 7 days on 32 NVIDIA A100 GPUs with batch-size of 128. The number of experts K in Eq. (2) is set 3. For inference of DreamClear, we adopt iDDPM [51] with 50 sampling steps, CFG guidance scale w = 4.5.
(8) Vote Percentage
(b) Top-K Ratio
190%
97%
DreamClear (Curs)
72%
SUPIR
12
DiEBIR
17%
I SinSR
StahkeRR
125
0%
Top-1
Top
Tops
Tag-4
Tay-5
Figure 5: User study. Vote percentage denotes average user preference per model. The Top-K ratio indicates the proportion images preferred by top users. Our model is highly preferred, with most images being rated as top quality by the majority.
5.2 Comparison with State-of-the-Art Methods
We compare our method with state-of-the-art GAN-based methods (BSRGAN [85], Real- ESRGAN [64], SwinIR-GAN [41], and DASR [43]) and recent diffusion-hased methods (Sta-
bleSR [63], DifIBIR [46], ResShift [82], SinSR [65], SeeSR [70], and SUPIR [80]..
Quantitative Comparisons. Tab. presents quantitative results on various benchmarks. Our method consistently excels in perceptual metrics (LPIPS, DISTS, FID) on synthetic datasets, sig- nifying high perceptual quality. On real-world benchmarks, our method performs strongly across most no-reference metrics (NIQE, MANIQA, MUSIQ. CLIPIQA), attesting to the high quality of our restorations. Our diffusion- based method prioritizes photorealistic restoration. Despite lower PSNR/SSIM scores, recent works [79, 80] argue these metrics inadequately represent visual quality, and it necessary to reconsider the reference values of existing metrics and propose more effective methods to evaluate modern image restoration methods. We believe that as the field of image quality assessment (IQA) evolves, more suitable metrics will emerge to adequately measure the performance of advanced IR models.

---

6.44
Nambar Traniig Images
257
Number Truin impg
Nutibor Traning tmaga
(a)
(b)
Cevernta mape
DI5 mayad
Number Troinng mages
Kumrer rean Im1get
Sunterr Troknes mage
(d)
Figure 6: Impact of synthetic training data. As data size increases, performance improves LSDIR-Va Vol.
Table 3: Ablation results D/V2K-Val Val, COCO val2017 and ADE20K for DreamClear.
LPIPS
DISTS
FDD 4
MANIQA
MUSIQ f
CLIPIQA
A Ap
Ap
mloUi
Mixture of
0.3657
0.1637
20.61
0.4320
68.44
0.6963
16.7
31,9
AM
0,3981
0.1843
25.75
0.4067
66.18
0.6646
18.0
15.6
28.6
Cross-Attention
0.4177
0,2016
29.74
0.3785
63.21
0.6497
17.2
15.1
26.3
Zero-Linear
0,4082
0,1976
0,4122
66.11
0.6673
27.2
Dual Branch
0.3657
0.1637
20.61
0.4320
68.44
0.6963
19.3
16.7
31.9
Reference Branch
0.4207
0.2033
30.91
0.3985
64.04
0.6582
15.9
14.0
24.7
Detailed Text Prompt
0.3657
0,1637
20.61
0.4320
68.44
0.6963
19.3
16.7
31,9
Null Prompt
0,3521
0.1607
20.47
0.4230
67.26
0.6812
18.8
16.2
29.8
Qualitative Comparisons. We provide qualitative comparisons in Fig. 4. When handling severe degradations (the first row), only our DreamClear not only reason the correct structure but also generate clear details, while other methods may generate deformed structure and blurry results. When comes real-world images, method achieve results that rich in detail and more natural (the third row). More real-world visual comparisons are Apendix A.4.
study to evaluate model's restoration quality using 100 low-quality images, restored by our method and five others. Users were asked to rank the restorations considering visual quality, naturalness, and accuracy, among others. The study, involving 256 evaluators, was designed for fairness and wide participation. Two metrics, vote percentage and Top-K ratio, were used to analyze the results. As shown in Fig. 5, model led both metrics, receiving over 45% of total votes and being the top choice for 80% of the images, demonstrating its consistent superiority in producing high-quality images. Refer to Appendix A.2 for more details.
User Study. We conducted
5.3 Evaluation on Downstream Benchmarks
assess benefits of image restoration for downstream high-level vision tasks by conducting detection and segmentation experiments on the COCO 2017 [45] and ADE20K [94] datasets using various restoration models. Low-quality images are generated and restored under the same conditions in training. We use the robust visual backbone RMT [19] (with Mask R-CNN [27] 1x for COCO, with UperNet [73] for ADE20K) for these tasks. Tab. shows that our model obtains the best performance, implying its superiority in benefiting downstream tasks. Despite its superior performance semantic restoration and fine-graiee fne-grained image recognition tasks, there's 's still substantial room for improvement.
5.4 Ablation Study
Analysis Generated Dataset for Real-World IR. Due to the extensive time required train diffusion-based models, SwinIR-GAN to investigate the impact of generated datasets

---

real-world image restoration (IR). SwinIR is trained on varying quantities of generated data for comparison with the Dranetrained model. Fig. shows that the model trained on an equivalent number of generated images exhibits marginally lower perceptual but higher no-reference metrics than the DF2K model. As the dataset size increases, all metrics improve, reinforcing our belief that larger datasets enhance model generalizability and restoration performance. Notably, the model trained with 100,000 generated images outperforms the DF2K model, underscoring the advantages of utilizing large-scale synthetic datasets for real-world IR. More ablations of GenlR are provided in Appendix A.3.
Ablations of DreamClear. We conduct ablation studies to scrutinize the contribution of each component within DreamClear. Evaluating perceptual fidelity via LPIPS, DISTS, and FID metries, and assessing the image quality of restoration results with MANIQ, MUSIQ, and CLIPIQA, we find that DreamClear outperforms its ablated versions across most metrics, substantiating the importance of these components (Tab. 3). Notably, null prompts slightly outperform detailed prompts perceptual metrics. However, superior results on no-reference and high-level vision metrics suggest that the MLLM-providee detailed text prompts better preserve semantic information. Visual comparisons detailed in Appendix A.3 further reinforce the benefits of semantic guidance in image restoration provided by text prompts.
6 Limitations and Broader Impact
Our model leverages the generative prior of pre-trained diffusion models for image restoration, with diverse synthesized datasets used during training to enhance model performance. In situations of severe image degradation, while our method could predict reasonable and realistic results, the synthesized texture details may not exist the ground-truth image. A high-quality reference image or explicit human instruction may compensate such limitation in some degree.
Another limitation lies in the deployment in practical applications. Our model is diffusion- based model and it needs multiple inference steps to restore the input low-quality image. While our model can predict more plausible results than existing methods, it can not meet the requirement of real-time inference speed in many practical applications. Model distillation and model quantization may compensate the limitation of inference speed.
This paper is a purely academic study of real-world image restoration (IR). However, considering image restoration's 's vital role in many practical applications, this work may not only bring some positive societal influence, e.g., improving the quality of images captured by smartphones, but also lead to some potential risks like privacy information leakage from photos on social media. However, the positive societal effects of image restoration far exceed the potential negative impacts, and people may make use of some other techniques, such as inpainting and watermarking, remove the private information in images.
7 Conclusion
To address the challenges in real-world image restoration (IR), we develop GenIR, a privacy-safe automated pipeline that generates large-scale dataset of one million high-quality images, serving as a robust training resource for IR models. Additionally, we introduce DreamClear model that seamlessly integrates degradation priors into diffusion- based IR models. It introduces the novel Mixture of Adaptive Modulator (MoAM) to adapt to diverse real-world degradations. Our comprehensive experiments underline its outstanding performance in managing complex real-world situations, marking a substantial progression in IR.
potent IR
Acknowledgements
This research is partially funded by Beijing Nova Program (20230484276), and Youth Innovation Promotion Association CAS (Grant No. 2022132).
10

---

References
[1] J. Achiam, S. Adler S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida,
J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint
XrX1v:2303.08774 2023. 2
[2] E. Agustsson and R. Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In CVPRW. pages 126-135. 2017. 2, 6,6
[] Y. Ai, H. Huang, and R. He. Lora-ir: Taming low-rank experts for efficient all-in-one image
restoration. arXiv preprint wrXi::2410.55382 2024. 3
[4] Y. Ai, H. Huang, X. Zhou, J. Wang. and R. He. Multimodal prompt perceiver: Empower
adaptiveness generalizability and fidelity for all-in-one image restoration. In CVPR, pages
25432-25444. 2024. 3
[5] Y. Ai, X. Zhou, H. Huang. L. Zhang, and R. He. Uncertainty aware freeee-free adaptive image
super-resolution with wavelet augmentation transformer. In CVPR, pages 8142-8155 -152, 2024. [6] S. Azizi, S. Kornblith, C. Saharia, M. Norouzi, and D. J. Fleet. Synthetic data from diffusion
models improves imagenet classification. arXiv preprint arXiv:2304. 08466, 2023. 4
[7] Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, Jing, D. Schnurr, J. Taylor, T. Luhman,
E. Luhman, C. Ng. R. Wang, and A. Ramesh. Video generation models world simulators,
2024.
[8] J. Cai, H. Zeng, H. Yong, Z. Cao, and L. Zhang. Toward real-world single image super- resolution: A new benehmark and new model. In ICCV, pages 3086-3095, 2019. 2, 3.
7
[9] J. Cao, Y. Shi, K. Zhang, Y. Zhang, R. Timofte, and L. Van Gool. Deep equilibrium diffusion restoration with parallel sampling. In CVPR, pages 2824-2834, 2024.
[10] M. Cao. C. Mou, F. Yu, X. Wang. Y. Zheng, J. Zhang, C. Dong. G. Li, Y. Shan, R. Timofte, et al. Ntire 2023 challenge on 360deg omnidirectional image and video super-resolution: Datasets,
methods and results. In CVPRW, pages 1731-1745, 2023. 2
[1]] C. Chen, Z. Xiong, X. Tian, Z.-J. Zha, and F. Wu. Camera lens super-resolution. In CVPR,
pages 1652-1660, 2019. 2
[12] H. Chen, J. Gu, Y. Liu, S. A. Magid, C. Dong, Q. Wang. H. Plister, and L. Zhu, Masked image
training for generalizable deep image denoising. In CVPR, pages 1692-1703, 2023. 2
[13] Chen, Y. Jincheng. G. Chongjian, L.. Yao, E. Xie, Wang, Kwok, P. Luo, H. Lu, and Z. Li.
Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In
ICLR, 2024. 5,
[14] X. Chen, H. Li, M. Li, and J. Pan. Learning a sparse transformer network for effective image deraining. In CVPR, pages 5896-5905, 2023. 3
[15] Choi, S. Kim, Y. Jeong, Y. Gwon, and S. Yoon. Ilvr: Conditioning method for denoising
diffusion probabilistic models. In ICCV, pages 14367-14376, 2021. 3
[16] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical
image database. In CVPR, pages 248-255, 2009.
[17] K. Ding, K. Ma, S. Wang, and E. P. Simoncelli. Image quality assessment: Unifying structure and texture similarity. TPAMI, 44(5):2567-255
2020.
[18] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer,
F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.032200
2024. 3
[19] Q. Fan. H. Huang, M. Chen, H. Liu, and R. He. Rmt: Retentive networks meet vision
transformers. In CVPR, 2024. 9
[20] B. Fei, Z. Lyu, L. Pan. J. Zhang, W. Yang, T. Luo, B. Zhang, and B. Dai. Generative diffusion prior for unified image restoration and enhancement. In CVPR, pages 9935-9946, 2023. 3
[21] C. Fu, X. Wu, Y. Hu, H. Huang. and R. He. Dvg-face: Dual variational generation for
heterogeneous face recognition. TPAMI. 44(():2938-25) -2952, 2021. 4
[22] S. Gu, A. Lugmayr. M. Danelljan, M. Fritsche, J. Lamour, and R. Timofte. Div8k: Diverse 8k
resolution image dataset. In ICCVW, pages 3512-3516 2019. 4.
[23] Y. Guo, J. Chen, J. Wang, Q. Chen, J. Cao, Z. Deng, Y. Xu, and M. Tan. Closed-loop matters:
Dual regression networks for single image super-resolution. In CVPR, pages 5407-5416, 2020.
2
[24] Y. Guo, L. Zhang, Y. HU, X. He, and J. Gao. Ms-celeb-Im: A dataset and benchmark for large-scale face recognition. In ECCV, pages 87-102, 2016. 4
[25] H. A. A. K. Hammoud, H. Itani, F. Pizzati, P. Torr, A. Bibi, and B. Ghanem. Synthelip: Are we AXXv:2442200182) 2024. [26] X. Han, Y. Jian, X. Hu, H. Liu, Y. Wang, Q. Fan, Y. Ai, H. Huang, R. He, Z. Yang, et al. Infimm-
ready for fully synthetic clip training? arXiv preprint
webmath-40b: Advancing multimodal pre-training for enbanced mathematical reasoning. arXiv
11

---

preprint mrXiv:2409.125688 2024. 4
[27] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In ICCV, pages 2017.
9
[28] R. He. S. Sun, X. Yu, C. Xue, W. Zhang, P. Torr. S. Bai. and X. Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210.07574, 2022. 4
[29] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two
time-scale update rule converge to a local nash equilibrium. NeurlPS, pages 6626-6637, 2017.
7
[30] J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12590
2022.
[31] H. Huang, R. He, Z. Sun, and Tan. Wavelet-srnet: A wavelet-based enn for multi-scale face super resolution. In ICCV, pages 1689-1697, 2011 3
[32] H. Huang, M. Luo, and R. He. Memory uncertainty learning for real-world single image
deraining. TPAMI, 45(3):344633460, 2022. 3
[33] H. Huang, A. Yu, 7. Chai, R. He, and T. Tan. Selective wavelet attention learning for single
image deraining. IJCV, 129(4):1282-1300, 2021. 3
[34] H. Huang, A. Yu, and R. He. Memory oriented transfer learning for semi-supervised image deraining. In CVPR, pages 7732-7741, 20
[35] Karras, A. Holynski, T.-C. Wang, and Kemelmacher-Shlizemmein
Dreampose: Fashion image-to-video synthesis via stable diffusion. In ICCV, pages 22623-22633, 2023. 3 [36] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial
networks. In CVPR, pages 4401-4410, 2019. 3
[37] B. Kawar, M. Elad, S. Ermon, and J. Song. Denoising diffusion restoration models. In NeurlPS, pages 23693, 2022. 3
[38] H. Lee, K. Kang. H. Lec, S.-H. Back, and S. Cho. Ugpnet: Universal generative prior for image restoration. In WACV, pages 1598-160M 2024. 3
[39] Y. Li, K. Zhang, J. Liang, Cao, C. Liu, Gong, Y. Zhang, H. Tang, Y. Liu. D. Demandolx, et al. Lsdir: A large scale dataset for image restoration. In CVPRW, pages 1775-1787, 2023. 4,
[40] Y. Li, Y. Zhang, R. Timofte, L. Van Gool, Z. Tu, K. Du, H. Wang, H. Chen, W. Li, X. Wang. et al. Ntire 2023 challenge on image denoising: Methods and results. In CVPR, pages 1904-1920, 2023. [41] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte. Swinir: Image restoration using swin transformer. In ICCVW, pages 133--1844, 2021. 3, 5, 5,7,8
[42] J. Liang, H. Zeng, and L. Zhang. Details or artifacts: A locally discriminative learning approach
to realistic image super- reportresolution. In CVPR, pages 5657-5666, 2022. 7
[43] Liang, H. Zeng, and L. Zhang. Efficient and degradation-adaative network for real-world image super-resolution. In ECCV, pages 574-591, 2022. 3, 7,
[44] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee. Enhanced deep residual networks for single
image super-resolution. In CVPRW, pages 136-144, 2017. 2, 4,
[45] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.
Microsoft coco: Common objects in context. In ECCV, pages 740-755, 2014. 9
[46] X. Lin, J. He, Z. Chen, Z. Lyu, B. Fei, B. Dai, W. Ouyang, Y. Qiao, and C. Dong. Diffbir: To-
wards blind image restoration with generative diffusion prior. arXiv preprint 2rXiv:2308.15000
2023. 7, 8, 19, 20,21
[47] H. Liu, C. Li, Q. Wu, and J. Lee. Visual instruction tuning. In NeurlPS, 2023. 5,8
[48] M. Liu, Y. Wei, X. Wu, W. Zuo, and L. Zhang. Survey on leveraging pre- pre-trained generative
adversarial networks for image editing and restoration. Science China Information Sciences, 66(5):15100, 2023. 3
[49] Z. Luo, Y. Huang. S. Li, L. Wang, and T. Tan. Learning the degradation distribution for blind image super-resolution. In CVPR, 2022. 2
[50] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon. Sdedit: Guided image
synthesis and editing with stochastic differential equations. In ICLR, 2022. 2, 3, 4, 15
[51] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In JCML,
pages 8162-8171, 2021. 8
[52] X. Pan, X. Zhan, B. Dai, D. Lin, C. Loy, and P. Luo. Exploiting deep generative prior for
versatile image restoration and manipulation. TPAMI, 44<11):7474-744 2021. 3
[53] W. Peebles and S. Xie. Scalable diffusion models with transformers. In ICCV, pages
2023. 3.5
[54] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.
12

---

[55] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In ICLR. 2023. [56] Y. Quan, Z. Wu, and H. Ji. Neumann network with recursive kernels for single image defocus
deblurring. In CVPR, pages 5754-5763, 2023. 3
[57] Rim, H. Lee, Won, and S. Cho. Real-world blur dataset for learning and benchmarking
deblurring algorithms. In ECCV, pages 184-201, 2020. 3
[58] E. Ristani. F. Solera, R. Zou, R. Cucchiara, and C. Tomasi. Performance measures and data
set for multi-target, multi-camera tracking. In ECCV, pages 17-35, 2016. 4
[59] R. Rombach, A. Blattmann, D. Lorenz. P. Esser, and B. Ommer. High-resolution image synthesis
with latent diffusion models. In CVPR. pages 10684-10695, 2022. 2. 3. 7
[60] O. Ronneberger, P. Fischer, and T. Brox. segmentation. In MICCAI, pages 234-241, 2015. 5
Convolutional networks for biomedical image
[61] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously
large neural networks: The sparsely-gated mixture-of-cxpert layer. In ICLR, 2017. 3 [62] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint
ar arXiv:2312.111 18055, 2023. 2, 4
[63] J. Wang. 7. Yue, S. Zhou, K. C. Chan, and C. C. Loy. Exploiting diffusion prior for real-world
image super-resolution. arXiv preprini arXiv:2305.02) 070/5, 2023. 3, 6, 7, 8, 19, 20, 21
[64] X. Wang. I.. Xie, C. Dong, and Y. Shan. Real-esrgan: Training real-world blind super-resolution
with pure synthetic data. ICCVW, pages 1904-1914, 2021. 2, 3, 6, 7,
[65] Y. Wang, W. Yang. X. Chen, Y. Wang, L. Guo, L.-P. Chau, 7. Liu, Y. Qiao, A. C. Kot, and
B. Wen. Sinsr: Diffusion-based image super-resolution in single step. In CVPR, 2024. 3, 7, 8
[66] Y. Wang. J. Yu, and J. Zhang. Zero-shot image restoration using denoising diffusion null-space
model. arXiv preprint
2022. 3
[67] Z. Wang, Y. Fu, J. Liu, and Y. Zhang. Lg-bpn: Local and global blind-patch network for world denoising. In CVPR, pages 181 18156-18165, 2023. 3
self-supervise
[68] P. Wei, Z. Xie, H. Lu, Z. Zhan, Q. Yc, W. Zuo, and L. Lin. Component divide-and-conquer for real-world image super-resolution. In ECCV, pages 101-117, 2020. 2,
[69] J. Whang. M. Delbracio, H. Talebi, C. Saharia, A. G. Dimakis, and P. Milanfar. Deblurring via
stochastic refinement. In CVPR, pages 16293-16303 2022. 3
[70] R. Wu, T. Yang. L. Sun, Z. Zhang. S. Li, and L. Zhang. Seesr: Towards semantics-aware
real-world image super-resolution. In CVPR, 2024. 2, 6, 7, 8, 19, 20, 21
[71] Y. Wu, X. Wang, G. Li. and Y. Shan. Animesr: Learning real-wont world super-resolution models for
animation videos. In NeurlPS. pages 11241-11252, 2022. 2.
[72] B. Xia, Y. Zhang, Y. Wang. Y. Tian, W. Yang, R. Timofte, and L. Van Gool. Knowledge
distillation based degradation estimation for blind super-resolution. In ICLR, 2023.
[73] T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun. Unified perceptual parsing for scene understanding.
In ECCV, pages 418-434, 2018. 9
[74] X. Xu, P. Wei, Chen, Y. Liu, M. Mao, L. Lin, and G. Li. Dual adversarial adaptation for cross-device real-world image super-resolution. In CVPR, pages 5667-5676, 2022. 2, [75] S. Yang, T. Wu, S. Shi, S. Lao, Y. Gong, M. Cao, J. Wang, and Y. Yang. Maniqa: Multi- dimension attention network for no-reference image quality assessment. In CVPR, pages 1191-1200, 2022. 7
[76] T. Yang. P. Ren, X. Xie, and L. Zhang. Gan prior embedded network for blind face restoration
in the wild. In CVPR, pages 672-681, 2021. 3
[77] T. Yang. R. Wu,, P. Ren, X. Xie, and L. Zhang. Pixel-aware stable diffusion for realistic image super-resolution and personalized stylization. arXiv preprint arXiv:2308. /4469, 2023. [78] H. Ye, J. Zhang, S. Liu, X. Han, and W. Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308. 007211..., 2023.
[79] Z. You, Z. Li, Gu, Z. Yin, T. Xue, and C. Dong. Depicting beyond scores: Advancing image quality assessment through multi modal language models. arXiv preprint arXiv:2312.08866
2023.
[80] F. Yu, J. Gu, Z. Li, J. Hu, X. Kong, X. Wang, J. He. Y. Qiao, and C. Dong. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. In CVPR,
2024. 2, 6, 7, 8, 19, 20, 21
[81] Y. Yuan, S. Liu, J. Zhang, Y. Zhang, C. Dong, and L. Lin. Unsupervised image super- resoltteso
using cycle-in-cycle generative adversarial networks. In CVPRW. pages 701-710. 2018. 2 [82] Z. Yue, J. Wang. and C. C. Loy. Resshift; Efficient diffusion model for image suser-resolution
by residual shifting. In NeurlPS, 2023. 7, 8
[83] X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer. Scaling vision transformers. In CVPR, pages
22104-12113 2022. 4
13

---

[84] J. Zhang, Y. Zhang, J. Gu, J. Dong, L. Kong, and X. Yang. Xformer: Hybrid x-shaped transformer for image denoising. In ICLR, 2024. 3
[85] K. Zhang, J. Liang. L. Van Gool, and R. Timofte. Designing a practical degradation model for deep blind image super-resolution. In ICCV, pages 4791 2021. 3,
[86] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang. Beyond a gaussian denoiser: Residual learning of deep enn for image denoisiin. TIP, 26(7):314233155 2017. 7
[87] K. Zhang, W. Zuo, and L. Zhang. Ffdnet: Toward fast and flexible solution for cnn-based
image denoising. TIP, ((1)):608-46222 2018. 3
[88] L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion
models. In ICCV. pages 3836-3847, 2023.
[89] L.. Zhang, L. Zhang, and A. C. Bovik. A feature-enriched completely blind image quality evaluator. TIP, 24(8):2579-25 2591, 2015.
[90] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, pages 586-595, 2018. 7
[91] X. Zhhang Q. Chen, R. Ng. and V. Koltun. Zoom to learn, learn to zoom. In CVPR, pages
3762-3770, 2019.
[92] Y. Zhang, K. Zhang, 7. Chen, Y. Li, R. Timofte, J. Zhang, K. Zhang, R. Peng, Y. Ma, L.. Jia,
al. Ntire 2023 challenge image super-resolution (x4): Methods and results. In CVPRW,
pages 1864-1883, 2023. 2
[93] 7. Zheng, X. Peng, and Y. You. Open-sora: Democratizing efficient video production for all, 2024.
[94] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through
ade20k dataset. In CVPR, pages 633-641, 2017. 9
[95] X. Zhou, H. Huang, R. He, Z. Wang. J. Hu, and T. Tan. Msra-sr: Image super-resolution
transformer with multi-scale shared representation acquisition. In ICCV, pages 12665-12676
2023.
[96] X. Zhou, H. Huang, Z. Wang, and R. He. Ristra: Recursive image super-resolution transformer
with relativistic assessment. 7MM, 26(8):6475-653 2024. 3
[97] X. Zhou, Li, Z. Wang, R. He, and T. Tan, Image inpainting with contrastive relation network. In ICPR, pages 4420-44- 4427, 2021. 3
14

---

A Appendix
A.1 More Implementation Details
For generating negative samples, the strength in SDEdit [50] is set 0.6. To minimize the risk of generating images that contain private information in GenlR, we employ Gemini for both text prompt filtering and generated image filtering. The prompts for Gemini are set as "You are an Al language assistant, and you are analyzing a series of text prompts. Your task is to identify whether these text prompis contain any inappropriate content such as personal privacy violations or NSFW material. Delete any inappropriate text prompts and return the remaining ones in their original format." " and You are an Al visual assistant, and you are analyzing a single image. Your task is to check the image for any anomalies, irregularities, or content that does not align with common sense or normal expectations. Additionally, identify any inappropricte content such as personal privacy violations or NSFW material. If the image does not contain any of the aforementioned issues, it has passed the inspection. Please determine whether this image has passed the inspection (answer yesho) and provide your reasoning." respectively.
A.2 More Details of User Study
To evaluate our approach, we conducted user study emphasizing restoration quality. We randomly selected 100 low-quality images from the test sets (Tab. 1), applying our model and five other leading methods produce restored images, generating 100 groups of seven images each. Users, guided by the low-quality image, were asked to select the best-restored image from each group, considering factors such visual quality, naturalness, detail accuracy, and absence of distortions or artifacts. Fairness was ensured by presenting each user with 10 randomly selected groups, randomizing image sequence within each group, and masking the methods. We widely disseminated the online questionnaire without restrictions, amassing feedback from 256 evaluators.
We employed two metrics to quantify our study's results: vote percentage and Top-K K ratio. The former represents the proportion of total votes each method received, while the latter measures the frequency method was among the top-k selections, indicating the proportion of most preferred images and a model's consistency in producing high-quality images. The Top-K ratio is defined as: R = N Ejnm 1(1 E Fropk (s,, k)), where sj = {sigl 0....,5) 5) are the selection scores of the j-th group from N groups (with N = 100), and Fiopk is the top-k operation.
As shown in Fig. 5, our model outperformed others both metrics. Our model received over 45% of the total votes, indicating a strong user preference. Additionally, our method was the top choice for 80% of the images, and it was ranked first second for 98% of the images, highlighting our method's reliable ability to generate superior quality images compared to other methods.
A.3 More Ablations
More analysis of DreamClear Ablation. We give a more detailed analysis in the following.
(a) Mixture of Adaptive Modulator (MoAM). MoAM acts as interaction module between the LQ branch and the reference branch, aiming to enhance the model's s robustness to intricate real-world degradations by explicitly guiding it through the introduction of loken-wiss degradation priors. It obtains the degradation map through cross-attention between the LQ features and reference features, guiding the dynamic fusion of expert knowledge to modulate LQ features.
Tab. 3 presents the ablation studies of MoAM. Notably, when we substitute the Mixture of AM design with AM, all metrics undergo substantial decrease, underscoring the importance of the degradation prior guidance in MoAM for steering restoration experts. Moreover, we conducted experiments replacing AM with cross-attention and zero-linear layer. The use of cross-attention, in comparison to AM, leads to a general reduction in model performance. While the zero-linear layer provides minor improvements in MANIQA and MUSIQ scores, it results in a significant drop in perceptual fidelity.
(b) Dual-branch framework. The incorporation of a reference branch allows the model to focus less on degradation removal and more enhancing image details through generative priors, ultimately producing more photorealistic images. The results in Tab. 3 indicate that our dual-branch structure significantly outperforms using only the LQ branch across all metrics.
15

---

LQ In.input
w/o Text Prompt
w/o MoAM, w/ AM
w/o MoAM, w/ Zero-Linear
w/o MoAM, w/
DreamClear [Ours]
Figure Visual comparisons for ablation study on DreamClear.
Table 4: Ablations for GenlR on LSSIR- Val using SwinIR-GAN.
Training Data
LPIPS +
DISTS 4
FTD 4
MANIQA t
MUSIQ f
CLIPIQA
Pre-trained T2I Model (3450images)
0.4819 0.4578
0.2790 0.2435
60.12 51.29
0.3271
61.94
0.5423
Ours GenlR (3450images)
0.3691
63.12
0.5647
(C) Text prompt guidance. We evaluate the impact of using detailed text prompts generated by MLLMs versus null prompts. Despite null prompts slightly outperforming detailed prompts perceptual metrics, the superior performance on no-reference and high-level vision metrics indicates that text prompts more effectively retain semantic information. Visual comparisons in Fig. 7 further underscore the advantages text prompts in providing semantic guidance for image restoration.
Visual results for DreamClear Ablation. Following the setting in Tab. we provide more visual comparison results in Fig. 7. We find that when using a null prompt instead of text prompt generated by MLLM, there are significant semantic errors in the eyes of the bear the restoration results. This demonstrates that the semantic information provided by MLLM-generated detailed text prompts helps the model achieve more ideal restoration results. When using AM, zero-lineat and cross-attention instead of MoAM, the model tends to produce results that are either too smooth or contain semantic MoAM. Removing the reference branch results significant
errors, proving the effectiveness deterioration of the restoration outcomes. Overall, our full model, DreamClear, achieves the best results in terms of fidelity and perception.
Ablations for GenlR. We use the exact same prompts and sampling parameters to compare images generated by GenlR with those generated by the originally pre-trained T2I model. As shown in Fig. 10, the images generated by our proposed GenlR are more realistic and contain more texture details, while those generated by the pre-trained T2I model tend to have issues like being overly smooth and blurry. Intuitively, images generated by GenlR world IR. To verify this, following the setting of Fig. 6, we use manually designed prompts with the pre-trained T2I model to generate 3450 images for training SwinIR-GAt As shown in Tab. 4, the model trained using images generated by our GenlR shows significant improvements across all metrics, quantitatively demonstrating the effectiveness of our approach.
likely to be more helpful for real-
In addition, we provide visual comparisons in Fig. to verify the effectiveness of dual-prompt learning in GenlR. It shows that the dual-prompt learning strategy can effectively enhance image texture details, making the generated images more suitable for image restoration training. Fig. also demonstrates the effectivenees of our generated data in enhancing the visual effect of IR models.
16

---

w/ dual-promp learning
w/o dual prompt learning
w/ dual prompt learning
w/o dual-promp learning
Figure 8: Visual comparisons for ablation study on GenlR.
Low-Quality Input
Figure 9: Visual comparisons for ablation study training datasets.
A.4 More Real- World Visual Comparisons
We provide more real-world visual comparisons with tfate-of-the-arr diffusion-bart based real-world image restoration methods in Fig. 11, Fig. 12 and Fig. 13.
17

---

Pre-trained T2I Model
GenlR (Ours)
Figure 10: Visual comparison of images generated using the pre-trained T2I model and GenIR. Our proposed GenlR produces images with enhanced texture and more realistic details, exhibiting less blurring and distortion. This makes it better suited for training real-world IR models.
18

---

LQ Input
StableSR [63]
DiffBIR [46]
SeeSR [70]
SUPIR [80]
Dream Clear (Ours)
LQ Input
StableSR [63]
DiffBIR [46]
SeeSR [70]
SUPIR [80]
Dream Clear (Ours)
Figure 11: Visual comparisons on real-world benchmarks (1/3). Please zoom in for better view.
19

---

LQ Input
StableSR [63]
DiffBIR [46]
SeeSR [70]
SUPIR [80]
Dream Clear (Ours)
LQ Input
StableSR [63]
DiffBIR [46]
SeeSR [70]
SUPIR [80]
Dream Clear (Ours)
Figure 12: Visual comparisons on real-world benchmarks (2/3). Please zoom in for better view.
20

---

LQ Input
StableSR [63]
DiffBIR [46]
SeeSR [70]
SUPIR [80]
Dream Clear (Ours)
LQ Input
StableSR [63]
DiffBIR [46]
SeeSR [70]
SUPIR [80]
Dream Clear (Ours)
Figure 13: Visual comparisons on real-world benchmarks (3/3). Please zoom in for better view.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflect the paper's contributions and scope, and we summarize the contributions the end of Section I.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of the work in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
There isn't theoretical derivation in our work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We introduce the method clearly in Section and Section 4, and present the detailed experimental settings and implementation details in Section 5.1.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: The code and pre-trained models are available at https://githh/ com/ shallowdream/ aaeaam000DDreaeeaaaa
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips. ctips.cc/public//uu guwdee/dss///// CCodeSubeesiaceecc simiisssinn sssssnoliccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Detailed experimental settings can be found in Section 5.1.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our model is trained on large dataset and the inference speed of diffusion models is relatively slow. Due to limited computational resources, we can't 't afford the statistical significance experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the computational resources in Section 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttp///eerrrp/eeeeee/eee/ hccppublcc/thhicccuiieeee
Answer: [Yes]
Justification:
All the authors have reviewed the code of ethics and obey the code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed both potential positive societal impacts and negative societal impacts of the work in Section 6.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited related original papers in paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We have included the full text of instructions given to participants the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We adhere to the NeurIPS Code Ethics and the guidelines for our institution.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "e2y0lWLD8q", "reviewer": "Reviewer_CpjF", "summary": "The paper introduces \"DreamClear,\" a high-capacity image restoration model, and \"GenIR,\" an innovative data curation pipeline for image restoration (IR) tasks. DreamClear leverages Diffusion Transformer (DiT) models and generative priors from text-to-image (T2I) diffusion models, combined with multi-modal large language models (MLLMs), to achieve photorealistic restorations. GenIR is a dual-prompt learning pipeline that generates a large-scale dataset of high-quality images while ensuring privacy and copyright compliance. The combination of these strategies enhances the model's ability to handle diverse real-world degradations.", "questions": "1. Some technical details are not clear. For example, what kind of loss do you use in this paper? While the paper outlines the steps of the GenIR pipeline, it lacks specific details on parameter settings, such as the hyperparameters used in the generative models, data augmentation methods during training, and strategies for generating negative samples. There is insufficient detail on the criteria and standards used to evaluate and select the generated dataset.\n\n2. One potential issue with the DreamClear approach is the risk of generating artificial textures during the image restoration process. Taking Figure 4 as an example, although the result is clearer, it may produce artifacts because the beak of this bird does not look like this.\n\n3. In Figure 6, as data size increases, performance improves. Is there further improvement for more datasets, e.g., increasing the number of training images to 200000? \n\n4. It would be better to compare the efficiency of different methods, e.g., model size, training/inference time, FLOPs.\n\n5. Some necessary related work should be discussed. There are many diffusion-based image restoration methods, e.g., DDRM, DDNM, DeqIR, etc. It would be better to cite and discuss them.", "limitations": "Please refer to the details above.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. GenIR provides a novel, privacy-safe method to create large-scale datasets, overcoming the limitations of existing datasets that are often small and not comprehensive.\n2. DreamClear integrates generative priors from T2I diffusion models and employs a Mixture of Adaptive Modulator (MoAM) to handle diverse degradation scenarios effectively.\n3. The model achieves high perceptual quality in restored images, compared with other methods in various benchmarks.", "weaknesses": "1. Complexity in Implementation: The dual-branch structure and the integration of multiple experts for degradation handling add complexity to the model.\n2. Some Technical details and more experiments should be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for the valuable and positive comments on our work. We address the questions and clarify the issues accordingly as described below.\n\n**Q1: About Model Implementation**\n\n**[Reply]** Thanks. We explain the motivation behind our model design as follows:\n* Dual Branch:  Our dual-branch structure incorporates the reference image, allowing the model to focus less on degradation removal and more on enhancing image details through generative priors, ultimately producing more photorealistic images. Considering potential detail loss in the reference image, we use both the LQ image and the reference image to jointly guide the diffusion model in order to obtain a realistic image that remains faithful to the original.\n* Multiple Experts: The proposed Mixture-of-Adaptive-Modulator (MoAM) aims to enhance the model's robustness to intricate real-world degradations by explicitly guiding it through the introduction of token-wise degradation priors. Specifically, we feed the token-wise degradation map into a router network to dynamically select different restoration experts for effective feature fusion. MoAM dynamically fuses expert knowledge, leveraging degradation priors to tackle complex degradations.\n\nTable 3 and Figure 7 provide quantitative and qualitative ablations, respectively, to demonstrate the effectiveness of our model design. Besides, we also provide the efficiency comparison with SOTAs (Please refer to the **global response**). The efficiency of DreamClear is comparable to SeeSR and SUPIR. Therefore, we believe that our design is effective and not redundant.\n\n**Q2: About technical details**\n\n**[Reply]** Thanks for pointing out this. We provide **more training details** in the **global response**.\n\nTo **generate negative samples**, we adopt SDEdit [1] as the image editing technique. Specifically, we use text prompts such as _\"cartoon, painting, blur, messy, low quality, deformation, low resolution, over-smooth, dirty\"_ to edit the positive samples. We set the strength in SDEdit to $0.6$, resulting in the corresponding negative samples. During the fine-tuning phase, the number of negative samples is controlled to be one-tenth of the number of positive samples.\n\nTo **evaluate & select appropriate samples** for our dataset, we first employ a quality classifier to screen the quality of the generated samples. Then we utilize Gemini as a powerful MLLM to ascertain whether the images exhibit blatant semantic errors or inappropriate content. The implementation details are as follows:\n* We use a convolutional neural network (CNN) as a binary classifier, trained with an equal number of positive and negative samples. Specifically, the negative samples are obtained through SDEdit with the strength set to $0.25$. Samples with a classification probability of less than $0.8$ are filtered out.\n* The prompt for Gemini is set to _\"You are an AI visual assistant, and you are analyzing a single image. Your task is to check the image for any anomalies, irregularities, or content that does not align with common sense or normal expectations. Additionally, identify any inappropriate content such as personal privacy violations or NSFW material. If the image does not contain any of the aforementioned issues, it has passed the inspection. Please determine whether this image has passed the inspection (answer yes/no) and provide your reasoning.\"_ Samples that do not pass the inspection will be eliminated.\n* Only the samples that have passed these two rounds of screening will be retained.\n\nWe will add all these technical details in the final version.\n\n**Q3: About artificial textures**\n\n**[Reply]** Thank you for your valuable feedback. Real-world image restoration is an ill-posed problem, meaning there isn't a single definitive solution. Our diffusion-based approach aims to generate visually pleasing restored images in most cases. However, when the degradation of the input image is very severe, it becomes challenging to ensure that certain details in the restored image are faithful to the input.\n\nRegarding the bird (gyrfalcon) in Figure 4, the benchmark does not include a corresponding GT image. Therefore, we sourced photos of gyrfalcons from the internet to compare the results of different methods. These comparison results are provided in the **attached PDF file** (Figure I). Despite some differences between our restored image and the real gyrfalcon, our approach significantly surpasses SeeSR and SUPIR in terms of clarity and semantics perseveration.\n\nBesides, we also provide more real-world comparisons with SUPIR in the **attached PDF file** (Figure II). It demonstrates that our method can achieve clearer details and fewer artifacts when dealing with real-world cases. We'll add more real-world comparisons in our final paper.\n\n**Q4: About the increasing data size**\n\n**[Reply]** Thanks for the advice, after increasing the data size, the results are provided as follows:\n|Image Number|LPIPS $\\downarrow$|DISTS $\\downarrow$|FID $\\downarrow$|MANIQA $\\uparrow$|MUSIQ $\\uparrow$|CLIPIQA $\\uparrow$|\n|:-------:|:------:|:------:|:------:|:------:|:------:|:------:|\n|100000|0.3902|0.1982|43.27|0.4413|68.27|0.6382|\n|200000|0.3873|0.1951|42.13|0.4502|68.83|0.6469|\n\nAfter increasing the data size, the performance of the model is improved. We'll add more results using differenet data sizes in our final paper.\n\n**Q5: About the efficiency comparison**\n\n**[Reply]** Thanks for this valuable suggestion. Please refer to the **global response**.\n\n**Q6: About the related work**\n\n**[Reply]** Thanks for your reminder. DDRM, DDNM, and DeqIR are all excellent training-free methods that handle various image restoration tasks by improving the sampling process of pre-trained diffusion models. We'll cite all these related diffusion-based works and discuss them in our final paper.\n\n**References**\n\n[1] Sdedit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8vhlXNEsYM", "reviewer": "Reviewer_A2Rj", "summary": "The authors propose GenIR, a privacy-safe automated pipeline that generates a large-scale dataset of one million high-quality images for training of the image restoration (IR) models. Additionally, they introduce DreamClear, a  IR model that seamlessly integrates degradation priors into diffusion-based IR models. DreamClear features the novel Mixture of Adaptive Modulator (MoAM), which adapts to diverse real-world degradations. Comprehensive experiments demonstrate its outstanding performance in managing complex real-world situations, marking a substantial progression in IR technology.", "questions": "1. Does the Eq. 2 and 3 holds only for the x_lq, does not it hold for x_ref?", "limitations": "Yes, the authors addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The authors propose an automated data curation pipeline for image restoration and extensive experiments across both low-level and high-level benchmarks demonstrates the DreamClear’s state-of-the-art performance in handling intricate real-world scenarios.", "weaknesses": "1. The overall impact of the generated dataset is not that convincing. Why is the training also performed on DIV2K, FLickr2K, LSDIR and Flickr8K. \n2. The overall architecture in terms of contribution is too limited where the concept of mixture of experts modified to mixture of adaptive modulators and the text to image diffusion model have been widely already explored in the field of image restoration.\n3. The first ablation study is not highlighting the impact of the proposed dataset.\n4. It would be interesting to see the results without dual-based prompt learning if time permits?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for the valuable and positive comments on our work. We address the questions and clarify the issues accordingly as described below.\n\n**Q1: About the generated dataset**\n\n**[Reply]** Thanks for your valuable suggestion. The primary goal of the proposed GenIR is to expand the dataset as much as possible to train a more robust model. As shown in Figure 6, the performance of the generated dataset of equivalent size is inferior to that of the real dataset. Therefore, by combining existing IR datasets with our generated dataset, we aim to achieve optimal model performance.\n\nWe also conduct ablation studies to explore the influence of different training datasets. We retrain DreamClear using different datasets, with the results provided below (evaluated on DIV2K-Val):\n\n| Trainset   | LPIPS $\\downarrow$ | DISTS $\\downarrow$  |FID $\\downarrow$  |MANIQA $\\uparrow$  |MUSIQ $\\uparrow$  |CLIPIQA $\\uparrow$  |\n| :------- | :------: | :------: | :------: | :------: | :------: |:------: |\nExisting IR+Generated | 0.3391 |0.1791|23.24|0.4327|68.93| 0.6978|\nGenerated | 0.3601 |0.1847|27.84|0.4295|69.31| 0.6823|\nExisting IR | 0.3493 |0.1873|24.10|0.4154|63.24| 0.6435|\n\nIt shows that using the generated dataset can achieve perceptual metrics (LPIPS, DISTS) similar to those obtained with existing IR datasets, and has obtained a significant advantage in no-reference metrics (MANIQA, MUSIQ, CLIPIQA). We speculate that the gap in FID is mainly due to the consistent distribution between the training and validation sets in the DIV2K dataset. The results indicate that our generated data can effectively improve the quality of restored images compared to existing IR datasets, and combining the two can achieve the best performance. We also provide some visual comparisons in the **attached PDF file** (Figure IV), which demonstrates the effectiveness of our generated data in enhancing visual effects. We'll add the ablations in the final version.\n\n**Q2: About our contribution**\n\n**[Reply]** Thanks. In this paper, our technical contributions are mainly divided into two aspects:\n\n**GenIR.** We introduce the GenIR pipeline, which provides a privacy-safe and cost-effective method to generate data for image restoration. Ultimately, we obatin  a dataset containing one million high-quality images and verify the effectiveness of the generated data for real-world image restoration.\n\n**DreamClear.** Recent works based on text-to-image diffusion models indeed demonstrate their superior performance in real-world image restoration. Compared to existing works, the architecture of DreamClear mainly differs in the following three aspects:\n\n* We propose a dual-branch structure, which incorporates the reference image, allowing the model to focus less on degradation removal and more on enhancing image details through generative priors, ultimately producing more photorealistic images.\n* We propose a novel Mixture of Adaptive Modulator (MoAM) to enhance our model’s robustness to intricate real-world degradations by explicitly guiding it through the introduction of token-wise degradation priors. Unlike the commonly used MoE structure in LLM, MoAM dynamically guide different restoration experts through degradation maps to achieve more robust image restoration.\n* To the best of our knowledge, our work is the first to explore the performance of the diffusion transformer (DiT) architecture in image restoration.\n\nWe will add the discussion on how our work differs from related works in our final paper. We also hope to get more comments and suggestions from you, such as related works built upon MoE structures in low-level vision, to further improve our paper.\n\n**Q3: About the first ablation**\n\n**[Reply]** Thanks. The main purpose of the first ablation is to explore whether expanding the scale of the generated data can improve the effects of real-world image restoration. To examine the impact of the proposed dataset, we conduct ablations using different training datasets. Please refer to the **Reply of Q1**.\n\n**Q4: About dual-based prompt learning**\n\n**[Reply]** Thanks for your valuable advice.  We remove the dual-based prompt learning strategy and retrain GenIR. We provide qualitative comparisons of the GenIR-generated images for the ablation study. The results are provided in the **attached PDF file** (Figure V). It shows that dual-based prompt learning strategy can effectively enhance image texture details, making the generated images more suitable for image restoration training. We'll add more ablation results in the final version.\n\n**Q5: About Eq. (2) and Eq. (3)**\n\n**[Reply]** Thanks. As illustrated in the paper (Line 206-208), $x_{ref}$ is directly fed into AM as the conditional information, while $x_{lq}$ is fed into the mixture of degradation-aware experts structure. Figure 3(c) also depicts this process. Therefore, Eq. (2) and Eq. (3) hold only for $x_{lq}$, but not for $x_{ref}$. We'll give a clear illustration of this process in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IIAn75ju94", "reviewer": "Reviewer_FDew", "summary": "The paper introduces a dual strategy to tackle the challenges of image restoration (IR) datasets and the development of high-capacity models for image restoration. This strategy comprises:\nGenIR: An innovative data curation pipeline designed to bypass the laborious data crawling process, providing a privacy-safe solution for IR dataset construction. The authors generated a large-scale dataset of one million high-quality images.\nDreamClear: A Diffusion Transformer (DiT)-based image restoration model utilizing generative priors from text-to-image (T2I) diffusion models and the perceptual capabilities of multi-modal large language models (MLLMs) to achieve photorealistic restorations.", "questions": "Given that a million images were generated, how did the authors verify that none of the images had personal information?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The authors made a significant contribution to creating a large and robust image restoration dataset while providing a privacy-safe solution.\n\nThe authors demonstrated detailed evaluation with recent SOTAs.\n\nThe restoration model shows good performance across different evaluation metrics.\n\nThe model is robust enough to handle various degradations such as deblurring and denoising.", "weaknesses": "For the created data, the paper claimed they maintained privacy standards to ensure no personal information was embedded in the generated images. However, the author failed to provide clear details on how these were achieved. Hence, it is difficult to establish the robustness and effectiveness of the privacy-preserving measures. \n\nGiven that the SR model, DreamClear, is an integration of various restoration models and is also built on PixArt, a detailed comparative analysis of the computational complexity between the proposed framework and other existing methods would be beneficial.\n\nWhile DreamClear exhibits good performance in perceptual quality, its performance on traditional metrics like PSNR and SSIM is not as strong.\n\nSome grammatical errors are observed on line 96, and incomplete statements on lines 109 and 110 should be corrected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for the valuable and positive comments on our work. We address the questions and clarify the issues accordingly as described below.\n\n**Q1: About privacy preservation**\n\n**[Reply]** Thanks for this valuable advice. To minimize the risk of generating images that contain private information, we have implemented constraints from two aspects:\n\n* **Text Prompt Filtering**: As illustrated in the paper (Line 160-162), we levarage Gemini to generate millions of text prompts for the T2I model. Instead of directly using these text prompts, we utilize Gemini to review them and filter out any prompts that contain private information. We set the prompt for Gemini as _\"You are an AI language assistant, and you are analyzing a series of text prompts. Your task is to identify whether these text prompts contatin any inappropriate content such as personal privacy violations or NSFW material. Delete any inappropriate text prompts and return the remaining ones in their original format.\"_ All text prompts will be filtered through Gemini, and ultimately we retained one million text prompts.\n* **Generated Image Filtering**: As shown in Figure 2, we utilize Gemini as a powerful MLLM to ascertain whether the generated images exhibit blatant semantic errors or personal private content.  We set the prompt for Gemini as _\"You are an AI visual assistant, and you are analyzing a single image. Your task is to check the image for any anomalies, irregularities, or content that does not align with common sense or normal expectations. Additionally, identify any inappropriate content such as personal privacy violations or NSFW material. If the image does not contain any of the aforementioned issues, it has passed the inspection. Please determine whether this image has passed the inspection (answer yes/no) and provide your reasoning.\"_ Samples that do not pass the inspection will be eliminated.\n\nCompared to directly crawling data from the web, the proposed GenIR pipeline significantly lowers the risk of infringing on individuals' privacy. We believe that the GenIR pipeline presents a promising solution for addressing security and ethical concerns in contemporary research involving large models. We'll add the privacy preservation details in our final paper.\n\nWe acknowledge that ensuring robust privacy preservation poses significant challenges. In this paper, we conduct an exploration of privacy preservation methods with the assistance of advanced MLLMs. Given that the privacy preservation is not the primary focus of our paper, we anticipate exploring this significant area with the community in the future.\n\nRegarding the generation of one million images, we employ the aforementioned methods with the help of MLLMs to minimize the inclusion of personal information as much as possible. We acknowledge that achieving 100% avoidance remains a significant challenge. However, before publicly releasing the dataset, we will screen out all images containing faces and compare them with publicly available face datasets (e.g., CelebA, FFHQ) to remove any images that contain personal information.\n\n**Q2: About computational complexity**\n\n**[Reply]** Thanks for this valuable suggestion. Please refer to the **global response**.\n\n**Q3: About PSNR and SSIM**\n\n**[Reply]** Thank you for highlighting this point. For real-world image restoration, when the degradation is severe, it is challenging to pursue highly accurate reconstruction; instead, the focus is more on achieving better visual quality [1,2]. Table 1 shows that both SUPIR and DreamClear perform poorly in terms of PSNR and SSIM. However, despite lacking an advantage in full-reference metrics like PSNR and SSIM, SUPIR and DreamClear can produce excellent visual effects.\n\nAs mentioned in the paper (Line 244-245), many recent works [1,3,4] also observe this phenomenon and suggest that it is necessary to reconsider the reference values of existing metrics and propose more effective methods to evaluate advanced image restoration methods. Therefore, we conducted a user study in the paper to more comprehensively measure the capabilities of different IR models. We believe that as the field of image quality assessment (IQA) evolves, more suitable metrics will emerge to adequately measure the performance of advanced IR models. We will include further discussion on this topic in our final paper.\n\n**Q4: About writing errors**\n\n**[Reply]** Thanks for pointing out this. We'll carefully check the whole paper and correct these errors in the final version.\n\n**References**\n\n[1] Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. In CVPR, 2024.\n\n[2] SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution. In CVPR, 2024.\n\n[3] Depicting beyond scores: Advancing image quality assessment through multi-modal language models. In ECCV, 2024.\n\n[4] Descriptive Image Quality Assessment in the Wild. arXiv preprint arXiv:2405.18842, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C5ChvN5FGF", "reviewer": "Reviewer_bAzb", "summary": "This paper introduces a dual strategy for real-world image restoration: GenIR and DreamClear. GenIR, a novel data curation pipeline, addresses dataset limitations by creating a large-scale dataset of one million high-quality 2K images. DreamClear, a Diffusion Transformer-based model, utilizes generative priors from text-to-image diffusion models and multi-modal large language models for photorealistic restorations. It also incorporates the Mixture of Adaptive Modulator (MoAM) to handle various real-world degradations. Extensive experiments confirm DreamClear's superior performance, demonstrating the effectiveness of this approach.", "questions": "See the above weaknesses part.", "limitations": "The authors have discussed the relevant limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The writing and structure of the paper are clear and well-organized, making it easy to follow the authors' arguments and methodologies.\n\n2.The authors' introduction of a privacy-safe dataset curation pipeline is significant. This is especially crucial in the era of large models, where data privacy and security are paramount concerns.\n\n3.The experimental results convincingly demonstrate the potential of the proposed method, highlighting its effectiveness and applicability in real-world image restoration scenarios.", "weaknesses": "1.The quality assessment of the newly constructed dataset is lacking, both quantitatively and qualitatively. Given the goal of real-world image restoration, I am concerned about whether the current pipeline effectively advances this objective. The authors need to include a discussion on this matter.\n\n2.In the ablation studies, the authors should provide a detailed discussion on the interaction between the dual branches. The motivation for this design choice requires further explanation.\n\n3.Real-world degradations are complex and intertwined. The authors should compare their approach with SUPIR using more challenging examples to better demonstrate the robustness and applicability of their method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for the valuable and positive comments on our work. We address the questions and clarify the issues accordingly as described below.\n\n**Q1: Quality assessment of the dataset**\n\n**[Reply]** Thank you for this valuable advice. We calculate the FID value between different datasets and DIV2K. The FID results are as follows:\n\n| Flickr2K   | DIV8K | LSDIR  | OST  | Ours (Generated)|\n| :-------: | :------: | :------: | :------: | :------: |\n| 69.83    | 72.28    | 60.22   | 110.29 | 72.88|\n\nThe results indicate that our generated dataset achieves FID values comparable to those of commonly used image restoration datasets. This suggests that our dataset maintains a relatively high image quality. Additionally, we provide some example images from our dataset and LSDIR in the **attached PDF file** (Figure III).\n\nWe also conduct ablation studies to explore the influence of different training datasets. We retrain DreamClear using different datasets, with the results provided below (evaluated on DIV2K-Val):\n\n| Trainset   | LPIPS $\\downarrow$ | DISTS $\\downarrow$  |FID $\\downarrow$  |MANIQA $\\uparrow$  |MUSIQ $\\uparrow$  |CLIPIQA $\\uparrow$  |\n| :------- | :------: | :------: | :------: | :------: | :------: |:------: |\nExisting IR+Generated | 0.3391 |0.1791|23.24|0.4327|68.93| 0.6978|\nGenerated | 0.3601 |0.1847|27.84|0.4295|69.31| 0.6823|\nExisting IR | 0.3493 |0.1873|24.10|0.4154|63.24| 0.6435|\n\nIt shows that using the generated dataset can achieve perceptual metrics (LPIPS, DISTS) similar to those obtained with existing IR datasets, and has obtained a significant advantage in no-reference metrics (MANIQA, MUSIQ, CLIPIQA). We speculate that the gap in FID is mainly due to the consistent distribution between the training and validation sets in the DIV2K dataset. The results indicate that our generated data can effectively improve the quality of restored images compared to existing IR datasets, and combining the two can achieve the best performance. We also provide some real-world visual comparisons in the **attached PDF file**, which demonstrates the effectiveness of our generated data in enhancing visual effects. We'all add the ablations in the final version.\n\n**Q2: Discussion on the dual branch design**\n\n**[Reply]** Thanks for this insightful suggestion. Table 3 presents the quantitative results, while Figure 7 illustrates the qualitative outcomes of the ablation experiments. Some discussions are illustrated in Appendix A.3. Below, we provide a detailed discussion on the dual-branch design:\n\n_Reference Branch:_ The incorporation of reference branch allows the model to focus less on degradation removal and more on enhancing image details through generative priors, ultimately producing more photorealistic images. As shown in Table 3, when the reference branch is removed, all metrics on the low-level and high-level benchmarks deteriorate significantly. Figure 7 also shows that the reference branch can significantly enhance the quality of the image. This indicates that the reference branch plays an important role in DreamClear for improving both image quality and semantic information.\n\n_Interaction Modules_: The proposed Mixture-of-Adaptive-Modulator (MoAM) acts as an interaction module between the LQ branch and the reference branch, aiming to enhance the model's robustness to intricate real-world degradations by explicitly guiding it through the introduction of token-wise degradation priors. It obtains the degradation map through cross-attention between the LQ features and reference features, guiding the dynamic fusion of expert knowledge to modulate LQ features. To validate its effectiveness, we replace MoAM with different interaction modules:\n\n* AM: As shown in Table 3, when replacing MoAM with AM, all metrics undergo a substantial decrease. Similarly, Figure 7 demonstrates that this replacement leads to more blurred textures in both the bear and the snow. These findings underscore the importance of the degradation prior guidance in MoAM for steering restoration experts. \n* Corss-Attention / Zero-Linear: In addition to AM, we also tried other feature interaction modules, including cross-attention and zero-linear. Table 3 shows that their performance is inferior to AM across all metrics. Figure 7 shows that when using zero-linear, the bear's texture becomes blurred and there are semantic errors on its back. When using cross-attention, many artifacts appear, and there are also semantic errors on the bear's nose. Therefore, using AM as the interaction module is more suitable for the IR task to achieve better restoration results.\n\nWe'll add the detailed discussion in the final version.\n\n**Q3: More real-world comparisons**\n\n**[Reply]** Thanks for this valuable suggestion. First, we reanalyze the data from the user study. Specifically, we focuse on the analysis of 33 real-world samples (w/o GTs) out of the original 100 images. The results are provided as follows:\n\n| Method   | SinSR | StableSR  |DiffBIR  |SeeSR  |SUPIR  |DreamClear  |\n| :-------: | :------: | :------: | :------: | :------: | :------: |:------: |\n| Vote Percentage | 8.1%    | 5.5%   | 7.6%   | 11.2%   | 23.1%   | **44.4%**   |\n| Top-1 Ratio | 3.0%    | 0.0%   | 0.0%   | 0.0%   | 15.2%   | **81.8%**   |\n| Top-2 Ratio | 9.1%    | 3.0%   | 9.1%   | 9.1%   | 69.7%   | **100.0%**   |\n\nIt shows the superiority of DreamClear in restoring real-world images. Besides, we also provide more real-world comparisons with SUPIR in the **attached PDF file** (Figure II). It demonstrates that our method can achieve clearer details and fewer artifacts when dealing with real-world cases. We'll add more real-world comparisons in our final paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all reviewers for their valuable and positive comments on our work. This is a **global response**  to reviewers' questions.

**1. Efficiency Comparison**

We provide the efficiency comparison results as follows:

|   | |   |   |
| :-------: | :------: | :------: | :------: |
| **Methods**   | **Params** | **Training Time**  | **Inference Time**  |
| Real-ESRGAN    | 16.7M    | N/A   | 0.08s |
| ResShift    | 173.9M    | N/A   | 1.11s|
| SinSR    | 173.9M    | N/A   | 0.27s|
| StableSR    | 1409.1M  | $\approx$ 7 days using 8 V100 | 12.36s|
| DiffBIR   | 1716.7M    | N/A   | 3.45s|
| SeeSR    | 2283.7M    | N/A   | 4.65s|
| SUPIR   | 3865.6M    | $\approx$ 10 days using 64 A6000  | 16.36s|
| DreamClear  | 2283.7M    | $\approx$ 7 days using 32 A100  | 15.84s|

The inference speed are tested on a single NVIDIA A100 GPU to generate $512 \times 512$ images from $128 \times 128$ inputs. StableSR, DiffBIR, SeeSR, SUPIR, and DreamClear are all built upon the pre-trained T2I model, resulting in a larger number of parameters.

When compared to two recent SOTAs, SeeSR and SUPIR, DreamClear has a similar number of parameters to SeeSR but approximately 1600M fewer parameters than SUPIR. Due to their foundation on MLLM, both DreamClear and SUPIR exhibit slower inference speed than other methods, with DreamClear being about 0.5 seconds faster than SUPIR.

Additionally, SUPIR and DreamClear require more training time and computational resources than other methods due to their use of larger training datasets. Nonetheless, they achieve superior visual results compared to other methods. We'll add the efficiency comparison results in the final version.

**2. More Training Details**

For training GenIR and DreamClear, we both use latent diffusion loss [1], which can be formulated as

$$\mathcal{L} _ {\text{LDM}}=\mathbb{E} _ {z_0,c,t,\epsilon}[|| \epsilon-\epsilon_\theta(\sqrt{\bar{\alpha}_t}z_0+\sqrt[]{1-\bar{\alpha }_t}\epsilon ,c,t)||^2_2],$$

where $\epsilon \in \mathcal{N}(0,\mathbf{{I}} )$ is the ground truth noise map at time step $t$. $c$ represents the conditional information. $\bar{\alpha}_t$ is the diffusion coefficient.

We provide detiled training hyperparameters as follows:
||||
|:-------|:------:|:------:|
|**Configuration**|**GenIR**|**DreamClear**|
|Optimizer|Adafactor|AdamW|
|Optimizer hyper-parameters|$eps_1=10^{-30},eps_2=0.001,decay=-0.8$|$\beta_1=0.9,\beta_2=0.999,eps=10^{-10}$|
|Peak learning rate|$4\times10^{-7}$|$5\times10^{-5}$|
|Learning rate schedule|constant|constant|
|Warmup steps|2000|0|
|Gradient clip|1.0|1.0|
|Global Batch size|256|256|
|Numerical precision|$\text{bfloat16}$|$\text{fp16}$|
|Computational Resources|16 A100 (80GB)|32 A100 (80GB)|
|Training Time|$\approx$ 5 days|$\approx$ 7 days|
|Data Augmentation|random crop, flip|random crop|

**3. More Figures**

We provide some figures for rebuttal in the **attached PDF file**.

**References**

[1] High-Resolution Image Synthesis with Latent Diffusion Models. In CVPR, 2022.

Meta-review: The paper proposes a large scale synthetic dataset for Image restoration and a model that achieves SOTA results using this dataset. Although the overall method appears somewhat over-engineered, all reviewers agreed that the dataset  is valuable especially in the era of large foundational models and 3 of them propose acceptance. The 4th  reviewer did not respond after rebuttal, but it seems that the rebuttal addressed most of the reviewers concerns. A common issue that remains is the computational efficiency of the method.
Decision: Accept (poster)