Title: UIR-LoRA: Achieving Universal Image Restoration through Multiple Low-Rank Adaptation
Venue: NeurIPS 2024
Accepted: no
Keywords: Image Restoration

Transferring Learning
Abstract: Existing unified methods typically treat multi-degradation image restoration as a multi-task learning problem. Despite performing effectively compared to single degradation restoration methods, they overlook the utilization of commonalities and specificities within multi-task restoration, thereby impeding the model's performance. Inspired by the success of deep generative models and fine-tuning techniques, we proposed a universal image restoration framework based on multiple low-rank adapters (LoRA) from multi-domain transfer learning. Our framework leverages the pre-trained generative model as the shared component for multi-degradation restoration and transfers it to specific degradation image restoration tasks using low-rank adaptation. Additionally, we introduce a LoRA composing strategy based on the degradation similarity, which adaptively combines trained LoRAs and enables our model to be applicable for mixed degradation restoration. Extensive experiments on multiple and mixed degradations demonstrate that the proposed universal image restoration method not only achieves higher fidelity and perceptual image quality but also has better generalization ability than other unified image restoration models.

Abstract

Existing unified methods typically treat multi-degradation image restoration as a multi-task learning problem. Despite performing effectively compared to single degradation restoration methods, they overlook the utilization of commonalities and specificities within multi-task restoration, thereby impeding the model's performance. Inspired by the success of deep generative models and fine-tuning techniques, we proposed a universal image restoration framework based on multiple low-rank adapters (LoRA) from multi-domain transfer learning. Our framework leverages the pre-trained generative model as the shared component for multidegradation restoration and transfers it to specific degradation image restoration tasks using low-rank adaptation. Additionally, we introduce a LoRA composing strategy based on the degradation similarity, which adaptively combines trained LoRAs and enables our model to be applicable for mixed degradation restoration. Extensive experiments on multiple and mixed degradations demonstrate that the proposed universal image restoration method not only achieves higher fidelity and perceptual image quality but also has better generalization ability than other unified image restoration models.

1 Introduction

In the wild, a range of distortions commonly appear in captured images, including noise[56], blur[14, 47, 6], low light[58, 22, 8], and various weather degradations[15, 51, 54, 45]. As a fundamental task in low-level vision, image restoration aims to eliminate these distortions and recover sharp details and original scene information from corrupted images. With the assistance of deep learning, an abundance of restoration approaches [56, 3, 54, 2, 16, 14, 53] have made significant progress in eliminating single degradation from images. However, these approaches typically require additional training from scratch on specific image pairs in multi-degraded scenarios, which leads to inconvenience in usage and limited generalization ability.

For simplicity and practicality, some existing works [15, 31, 55]consider training a unified model (also called all-in-one model) to handle multiple degradations as multi-task learning. These studies primarily explore how to discern degradation from the image and integrate it into the restoration network. Nevertheless, these methods share all parameters across different degradations, resulting in gradient conflicts [40, 52] that hinder further improvement of unified models' performance.

Digging deeper, the underlying issue lies in that the similarities among different image restoration 31

tasks and the inherent specificity of each degradation are not well considered and utilized in the 32

training. This limitation drives us to seek solutions for multi-degradation restoration by leveraging 33

both commonalities and specificities. 34

noisy

hazy

low light

…

Pre-trained

generative

model

clean image

domain

specific

specific

specific

specific

𝑝 𝑥

…

shared

transfer

noise, text,

image

Image Generation

Image Restoration

Inspired by the successes of deep generative models[37, 36, 35] and fine-tuning techniques[11, 10, 4], 35

we propose addressing the aforementioned issue from the perspective of multi-domain transfer 36

37

learning, as presented in Figure 1. The pre-trained generative model exhibits powerful capabilities,

implying rich prior knowledge of clear image distribution p ( x ) , which is exactly what is needed 38

39

for image restoration.

Since image prior

p

(

x

)

is degradation-agnostic and applicable to all types

of degraded images, the pre-trained generative model is an excellent candidate for serving as the 40

shared component for multiple degradation restoration. To model the transition from the clean image 41

domain to different degraded image domains, minimal specific parameters are required to fine-tune 42

the pre-trained model for each degradation restoration task. This approach not only isolates conflicts 43

between each degradation task but also ensures efficiency and performance during training. 44

Following the idea of multi-domain transfer learning, we proposed a universal image restoration 45

framework based on multiple low-rank adaptations, named UIR-LoRA. In our framework, the pre46

trained SD-turbo [39] serves as the shared fundamental model for multiple degradation restoration 47

48

tasks due to its powerful one-step generation capability and extensive image priors. Subsequently,

we incorporate the low-rank adaptation (LoRA) technique [11] to fine-tune the base model for each 49

specific image restoration task. This involves augmenting low-dimensional parameter matrices on 50

selected layers within the base model, ensuring efficient fine-tuning while maintaining independence 51

between LoRAs for each specific degradation. Additionally, we propose a LoRA composition strategy 52

based on degradation similarity. We calculate the similarity between degradation features extracted 53

from degraded images and existing degradation types, utilizing it as weights for combining different 54

LoRA experts. This strategy enables our method to be applicable for restoring mixed degradation 55

images. Moreover, we conducted extensive experiments and compared our approach with several 56

57

existing unified image restoration methods. The experimental results demonstrate that our method

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

achieves superior performance in the restoration of various degradations and mixed degradations. Not only does our approach outperform existing methods in terms of distortion and perceptual metrics,

but it also exhibits significant improvements in visual quality.

Our contributions can be summarized as follows:

From the perspective of multi-domain transfer learning, we propose a novel universal image restoration framework based on multiple low-rank adaptations. It leverages the pre-trained generative model as the shared component for multi-degradation restoration and employs distinct LoRAs for multiple degradations to efficiently transfer to specific degradation restoration tasks.

We introduce a LoRAs composition strategy based on the degradation similarity, which adaptively combines trained LoRAs and enables our model to be applicable for mixed degradation restoration.

Through extensive experiments on multiple and mixed degradations, we demonstrate that the proposed universal image restoration method not only achieves higher fidelity and perceptual image quality but also has better generalization ability than other unified models.

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

2 Related Work

2.1 Image Restoration

Specific Degradation Restoration. According to degradation type, image restoration tasks are categorized into different groups, including denoising, deblurring, inpainting, draining .etc . Most existing image restoration methods [2, 53, 16, 56, 5, 14] mainly address the issue with a single degradation. Traditional approaches [27, 28, 7] have proposed image priors. While these priors can be applied to different degraded images, their capability is limited. Due to the remarkable capability of the deep neural network (DNN), numerous DNN-based methods [2, 53, 16] have been proposed to tackle image restoration tasks. While DNN-based methods have made significant progress, they struggle with multiple degradations and mixed degradations, since they typically require retraining from scratch on data with the same degradation.

Universal degradation restoration. Increasing attention is currently focused on developing a unified model to process multiple degradations. For example, AirNet[15] explores the degradation representation in latent space for separating them in the restoration network. PromptIR[31] utilizes a prompt block to extract the degradation-related features to improve the performance. Daclip-IR[20] introduces the clip-based encoder to distinguish the type of degradation and extract the semantics information from distorted images and embed them into a diffusion model to generate high-quality images. Despite the advancements, these unified models still have limitations. They also require retraining all parameters when unseen degradations arrive and have limited performance due to the gradient conflict.

2.2 Low-Rank Adaptation

LoRA [11] is proposed to fine-tune large models by freezing the pre-trained weights and introducing trainable low-rank matrices. This fine-tuning method leverages the property of "intrinsic dimension" within neural networks, lowering the rank of additional matrices and making the re-training process efficient. Concretely, given a weight matrices W ∈ R n × m in pre-trained model θ p , two trainable matrices B ∈ R n × r and A ∈ R r × m are inserted into the layer to represent the LoRA ∆ W = BA , where r is the rank and satisfy r ≪ mim ( n, m ) , the updated weights W ′ are calculated by

By applying LoRA in pre-trained models, numerous image generation methods [29, 13], show superior performance in the field of image style and semantics concept transferring. Additionally, fine-tuning methods like ControlNet [57], T2i-adapter [24] are also commonly employed in largescale pre-trained generative models such as Stable Diffusion [37], SDXL [30], and Imagen [38].

2.3 Mixture of Experts

Mixture of Experts (MoE) [41, 49, 48] is an effective approach to scale up neural network capacity to improve performance. Specifically, MoE integrates multiple feed-forward networks into a transformer block, where each feed-forward network is regarded as an expert. A gating function is introduced to model the probability distribution across all experts in the MoE layer. The gating function is trainable and determines the activation of specific experts within the MoE layer based on top-k values. Broadly speaking, our framework aligns with the concept of MoE. However, unlike traditional MoE layers, we employ the more efficient LoRA as experts in selected frozen layers and utilize a degradation-aware router across all selected layers to uniformly activate experts, reducing learning complexity and avoiding conflicts among different image restoration tasks on experts.

3 Methodology

3.1 Problem Definition

This paper seeks to develop a novel universal image restoration framework capable of handling diverse forms of image degradation in the wild by fine-tuning the pre-trained generative model. Consider a set of T image restoration tasks D = { D k } T k =1 , where D k = { ( x i , y i ) } n k i =1 is the training dataset containing n k images pairs of the k -th image degradation task. Within the set of tasks D ,

Pre-trained Weights

…

🔥

Trainable LoRAs

hazy

…

noisy

∑

…

Image

encoder

∆𝑊

!

∆𝑊

"

∆𝑊

#

Degraded Image

Restored Image

weights

…

𝑓

!

𝑥

𝑓

"

𝑥

𝑓

#

𝑥

𝑓

$

𝑥

&

%&#

'

𝑠

%

𝑓

%

𝑥

𝑓

$

𝑥

+

Degradation

similarity

𝑠

!

= 𝑑𝐵

×

𝑑

𝐵

…

𝑥

!"

𝑥

#$%

Text

encoder

Degradation-Aware

Router

Universal

Image Restorer

hazy, blurry,

…, noisy

…

𝑠 = 𝑛𝑜𝑟𝑚

"

𝑠

!

𝑡𝑜𝑝𝐾

each task D k only has a specific type of image degradation, with no intersection between any two 120 tasks. Given a pre-trained generative model θ p with frozen parameters, our objective is to learn a 121 set of composite { θ k } T k =1 to construct a unified model f θ that performs well on multi-degradation 122 restoration and mixed degradation restoration by transferring learning, where θ = θ p + ∑ T k =1 s k θ k 123 and s k represents the composite weight for θ k . The trainable { θ k } T k =1 can be optimized through 124 minimizing the overall image reconstruction loss: 125

We will present how to design and optimize the trainable { θ k } T k =1 and construct the composite 126 weights s in the next sections. 127

128

3.2 Overview of Universal Framework

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

3.3 Degradation-Aware Router

The Degradation-Aware Router is designed to provide the restorer with weights for LoRA combination based on degradation confidence. Following Daclip-ir [20], we utilized the pre-trained image encoder in CLIP [35] to obtain the degradation vector d ∈ R 1 × z from the input degraded image x , where z is degradation length in latent space. Differing from Daclip-ir [20], we use the degradation vector and existing degradations to calculate the similarity, instead of directly embedding the degradation vector into the restoration network in Daclip-ir [20]. The existing degradations refer to the vocabulary bank of diverse degradation types that we introduce in the router, such as "noisy", "blurry" and "shadowed". This vocabulary bank is highly compact and flexible when adding new degradation types. Similarly, by applying the text encoder of CLIP [35], the vocabulary bank can be encoded into the degradation bank B ∈ R z × T in the latent space. As presented in Figure 2, the original degradation similarity s o ∈ R 1 × T is calculated by:

Building upon the original similarity, we adopt a more flexible and controllable Top-K strategy to modify s o . Specifically, we select the Top-K largest values from the original similarity s o , and normalize them to reallocate the weights for LoRAs. The reallocation process can be formulated as :

where M K represents a binary mask with the same length as s o , where it is 1 when the corresponding value in s o is among the Top-K, otherwise it is 0. With a smaller value of K , the restorer activates fewer LoRAs, reducing its computational load. For instance, with K = 1 , only the most similar LoRA is activated and it yields effective results when s is accurate, but performance noticeably declines with inaccurate s . Conversely, as K increases, the restorer exhibits higher tolerance to s and the combination of LoRAs allows it to handle mixed degradation.

3.4 Universal Image Restorer

Our universal image restorer consists of a pre-trained generative model θ p and a set of LoRAs { θ k } T k =1 . As illustrated in Figure 2, our universal image restorer takes the degraded image x and similarity s predicted by the degradation-aware router as inputs. It then activates relevant LoRAs based on s to recover the degraded image along with the frozen base model. Since one of our objectives is to ensure that each LoRA serves as an expert in processing a specific degradation, the number of LoRAs in the restorer aligns with the number of degradation types, T . In practice, we select multiple layers from the base model, For a selected layer W of the pre-trained base model, a sequence of trainable matrices { ∆ W k } T k =1 are added into this layer, and the parameters of all chosen layers L form a complete LoRA θ k = { ∆ W j k } j ∈ L . As previously explained, each LoRA is a unique expert responsible for a specific degradation. Drawing inspiration from Mixture of Expert (MoE), we aggregate the outputs of each expert rather than directly merging parameters in [11]. Therefore, given the input feature x in of the current layer and the similarity s , the total output x out of this modified layer can be expressed as

where f i ( x in ) denotes the result of i -th trainable matrice W i , particularly f o ( x in ) is output of the 179 frozen base layer. From the equation 5, it can be observed that the introduced LoRAs interact with the 180 frozen base model at intermediate feature layers in our restorer. This interaction forces the restorer 181 to leverage the image priors of the pre-trained generative model and eliminate degradation with the 182 assistance of LoRAs. In contrast to employing stable diffusion [37] directly as a post-processing 183 technique, our restorer yields results closer to the true scene without introducing inaccurate structural 184 details. Since each W is implemented using two low-rank matrices like the formula 1, the total 185 trainable parameters of our framework are much smaller than that of the pre-train generative model. 186

187

3.5 Training and Inference Procedure

During the training phase, for the efficient training of the universal image restorer, we ensure that 188 each batch is sampled from the same degradation type D k , and activate the corresponding LoRA θ k 189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

for training. Since the dataset D is organized by degradation type without overlap and each LoRA is assigned to handle each type of degradation correspondingly, the overall optimization process in equation 2 can be decomposed into independent optimization processes for each degradation. This design and training process circumvent task conflicts among multiple degradations and makes it possible to use suitable loss functions for the specific degradation. Due to the availability of accurate s during training and the use of pre-trained encoders from CLIP [35] and Daclip-ir [20] in our router, the router was not utilized during training.

In the inference phase, the similarity s is unknown and needs to be estimated from the degraded image. The estimated similarity s serves as a reference in our framework and can also be manually specified by users. Subsequently, our universal image restorer composite LoRAs and recovers the input image with the guidance of s .

4 Experiments

4.1 Experimental Setting

Datasets. Wevalidate the effectiveness of our framework in multiple and mixed degradation scenarios. In the case of multiple degradations, we follow Daclip-IR [20] and construct a dataset using 10 different single degradation datasets. Briefly, the composite dataset comprises a total of 52800 image pairs for training and 2490 image pairs for testing. The degradation types included are commonly encountered in image restoration, such as blur, noise, shadow, JPEG compression, and weather degradations. For mixed degradations, we utilize two degradation datasets, REDS [25] and LOLBlur [58]. In REDS, the images are distorted by JPEG compression and blur, and those images in LOLBlur have blur and low light. For more details about datasets in our experiments, please refer to Appendix .

Metrics. The objective of the image restoration task is to output images with enhanced visual quality while maintaining high fidelity to the original scene information. This differs from image generation tasks, which prioritize visual quality. Therefore, to thoroughly evaluate the effectiveness of our method, we utilize reference-based image quality assessment techniques from both distortion and perceptual perspectives, including PSNR, SSIM, and LPIPS, as well as FID.

Comparison Methods. In the experiments, we primarily compare with several state-of-the-art methods in image restoration, which fall into two categories: regression model and generative model. Regression models include NAFNet [2], Restormer [53], as well as AirNet [15] and PromptIR [31] proposed for multiple degradation restoration. DiffBIR [17], IR-SDE [21] and Daclip-IR [20] are generative models built upon the diffusion model [9].

4.2 Implementation Details

During the training, we adapt an AdamW optimizer to update the weights of trainable parameters in our model. Before training LoRA for specific degradation, we add skip-connections in the VAE of SD-turbo[39] like [29, 44] and train them with multiple degraded images. We set the initialization

JPEG

GT

Restormer

PromptIR

Daclip-IR

Ours

Noisy

GT

Restormer

PromptIR

Daclip-IR

Ours

Raindrop

GT

Restormer

PromptIR

Daclip-IR

Ours

Inpainting

GT

Restormer

PromptIR

Daclip-IR

Ours

learning rate to 2e-4 and decrease it with CosineAnnealingLR . We trained every LoRA for 80K 225 iterations with batch size 8 and we keep the same hyper-parameters when training different LoRAs. 226 The default rank of LoRAs in VAE and Unet is 4 and 8, respectively. 227

228

229

230

231

232

233

234

235

236

237

238

239

4.3 Multiple Image Restoration

For fair comparisons, all methods are trained and tested on the multiple degradation dataset. The results are presented in Table 1. We can find that our model, UIR-LoRA, considerably surpasses all compared image restoration approaches across four metrics. This indicates that our approach can balance generating clear structures and details while ensuring the restored images closely resemble the original information of the scene. The visual comparison results depicted in Figure 7 also confirm this assertion. Regression models such as NAFNet [2]and Restormer [53], lacking extensive image priors, tend to produce blurred and over-smoothed images, leading to inferior visual outcomes. Conversely, generative models Daclip-IR [20] excessively prioritize perceptual quality, yielding artifacts and noise that diverge from the actual scene information. Our approach integrates the strengths of both categories of methods, enabling strong performance in both distortion and perceptual aspects

4.4 Mixed Image Restoration

To evaluate the transferability of UIR-LoRA, we conduct some experiments on mixed degradation 240 datasets from REDS[25] and LOLBlur [58]. Each image in these two datasets contains more than one 241 type of degradation, like blur, jpeg compression, noise, and low light. We test the mixed degraded 242 images using models trained on multiple degradations and set K to 2 in the router. As shown in 243 Table 2, our method achieves superior results in both distortion and perceptual quality, particularly 244 on the LOLBlur dataset. We also provide visual comparison results, as illustrated in Figure 4, our 245 approach effectively enhances the low-light image compared to SOTA methods, highlighting its 246 stronger transferability in the wild. More visual results can be found in Appendix . 247

248

249

250

251

252

253

254

255

256

257

258

259

Degraded image

GT

Restormer

PromptIR

Daclip-IR

Ours

4.5 Ablation Study

Complexity Analysis. We compare model complexity with SOTA models. The comparison results are shown in Table 1, where we report the number of trainable parameters and the runtime for a 256 × 256 image on an A100 GPU. The complexity of UIR-LoRA is comparable to regression models like NAFNet [2] and significantly more efficient than generative models like Daclip-IR [20].

Effectiveness of Degradation-Aware Router. The degradation-aware router plays a crucial role in determining which LoRAs are activated in the inference. To comprehensively demonstrate the impact of the router, we conduct experiments with different selection strategies. As illustrated in Table 3, we have five strategies: "random" indicates activating a LoRA at random, "average" denotes using average weights to activate all LoRAs, and "Top-1", "Top-2" and "All" correspond to setting K in the router to 1,2, and 10, respectively. From the comparison of these results, we can see that the random and average strategies result in poorer performance while using the strategy based on degradation

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

28.0

0.86

0.18

60

Deblurring

Deblurring

27.5

0.84

0.17

Denoising

50

Denoising

PSNR

27.0

SIM

0.80

0.82

PIPS

0.16

30

40

26.5

Deblurring

0.78

Deblurring

0.15

20

Denoising

Denoising

26.0

0.76

0.14

10

0

4

8

12

16

0

4

8

12

16

0

4

8

12

16

0

4

8

12

16

similarity achieves better outcomes. This suggests that the transferability between different types of degradation is limited and that specific parameters are needed to address their particularities. Furthermore, the selection of the K value also affects the model's performance. When an image has only one type of degradation, a smaller K value can result in comparable performance with lower inference costs. However, for mixed degradations, a larger K value is required to handle the more complex situation.

Impact of LoRA's Rank. Within our framework, LoRA is utilized to facilitate the transfer from the pre-trained generative model to the image restoration task. In order to investigate the impact of LoRA's rank on the performance of image restoration, we conduct experiments using deblurring and denoising tasks chosen from ten distinct degradation categories. We set the initial rank to 2 and incrementally increase the value by a factor of 2. The performance changes are depicted in Figure 5. It is evident that as the rank grows, the restoration results improve in distortion and perceptual quality, and at the same time, the number of trainable parameters also increases. Once the rank value exceeds 4, the performance improvement becomes progressively marginal. Therefore, we set the default rank to 4 in our restorer to balance between performance and complexity.

Impact of Predicted Degradation.

The resizing operation on input images in CLIP models [20, 35] may lead to inaccurate predictions of degradation types, especially for blurry images. To reduce its negative impact on performance, we introduce a simple way that uses the degradation vector of the image crop without resizing to correct the potential error in the resized image. Table 4 is the comparison conducted on blurry images from GoPro dataset. It can be observed that our model with modified operation has higher accuracy and better performance for deblurring.

5 Conclusion

In this paper, we propose a universal image restoration framework based on multiple low-rank adaptation, named UIR-LoRA, from the perspective of multi-domain transfer learning. UIR-LoRA utilizes a pre-trained generative model as the frozen base model and transfers its abundant image priors to different image restoration tasks using the LoRA technique. Moreover, we introduce a LoRAs' composition strategy based on the degradation similarity that allows UIR-LoRA applicable for multiple and mixed degradations in the wild. Extensive experiments on universal image restoration tasks demonstrate the effectiveness and better generalization capability of our proposed UIR-LoRA.

6 Limitation and Discussion

Although our UIR-LoRA has achieved remarkable performance in image restoration tasks under both multiple and mixed degradations, it still has limitations and problems for further exploration. For instance, adding new trainable parameters into the network for unseen degradations is unavoidable in image restoration tasks, although UIR-LoRA is already more efficient and flexible compared to other approaches.

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

References

Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops , pages 126-135, 2017.

Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In European conference on computer vision , pages 17-33. Springer, 2022.

Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen. Hinet: Half instance normalization network for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 182-192, 2021.

Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems , 35:16664-16678, 2022.

Xueyang Fu, Jiabin Huang, Xinghao Ding, Yinghao Liao, and John Paisley. Clearing the skies: A deep network architecture for single-image rain removal. IEEE Transactions on Image Processing , 26(6):2944-2956, 2017.

Dong Gong, Jie Yang, Lingqiao Liu, Yanning Zhang, Ian Reid, Chunhua Shen, Anton Van Den Hengel, and Qinfeng Shi. From motion blur to motion flow: A deep learning solution for removing heterogeneous motion blur. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2319-2328, 2017.

Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. Weighted nuclear norm minimization with application to image denoising. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2862-2869, 2014.

Chunle Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong. Zero-reference deep curve estimation for low-light image enhancement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1780-1789, 2020.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning , pages 2790-2799. PMLR, 2019.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022.

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 , 2017.

Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multiconcept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1931-1941, 2023.

Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiˇ rí Matas. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8183-8192, 2018.

Boyun Li, Xiao Liu, Peng Hu, Zhongqin Wu, Jiancheng Lv, and Xi Peng. All-in-one image restoration for unknown corruption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17452-17462, 2022.

Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1833-1844, 2021.

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Ben Fei, Bo Dai, Wanli Ouyang, Yu Qiao, and Chao Dong. Diffbir: Towards blind image restoration with generative diffusion prior. arXiv preprint arXiv:2308.15070 , 2023.

Yun-Fu Liu, Da-Wei Jaw, Shih-Chia Huang, and Jenq-Neng Hwang. Desnownet: Context-aware deep network for snow removal. IEEE Transactions on Image Processing , 27(6):3064-3073, 2018.

Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11461-11471, 2022.

Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Controlling vision-language models for universal image restoration. arXiv preprint arXiv:2310.01018 , 2023.

Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Image restoration with mean-reverting stochastic differential equations. International Conference on Machine Learning , 2023.

Long Ma, Tengyu Ma, Risheng Liu, Xin Fan, and Zhongxuan Luo. Toward fast, flexible, and robust low-light image enhancement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5637-5646, 2022.

David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 416-423, 2001.

Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 4296-4304, 2024.

Seungjun Nah, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee. Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 0-0, 2019.

Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3883-3891, 2017.

Jinshan Pan, Zhe Hu, Zhixun Su, and Ming-Hsuan Yang. Deblurring text images via l0regularized intensity and gradient prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2901-2908, 2014.

Jinshan Pan, Deqing Sun, Hanspeter Pfister, and Ming-Hsuan Yang. Blind image deblurring using dark channel prior. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1628-1636, 2016.

Gaurav Parmar, Taesung Park, Srinivasa Narasimhan, and Jun-Yan Zhu. One-step image translation with text-to-image models. arXiv preprint arXiv:2403.12036 , 2024.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 , 2023.

Vaishnav Potlapalli, Syed Waqas Zamir, Salman Khan, and Fahad Khan. Promptir: Prompting for all-in-one image restoration. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

Rui Qian, Robby T Tan, Wenhan Yang, Jiajun Su, and Jiaying Liu. Attentive generative adversarial network for raindrop removal from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2482-2491, 2018.

Xu Qin, Zhilin Wang, Yuanchao Bai, Xiaodong Xie, and Huizhu Jia. Ffa-net: Feature fusion attention network for single image dehazing. In Proceedings of the AAAI conference on artificial intelligence , pages 11908-11915, 2020.

Liangqiong Qu, Jiandong Tian, Shengfeng He, Yandong Tang, and Rynson WH Lau. Deshadownet: A multi-context embedding deep network for shadow removal. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4067-4075, 2017.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning , pages 8821-8831. Pmlr, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems , 35:36479-36494, 2022.

Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.17042 , 2023.

Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems , 31, 2018.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 , 2017.

H Sheikh. Live image quality assessment database release 2. http://live.ece.utexas. edu/research/quality , 2005.

Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, and Lei Zhang. Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops , pages 114-125, 2017.

Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. arXiv preprint arXiv:2305.07015 , 2023.

Yinglong Wang, Chao Ma, and Jianzhuang Liu. Smartassign: Learning a smart knowledge assignment strategy for deraining and desnowing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3677-3686, 2023.

Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv:1808.04560 , 2018.

Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16293-16303, 2022.

Xun Wu, Shaohan Huang, and Furu Wei. Mole: Mixture of lora experts. In The Twelfth International Conference on Learning Representations , 2023.

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

Yuan Xie, Shaohan Huang, Tianyu Chen, and Furu Wei. Moec: Mixture of expert clusters. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 13807-13815, 2023.

Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1357-1366, 2017.

Wenhan Yang, Robby T Tan, Shiqi Wang, Yuming Fang, and Jiaying Liu. Single image deraining: From model-based to data-driven and beyond. IEEE Transactions on pattern analysis and machine intelligence , 43(11):4059-4077, 2020.

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems , 33:5824-5836, 2020.

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5728-5739, 2022.

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, MingHsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14821-14831, 2021.

Cheng Zhang, Yu Zhu, Qingsen Yan, Jinqiu Sun, and Yanning Zhang. All-in-one multidegradation image restoration network via hierarchical degradation representation. In Proceedings of the 31st ACM International Conference on Multimedia , pages 2285-2293, 2023.

Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing , 26(7):3142-3155, 2017.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3836-3847, 2023.

Shangchen Zhou, Chongyi Li, and Chen Change Loy. Lednet: Joint low-light enhancement and deblurring in the dark. In European conference on computer vision , pages 573-589. Springer, 2022.

A Appendix 470

A.1 More Details about Datasets 471

For multiple degradations, we follow Daclip-IR [20] to construct the dataset, which includes a total 472 of ten distinct degradation types: blurry, hazy, JPEG-compression, low-light, noisy, raindrop, rainy, 473 shadowed, snowy, and inpainting. The data sources and data splits for each degradation type are 474 illustrated in Table 5. 475

For mixed degradations, we utilize images from REDS[25] and LOLBlur[58]to evaluate the trans476 ferability of models. We sample 60 images from REDS and 200 images from LOLBlur dataset for 477 testing. The degraded images from REDS dataset feature a variety of realistic scenes and objects, 478 which suffer from both motion blurs and compression. And the images from LOLBlur dataset cover 479 a range of real-world dynamic dark scenarios with mixed degradation of low light and blurs. 480

A.2 More Visual Results 481

Input

GT

Restormer

PromptIR

Daclip-IR

Ours

VERSAN

VERSAN

VERSAN

NVERSA

IVERSA

NVERSAN

A.3 Details about Metrics on Multiple Dagradation 482

Input

GT

Restormer

PromptIR

Daclip-IR

Ours

Input

GT

Restormer

PromptIR

Daclip-IR

Ours

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction1 accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper does discuss the limitations of the work in Sections 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper provides the full set of assumptions and a complete (and correct) proof in Sections 3.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper provides a comprehensive description of the experimental setting in Sections 4.1 and implementation details in Sections 4.2, which are crucial for reproducing the main results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Upon acceptance of the paper, we will release the code under an open-source license, which will allow the community to access and verify the experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper provides detailed information on all training and test aspects, including datasets, metrics, comparison methods, hyperparameters, the type of optimizer used, and other relevant details necessary to understand the results in Sections 4.1 and 4.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports error bars appropriately and includes correctly defined information regarding the statistical significance of the experiments, ensuring the transparency and reliability of the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides detailed information regarding the computer resources used in Sections 4.1, and time of execution in Table 1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper is in full compliance with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper provides both the potential benefits and the risks associated with the research, ensuring a comprehensive assessment of its societal implications.

Guidelines: 691

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not introduce assets that carry a high risk for misuse, therefore, no specific safeguards for data or model release are required.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper meticulously cites all external assets in references, including code, dataset, and models, acknowledging the contributions of their creators and respecting the associated licenses and terms of use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794

795

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The novel universal image restoration framework introduced in the paper is well documented, and the documentation is provided alongside the model in Sections 3, offering comprehensive details for replication and application.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not engage in crowdsourcing experiments or research with human subjects therefore, it does not include participant instructions, screenshots, or details about compensation.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve research with human subjects, so there are no participant risks to disclose, and no Institutional Review Board (IRB) approvals or equivalent reviews were required.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a universal image restoration framework UIR-LoRA based on multiple low-rank adapters. UIR-LoRA employs the pre-trained text-to-image diffusion model SD-turbo as the shared component. It utilizes a LoRA composing strategy based on the degradation similarity predicted by CLIP encoder to combine different LoRA modules. Experiments show the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The proposed LoRA-based Universal IR method is easy to understand and follow.\n2. The motivation of this paper is very clear to me.\n\nWEAKNESSES:\n1. UIR-LoRA adopts SD-turbo as the pre-trained backbone for image restoration. However, SD-tubo utilizes VAE with high compression rate to encode input images, resulting in severe detail distortion for image restoration. This issue has been widely discussed in recent published works [1,2]. However, the paper ignores this very important issue in the Method Section and only mentions the skip-connections for VAE in Line 223.\n2. The degradation-aware router seems to be unreliable. I do not believe that the original pre-trained CLIP Text Encoder can distinguish between different degradations through degraded text representations, such as \"rain\" and \"raindrop\". Therefore, DA-CLIP fine-tunes the original CLIP. But this paper doesn't contain any discussions about this.\n3. This paper does not provide complete technical details, such as how the LQ image is used as a condition for SD-turbo. Is ControlNet used, or is it directly concatenated? I do not see any information about this in the paper. \n4. Tab. 1 only reports the trainable Param for UIR-LoRA. I think it's necessary to report the overall Param of the model. In addition, the reported PSNR for DiffBIR is very low. Did the authors add skip-connections to the VAE of DiffBIR for a fair comparison?\n5. The visual results in Fig. 3 seem strange. The visual results of Restormer show noticeable artifacts between patches. Do the authors test Restormer using a tiled mode? As far as I know, using a single A100 GPU (Line 251), Restormer can restore the entire image without encountering out-of-memory issues.\n\n[1] Wang, Wenjing, et al. \"Zero-Reference Low-Light Enhancement via Physical Quadruple Priors.\" In CVPR, 2024.\n\n[2] Geng, Zigang, et al. \"Instructdiffusion: A generalist modeling interface for vision tasks.\" In CVPR, 2024.\n\nQUESTIONS:\n1. Authors should discuss the skip-connections for VAE in the Method Section with more details.\n2. Can authors provide the degradation prediction accuracy for more different predictions (eg, rain/raindrop)?\n3. Authors should provide more technical details of the proposed method.\n4. More experimental results and explanations should be included.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to perform universal image restoration via multiple low-rank adaptation. The key idea is to leverage a pre-trained stable diffusion model as the shared component and transfer it to specific degradations with LoRA adaptation. A degradation-aware router is further proposed to generate weights for LoRA combination based on degradation confidence. In experiments, the authors evaluated their method on multi-degradation and mixed-degradation datasets and conducted several ablation experiments on their core components.\n\nSTRENGTHS:\n- The idea of applying LoRA to a pre-trained SD for multi-task image restoration is promising and interesting.\n- The overall presentation is easy to follow.\n- The experimental results are good and the ablation studies make sense.\n\nWEAKNESSES:\n- ControlNet is the most popular approach to adapting SD models to other tasks. I'm curious why the authors chose LoRA? As far as I know, LoRA is often used for large language models (with billions of parameters). It would be great to provide more detailed motivation in the introduction.\n- In line 123, maybe it's better to use \"concatenate\" or other operators instead of \"add\" to present the unified parameters. Here, the weight $s_k$ can be ignored.\n- Can the authors use other SD models as the base model? I believe applying LoRA to a multi-step diffusion process can further illustrate its efficiency.\n- In Eq. (4), $s_0 \\cdot M_k$ is used in both numerator and denominator, which seems weird and confusing.\n- The mixed degradation experiment is cool. It would be interesting if the authors could apply their model to real-world degraded images.\n- Line 45: proposed -> propose\n\nQUESTIONS:\nIn the degradation-aware router, have you finetuned the CLIP to align degraded images with correct degradation names? How do you choose the degradation names as the vocabulary bank?\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission proposes a transfer-learning based strategy to address challenges related to image-degradation restoration. The premise is that a pre-trained generative model can be employed as a common starting component for multiple degradation types, upon which distinct sets of trainable parameters (ie. low-rank adaptors) can be added in order to address specific-degradation restoration tasks. Mixed-degradation restoration is enabled through a top-K hyperparameter, that affords a mixture of (degradation) experts to be active. The experimental setup considers multi and mixed image restoration problems where average results are offered across image-degradation datasets and appropriate standard quantitative metrics, qualitative examples, are reported in comparison with alternative approaches.\n\nSTRENGTHS:\n* The technique described for piping specific samples down specific low-rank adaptor chutes is relatively easy to understand and yet reportedly results in competitive restoration accuracy for investigated datasets. \n\n* Nascent investigations into mixed-degradation image restoration problems provide a promising seed to be followed.\n\n* The writing is of a reasonable standard.\n\nWEAKNESSES:\n* The key idea of leveraging pretrained VLM features (and specifically CLIP) for the task of image restoration from multiple degradations, pre-dates the current submission [R1]. While authors clearly go to some length to highlight their alternative CLIP-based scheme, which amounts to envoking specific (pre-existing [R2]) low-rank adaptors, the core technical contributions here can be regarded as somewhat limited.  \n\n* The phrase 'Universal Image Restoration' may not be a sufficiently accurate (or modest) description for the proposed method. The submission collates ten different image restoration tasks which, despite vague statements in the abstract, remains a 'multi-task' not a 'universal' setup. Samples for all ten degradation tasks are shared between train and test (Sec. A.1) and individual task adaptors appear to be trained independently on task-specific datasets (L188--196). Generalisation ability to previously unseen degradations is also not considered. Suggest method description requires reworking.\n\n* The claim that multi-task learning (MTL) frameworks, designed to handle image restoration for multiple degradations, share all parameters across different degradations (L029) is incomplete and somewhat misleading. Several existing MTL works (eg. [R3,R4]) make use of both shared and task-specific parameter subsets for multiple image restoration tasks. Indeed 'which proportion of parameters should be shared and which should be task specific' can be considered a fundamental (and long standing) MTL question. The idea of benefiting from commonalities between image restoration tasks is well understood and my concern is that this casts doubt on a core premise of the submission. \n\n\nReferences\n\nR1. Controlling Vision-Language Models for Multi-Task Image Restoration. ICLR 2024.\n\nR2. LoRA: Low-rank adaptation of large language models. ICLR 2022.\n\nR3. All in One Bad Weather Removal using Architectural Search. CVPR 2020.\n\nR4. Pre-Trained Image Processing Transformer. CVPR 2021.\n\nMinor:\n\nL076: 'draining' --> 'deraining'\n\nL099: 'mim' --> 'min'\n\nL238: 'aspects' --> 'aspects.'\n\nQUESTIONS:\n> 'for mixed degradations, a larger K value is required to handle the more complex situtation' (L264). \n\nCan additional results be provided for alternative hyperparameter settings (eg. K=1 and K=10) in Tab.2, towards evidencing this claim?\n\nLIMITATIONS:\nHalf of one sentence (L293) is apportioned to discussing method limitations. See above for suggestions on components that might make for valid additions here.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes universal image restoration framework using multiple low-rank adapters that learns task specific weights from to perform multi-domain transfer learning. the proposed method leverages the pre-trained generative model weights as the shared component and adapts it task specific low-rank adapters. At each layer in the restoration pipeline the proposed method uses the degradation similarity to combine LoRA adapters outputs, this enables the proposed to handle for mixed degradation restoration.\n\nSTRENGTHS:\n- The paper proposes LoRA adapters to learn task specific weights and proposes a strategy to combine the adapter outputs using degradation similarity measure\n- extensive experiments are performed showing the proposed strategy works better than random and average in table 3.\n- extensive experiments are performed to show the proposed methods performance against the sota methods in table 1 for mutliple degradation task.\n- Extensive experiments are performaed showing impact of LoRA rank and prediction accuracy\n\nWEAKNESSES:\n- In table of the paper authors compared proposed method against sota on REDS and LOLBlur datasets, both these datasets have mixed degradations  of blur, jpeg compression, noise, and low light. Although these comparisons performed on mixed degradations, it would be helpful to how the proposed method performs on mixed weather conditioned images (MID6), which is comparatively challenging than REDS and LOLBlur  datasets. \nMID6: Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restoration, CVPR, 2024.\n\n- Can authors confirm, whether network re-trained seperately for each experiment in table-1,  and table-2 separately, i.e. table-1 and table-2 trained network weights for proposed method are different.\n\nQUESTIONS:\n- Can the proposed network handle unknown degradation present in the input degradation image\n- From the table-3, it is evident that Top-1, and top-2 are almost same performance as All, can authors coment on this, this makes wonder if the input image has only one degradation as dominating for this experiment. Can authors show this experiment on different dataset like MID6\n\nLIMITATIONS:\n- authors have addresed limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

comments: SUMMARY:
This paper presents a framework to improve image restoration across various degradation types using Low-Rank Adapters (LoRA). The proposed method adapts a pre-trained generative model to each degradation type. It performs a weighted sum of the output of adapted models using the estimated degradation of input images. The proposed method performs impressive results in restoration accuracy and resources.

STRENGTHS:
The proposed method is interesting and reasonable.
Experimental results support this paper's contributions and the proposed method's effectiveness.

WEAKNESSES:
In Table 3, the 'Top-1' strategy performs almost the same as the 'All' strategy, which limits the motivation of the weighted sum of the adapted models.
Table 6 presents the restoration performance comparisons for each degradation. The proposed method underperforms previous works in significant degradation types such as blurry, low-light, raindrop, and rainy.
The average scores might mislead the evaluation performances.

QUESTIONS:
How about comparing the proposed method with the following paper?
Selective Hourglass Mapping for Universal Image Restoration Based on Diffusion Model, CVPR 2024

LIMITATIONS:
The proposed method is simple and effective, but evaluating average scores on multiple degradations can mislead its contribution.
The proposed method achieves near-best performance by selecting a single adapted model but underperforms in many major degradation types.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-16