Controlling Vision-Language Models for Multi-Task Image Restoration
Authors: Ziwei Luo
Fredrik K. Gustafsson
Zheng Zhao
Jens Sjölund
Thomas B. Schön
Venue: ICLR 2024 poster (2024)
Keywords: Image restoration
vision-language model
low-level vision
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Controlling vision-language models to understand image degradation and improve image restoration.

Abstract: Vision-language models such as CLIP have shown great impact on diverse downstream tasks for zero-shot or label-free predictions. However, when it comes to low-level vision such as image restoration their performance deteriorates dramatically due to corrupted inputs. In this paper, we present a degradation-aware vision-language model (DA-CLIP) to better transfer pretrained vision-language models to low-level vision tasks as a multi-task framework for image restoration. More specifically, DA-CLIP trains an additional controller that adapts the fixed CLIP image encoder to predict high-quality feature embeddings. By integrating the embedding into an image restoration network via cross-attention, we are able to pilot the model to learn a high-fidelity image reconstruction. The controller itself will also output a degradation feature that matches the real corruptions of the input, yielding a natural classifier for different degradation types. In addition, we construct a mixed degradation dataset with synthetic captions for DA-CLIP training. Our approach advances state-of-the-art performance on both degradation-specific and unified image restoration tasks, showing a promising direction of prompting image restoration with large-scale pretrained vision-language models. Our code is available at https://github.com/Algolzw/daclip-uir.

Published as a conference paper at ICLR 2024
CONTROLLING VISION-LANGUAG MODELS FOR MULTI-TASK IMAGE RESTORATION
Ziwei Luo, Fredrik K. Gustafsson, Zheng Zhao, Jens Sjölund, Thomas B. Schön Department of Information Technology. Uppsala University
(ziwei luo, fredrik gusta sssn, zheng. zhao } @@it. ue (jens. s llndd thomas schon 8 it. uu.se
ABSTRACT
Vision-language models such as CLIP have shown great impact on diverse down- stream tasks for zero-shot or label-free predictions. However, when it comes low-level vision such as image restoration their performance deteriorates dramati- cally due corrupted inputs. In this paper, we present a degradation-ane -waree vision- language model (DA-CLIP) to better transfer pretrained Aision-lanuuage models to low-level vision tasks as a multi-task framework for image restoration. More specifically, DA-CLIP trains additional controller that adapts the fixed CLIP image encoder to predict high-quality feature embeddings. By integrating the em- bedding into image restoration network via cross-attention, the model to learn a high-fidelity image reconstruction. The controller itself will also output degradation feature that matches the real corruptions of the input, yielding a natural classifier for different degradation types. In addition, we con- struct mixed degradation dataset with synthetic captions for DA-CLIP training. Our approach advances satte-of-the-ee performance on both settaaddttionsppcccie and unified image restoration tasks, showing promising direction of prompting image restoration with large-scale pretrained vision-language models. Our code is available https: / htt//ggtttt thw..b..ttt......o com/ A1 golgollle gwwzw///w///ooooo wwwoo///dwclliiioooooo ip-uur.
are able to pilot
INTRODUCTION
Image debluring
Image dehazing
Face inpainting
Accurncy
biutty
Inpaiiting
sanwy
JPEG artifoct deshuction
Image desnowing ingg
tow-ltgh
Shadowed
DA-LUP
naley
raiyy
raincrop
Low-light image enhancement
Degradation prediction
Image shadow removal
denoising
Image derabning
Image raindrop removal
Figure 1: This paper leverages large-scale pretrained vision-language models for multi-task image restoration. Compared to CLIP (Radford et al., 2021), approach precisely predicts the degrada- tion embeddings for corrupted inputs and also outputs high-quality features for better image restora- tion performance. For all examples above, the right results produced by our single unified model.

---

Published as a conference paper at ICLR 2024
Large-scale pretrained vision-languue models (VLMs) such CLIP (Radford et al., 2021) have recently garnered significant attention, in part because of their wide-reaching usefulness on many fundamental computer vision tasks (Gu et al., 2021; Zhang et al., 2022; 2023). However, existing VLMs have so far had limited impact on low-level vision tasks such as image restoration (IR), presumably because they do not capture the fine-grained difference between image degradation types such as "blurry" and "noisy" (Ni et al., 2023). Consequently, the existing VLMs often misalign image features to degradation texts. This not surprising, considering that VLMs are generally trained on diverse, web-scale datasets, in contrast to most image restoration models which are trained on comparatively small datasets that are curated for specific task without corresponding image-text pairs (Li et al., 2019; Zhang et al., 2017; Zamir et al., 2022).
Image restoration methods often simply learn to generate images pixel-by-pixel without leveraging task knowledge, which usually requires repeated training of the same model for specific degradation types. A recent line of work has, however, focused on unified image restoration, where a single model is trained on mixed degradation dataset and implicitly classify the type of degradation in the restoration process (Li et al., 2022a; Potlapalli et al., 2023). While the results are impressive, they still limited to small number of degradation types and the specific datasets that go with them. In particular, they do not make use of the vast amount of information embedded in VLMs.
In this paper, combine the large-scale pretrained ussionllaguaae model CLIP with image restora- tion networks and present a multi-task framework that can be applied to both degradationnsseeijj and unified image restoration problems. Specifically, aiming addressing feature mismatching be- tween corrupted inputs and clean captions, we propose an Image Controller that adapts the VLM's image encoder to output high-quality (HQ) content embeddings aligned with clean captions. Mean- while, the controller itself also predicts a degradation embedding to match the real degradation types. This novel framework, which we call degradation- aware CLIP DD-CL.,), incorporates the human- level knowledge from VLMs into general networks that improve image restoration performance and enable unified image restoration. In addition, to learn high-quality features and degradation types from low-quality (I.Q) inputs, we construct a large mixed-degradation dataset for ten different image restoration tasks based on BLIP (Li et al., 2022b). As shown in Figure I, our DA-CLIP accurately classifies the ten different degradation types and can readily be integrated into existing restoration models, helping produce visually appealing results across the different degradations.
Our main contributions are summarised follows: (1) We present DA-CLIP to leverage large-scale pretrained vision- langun-a models as a universal framework for image restoration. The key compo- nent is image controller that predicts the degradation and adapts the fixed CLIP image encoder to output high-quality content embeddings from corrupted inputs. (2) We use cross-attention to in- tegrate the content embedding into restoration networks to improve their performance. Moreover, we introduce a prompt learning module to better utilize the degradation context for unified image restoration. (3) We construct a mixed degradation dataset containing ten different degradation types with high-quality synthetic captions. This dataset can be used to train either DA-CLIP or a unified image restoration model. (4) We demonstrate the effectiveness of DA-CLIP by applying im- age restoration models for both degradation-specde and unified image restoration. Our approach achieves highly competitive performance across all ten degradation types.
BACKGROUND AND RELATED WORK
Image restoration Image restoration aims to recover a high-quality image from its corrupted coun- terpart, which is a fundamental and long-standing problem in computer vision and contains various tasks such as image denoising (Zhang et al., 2017), deraining (Ren et al., 2019), dehazing (Song et al., 2023), deblurring (Kupyn et al., 2018), etc. Most existing works focus on the degradation- specific task which trains multiple models for different tasks separately by directly learning the target with common pixel-based losses (e.g.. E or E2). Recently, however, increasing attention has been paid to unified image restoration where multiple image restoration tasks are handled with a single model (Li et al., 2022a; Potlapalli et al., 2023). These works achieve impressive results by using an additional encoder (Li et al., 2022a; Zhou al., 2022) or a visual prompt module (Potlapalli et al., 2023) to implicitly cluster inputs according to the degradation type. However, they are still limited to few image restoration tasks and do not consider auxiliary information about the degradation type.
2

---

Published as a conference paper at ICLR 2024
(a) Degradation-aware CLIP (DA-CLIP)
Image restortion with DA-CLIP
LQ Image
anawy
degradation
Text Encoder
preprocess
caption
seagull sitting post
e
Pretrained
Image Encoder
Attention
CLIP
Contrastive I learning
Diffusion- based
A
è Image
Encoder
Imagee Controller
gradiont
e'd
Prompt
LQ Image
copy
control
DA-CLIP
G Image Controller
gracient
HQ Image
e Caption Embedding from Text-encoder
e Degradation Embedding
Content Embedding e
Degradation Embedding from Image-centroller
from Text-encoder
from Imago-encoder
Overview of our method. DA-CLIP freezes both the text and image encoders of pre- trained CLIP but learns an additional image controller with contrastive learning. This controller predicts degradation features to match real corruptions and then controls the image encoder to output high-quality content features. Once trained, DA-CLIP can be integrated into other image restoration models by simply adding cross-attention module and degradation feature prompting module.
Figure 2:
Blind image restoration (BIR) To deal with unknown degradation levels, BIR comes into view and has shown promising results for photos captured in the real-world (Zhang et al., 2021; Wang et al., 2021: Xie et al., 2021). In particular, BSRGAN (Zhang et al., 2021) and Real-ESRGAN (Wang et al., 1) utilize GANs with practical degradation settings. Inspired by recent advanced diffusion models, StableSR (Wang et al., 2023) and DiffBIR (Lin et al., 2023) propose to exploit diffusion priors to generate realistic outputs. Although pretrained diffusion weights well utilized in these methods, they do not make use of the semantic information embedded in vision-language models.
Vision-language models (VLMs) Recent works have demonstrated the great potential of applying pretrained VLMs to improve downstream tasks with generic visual and text representation (Radford al., 2021; Jia et al., 2021: Li et al.. 2022b). A classic VLM usually consists of text encoder and an image encoder and tries to learn aligned multimodal features from noisy image-text pairs with contrastive learning (Radford et al., 2021). BLIP (Li al., 2022b) further proposes to remove noisy web data by bootstrapping synthetic captions. Although VLMs provide strong capability of zero- shot and label-free classification for downstream tasks, they have so far had limited effect on image restoration due the need for specialized and accurate terminology. A noteworthy approach for fine- tuning vision-language models is so-called prompt learning (Zhou et al., 2022), where the prompt's s context words represented by learnable vectors that are then optimized for the downstream task.
Text-to-image models such as stable diffusion (Rombach et al., 2022) have gained extraordinary attention from both researchers and the general public. ControlNet (Zhang & Agrawala, 2023) builds upon this work and proposes to add controls to the diffusion network to make it adapt to task task-specific conditions. InstructPix2PPxx (Brooks al., 2023) further combines GPT-3 (Brown et al., 2020) with stable diffusion to perform an instruction- translation. Although it can generate highly realistic images, it can be directly applied image restoration tasks since the latter requires highly accurate reconstruction abilities.
Text-to-image generation
image-to-image
3 DEGRADATION-AWAN CLIP
At the core of our approach is the idea of controlling a pre-trained CLIP model to output the high- quality image feature from corrupted image while simultaneously predicting the degradation type.
3

---

Published as a conference paper at ICLR 2024
LQ
LQ
Degradation embedding
Transformer Block A
Transforner Block &
Imago Encoder
Image Controller
Prompt
zero dense
dense layer x2
zero
softmax
T-Block
A
T-Block &
zero dense
dense layer
HQ content embedding
Degradation embedding
Prompt
embedding
(a) Controller for ViT-based image encoder
(b) Prompt with degradation embeddings
Figure 3: (a) Controlling the ViT-based image encoder with a trainable controller. (b) Learning the prompt with degradation embeddings predicted from the controller.
As summarised in Figure 2, the image content embedding e matches the clean caption embedding e." Moreover, the image degradation embedding e predicted by the controller specifies the corrup- tion type of the input, i.c. the corresponding degradation embedding e from text encoder. These features can then be integrated into other image restoration models to improve their performance.
3.1 IMAGE CONTROLLER
The image controller is a copy the CLIP image encoder but wrapped with a few zero-initialised connections to add controls to the encoder. It manipulates the outputs of all encoder blocks trol the prediction of the image encoder. In this paper. we use ViT (Dosovitskiy et al., 2020) as the default backbone for both the encoder and the controller. Figure 3(a) illustrates the controlling pro- cedure, where the output of the controller consists of two parts: an image degradation embedding e/ and hidden controls he- Note that the latter contains all outputs from the transformer blocks, which are subsequently added to the corresponding encoder blocks to control their predictions. The connections between the transformer blocks are simple dense neural networks with all parameters initialized to zero, which gradually influence the image encoder during training (Zhang & Agrawala, 2023). Since our training dataset is tiny compared to the web-scale datasets used VLMs, this con- trolling strategy alleviates overfitting while preserving the capability of the original image encoder.
Optimising the image controller We freeze weights of the pretrained CLIP model and only fine-tune the image controller. To make the degradation-emmbeddnng spaces discriminative and well- separated, we use a contrastive objective (Tian et al., 2020) to learn the embedding matching process. Let N denote the number of paired embeddings (from text encoder and image encoder/controllee)) in training batch. The contrastive loss is defined as:
exp (eTyI/T)
Lcon (2r, y) = N
Σ log
(1)
j-=l exp(xe) (a.p (ITT///))) y;/T)
where x and y are normalised vectors, and r a learnable temperature parameter that controls the contrastive strength. Minimising Equation 1 amounts to optimising the cosine similarity between correctly paired embeddings while enlarging the difference with other embeddings, similar to the cross-entropy loss (Radford et al., 2021). To optimise both content and degradation embeddings, use the following joint objective:
(e(w) = Loonle : :w) + Leoa(e ;
(2)
where w represents the learnable parameters of the controller. Note that all image- based embeddings (i.e.. e! and e/) are obtained from the LQ image and all text-based embeddings (i.e.. e. and e are from the clean caption and real degradation. Learning to align these embeddings enables DA-CLIP predict real degradation types and HQ content features for corrupted image inputs.
4

---

Published as a conference paper at ICLR 2024
3.2 IMAGE RESTORATION WITH DA-
We use IR-SDE (Luo et al.. 2023a) as the base framework for image restoration. It adapts a U-Net architecture similar to DDPM (Ho al., 2020) but removes all self-attention layers. To inject clean content embeddings into the diffusion process, we introduce cross-attention (Rombach et al., 2022) mechanism to learn semantic guidance from pre-trained VLMs. Considering the varying input sizes in image restoration tasks and the increasing cost of applying attention to high-resolution features, we only use cross-attention in the bottom blocks of the U-Net for sample efficiency.
On the other hand, the predicted degradation embeddings useful for unified image restoration, where the aim is to process low-quality images of multiple degradation types with a single model (Li al., 2022a). As illustrated in Figure 1, our DA-CLIP accurately classifies the degradation across different datasets and various degradation types, which is crucial for unified image restoration (Li al., 2022a). Moreover, make use of these degradation embeddings, we combine them with prompt learning (Zhou et al., 2022) module to further improve the results, as shown in Figure 3(b). Given state Ig and low-quality image μ., our final diffusion network conditioned on the time and additional embeddings e and c as c μ4, t, e/ ey), which can be trained with either noise- matching loss (Ho et al., 2020) or maximum likelihood loss (Luo et al., 2023a;b).
Generally, we can use cross-attention to integrate content embedding into networks to improve their performance on an image restoration task. contrast, the prompt module combined with degradation embedding specifically aims to improve the classification of the degradation type in the context of unified image restoration.
Generated image-text-dogrdaation pair
several people are sitting at tables outside of a building
BLIP
snewy
Degradation
HQ Image
Clean caption
LQ Image
type
Figure 4: An example of generating the image-text-degraaatio tuple with BLIP. The clean caption generated from the HQ image is accurate and does not convey the degradation information.
4 DATASET CONSTRUCTION
crucial point of this paper to leverage powerful vision-language models to learn multiple image degradations and extract degradation-fiee free features for image restoration. For this purpose, we collect a large dataset with ten different image degradation types: blurry, hazy, JPEG-compression, low- light, noisy, raindrop, rainy, shadowed, snowy, and inpainting. Table summarises the tasks and the number of training and testing images for each degradation type, and more details are provided in Appendix A. Example tow-quality images for the ten degradations are also shown in Figure 1.
dataset, image-tex--degradatie pairs In order to train DA-CLIP on the mixed-degradation the bootstrapped vision-language framework BLIP (Li et al., 2022b) to generate synthetic captions for all HQ images. Since the inputs are clean, the generated captions are assumed to be accurate and of high-quality. As illustrated in Figure 4, we then able construct image- text-degradation pairs by directly combining these clean captions, LQ images, and the corresponding degradation types. This dataset allows us train either langunne models (based on image- text-degradation pairs) or unified image restoration framework (based on LQ-HQ image pairs).
Generating
Table Details of the collected training and testing datasets with different image degradation types.
Dataset
Blurry
Hazy
JPEG
Low-light
Noisy
Raindrop
Rainy
Shadowed
Snowy
Inpainting
#Train #Test
103
6000
3,500
485
3,500
861
2680
1872
29
1111
100
29
58
100
408
601
100
5

---

Published as a conference paper at ICLR 2024
5 EXPERIMENTS
We experimentally evaluate our method on two types of tasks: degradetion-speeefee image restora- tion and unified image restoration. In the degradation-specifie setting (Section 5.1), restoration models are separately trained for each of the considered degradation types. In unified image restora- tion (Section 5.2), a single model is instead jointly trained on all degradation types. Implementation details and additional results are provided in Appendix B and Appendix C.
Model Evaluation Our DA-CLIP is mainly evaluated in terms of how it affects the performance of the downstream image restoration model, which we evaluate both on multiple Regradation-speciie tasks and on unified image restoration. IR-SDE (Luo al., 2023a) is used as our primary baseline model. We use the Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) and Fréchet inception distance (FID) (Heusel et al., 2017) as our main metries for perceptual evaluation, but also report PSNR and SSIM (Wang et al., 2004) for reference. Additionally, we evaluate DA- CLIP in terms of how well can classify the ten different degradation types on the mixed dataset.
5.1 DEGRADATION-SPECIIC IMAGE RESTORATION
We integrate our DA-CLIP into the baseline diffusion model IR-SDE and evaluate them on four perradation-speciiie tasks: image deraining on the Rain100H dataset (Yang et al., 2017), low-light image enhancement on LOL (Wei et al., 2018), image deblurring on the GoPro dataset (Nah et al., 2017), and image dehazing on RESIDE-6K (Qin et al., 2020). All training and testing datasets are taken from the mixed degradation dataset described in Section
Comparison approaches For all tasks, we compare our method to the prevailing approaches in their respective fields such as 1) JORDER (Yang et al., 2019), PReNet (Ren et al., 2019), and MPR- Net (Zamir et al., 2021) for deraining: 2) EnlightenGAN (Jiang et al., 2021). MIRNet (Zamir al., 2020), and URetinex-Ne (Wu et al., 2022) for low-light enhancement; 3) DeepDeblur (Nah et al., 2017), DeblurGAN (Kupyn et al., 2018), and DeblurGAN-v2 (Kupyn et al., 2019) for GoPro deblur- ring; 4) GCANet (Chen et al., 2019), GridDehazeeet (Liu et al., 2019), and DehazeFormer (Song al., 2023) for image dehazing. We also compare with MAXIM (Tu et al., 2022), advanced net- work architecture that achieves state-of-the- performance on multiple dogradattin-sseeiifc tasks.
Results The quantitative results on different datasets are summarised in Table Our method achieves the best perceptual results across all tasks, and even sets a new state-of-the-art perfor- mance for all metrics on the image deraining task. Compared to the baseline method IR-SDE, our approach consistently improves its results for all datasets and metrics, demonstrating that the HQ content embedding from DA-CLIP leads to better performance for downstream image restoration models. A visual comparison of our method with other approaches is also illustrated in Figure 5. Our method produces mostly clear and visually appealing results that are close to the HQ images.
5.2 UNIFIED IMAGE RESTORATION
We evaluate our method for unified image restoration on the mixed degradation dataset which tains ten different degradation types (see Section 4 for details).
Comparison approaches We compare our method with three approaches: Restormer (Zamir al., 2022), AirNet (Li et al., 2022a), and PrompuR (Potlapalli et al., 2023). Restormer is an advanced transformer-based network. AirNet trains an extra encoder to differentiate degradation types using contrastive learning, whereas PromptIR employs a visual prompt module to guide the restoration. The latter two methods are designed specifically for unified image restoration.
Results We illustrate the comprehensive comparisons in Figure 6 and also provide the average re- sults across all ten degradation types Table 3. The results show that our method achieves the best perceptual results (especially in terms of FID) across the ten degradations, while still having good distortion performance. More importantly, by simply integrating DA-CLIP into the network, we significantly outperform the IR-SDE baseline in terms of all four metrics. A visual compari- son is shown in Figure 8. On JPEG and noise removal, our method produces realistic-looking but somewhat noisy images, giving lower distortion metrics for those particular tasks. In Appendix B.3, we evaluate methods on nat-of-distribution light rain images, in which DA-CLIP surpasses all other unified approaches, demonstrating that our method can generalize unseen degradation levels.

---

Published as a conference paper at ICLR 2024
Table Quantitative comparison between our method with other state-of-the -frt approaches on four different degradation-spesn speciffic datasets. The best results are marked in boldface.
Distortion
Perceptual
Distortion
Perceptual
Method
Method
PSNRT
SSIMI
LPIPS.
FID
PSNRT
SSIMI
LPIPS
FID
JORDER
26.25
0.835
0.197
94.58
EnlightenGAN
17.61
0.653
0.372
94.71
PReNet
29.46
0.899
0.128
52.67
MIRNet
24.14
0.830
0.250
69.18
MPRNet
30.41
0.891
0.158
61.59
URetinex-Ne
19.84
0.824
0.237
52.38
MAXIM
30.81
0.903
0.133
58.72
MAXIM
23.43
0.863
0.098
48.59
IR-SDE
31.65
0.904
0.047
18.64
IR-SDE
20.45
0.787
0.129
47.28
Ours
33.91
0.926
0.031
11.79
Ours
23.77
0.830
0.083
34.03
(a) Deraining results on the Rain dataset.
(b) Low-light enhancement on the LOL dataset.
Distortion
Perceptual
Distortion
Perceptual
Method
Method
PSNRT
SSIM
LPIPS
FID
PSNRT
SSIM
LPIPS]
FID
DeepDeblur
29.08
0.913
0.135
15.14
GCANet
26.59
0.935
0.052
11.52
DeblurGAN
28.
0.858
0.178
27.02
GridDehazeNet
25.86
0.944
0,048
10.62
DeblurGANv2
29.55
0.934
0.117
13.40
DehazeFormer
30.29
0.964
0.045
7.58
MAXIM
32.86
0.940
0.089
11.57
MAXIM
29.12
0.932
0.043
8.12
IR-SDE
30.70
0.901
0.064
6.32
IR-SDE
25.25
0.906
0.060
8.33
Ours
30.88
0.903
0.058
6.15
0.936
0.030
5.52
Deblurring results on the GoPro dataset.
(d) Dehazing results RESIDE-60 dataset.
HQ Image
Rainy Image
MPRNet
MAXIM
IR-SDE
Ours
HQ Image
Low-light Image
Uet
MAXIM
IR-SDE
Ours
HQ Image
Hazy Image
DehazeFonnee
MAXIM
IR-SDE
Ours
HQ Image
Blurry Image
DeblurGAN-v2
MAXIM
IR-SDE
Ours
Figure 5: Comparison of our method with other approaches on 4 different degralltiin--specffe tasks.
Moreover, we further integrate DA-CLIP into an MSE-based network NAFNet (Chen et al., 2022) variant of method. The results illustrated in Table and Figure We observe that adding our degradation context significantly improves the results, and the final performance of NAFNet with DA-CLIP even surpasses PrompuR across all metrics. This demonstrates that DA-CLIP can be inte- grated with both diffusion-base and direct restoration models, further improving their performance for unified image restoration. More results can be found in Appendix B.4.
Finally, we evaluate DA-CLIP in terms of degradation type classification as shown in Figure 1 and Table 4 (in the Appendix). The original CLIP model achieves less than 2% accuracy in recognizing noisy and raindrop images, and even (%% accuracy for inpainting. which would confuse down- stream unified image restoration model. By training the controller on the mixed degradation dataset, DA-CLIP perfectly predicts all degradations except blurry, for which it achieves 91.6% accuracy.
We add the prompt module to all blocks and crosv-attention in the hottom layers.

---

Published as a conference paper at ICLR 2024
PSNR
LPIPS
Pazy
elturyy
impaintiin
hazty
biurry
irpainting
hazy
blurry
Irpainting
havy
bunyy
Inpainting
iPGG
newwy
JPEG
knowy
jPec
enewy
JPEG
snowy
low-light
shadowed
low-light
Shadowed
low-light
shadowed
low
lowlight
#hadowed
teindrop
raindrop
rwindrop
ralndrop
Figure 6: Comparison of our method with AirNet, PromptIR and IR-SDE for unified image restora- tion. Each radar plot reports results for the ten different degradation types, for one particular metric. For the perceptual metrics LPIPS and FID (the two rightmost plots), a lower value is better.
Table 3: Comparison of the average results over ten different
PSNR
datasets on the unified image restoration task.
blurry
inpainting
Distortion
Perceptual
Method
snowy
PSNR
SSIM
IPPPS)
FID
NAFNet
26.34
0.847
0.159
55.68
NAFNet Degradation
27.02
0.856
0.146
48.27
low-light
shadowed
NAFNet DA-CLIP
27.22
0.861
0.145
47.94
AirNet
Restormer
25.62
26.43
0.850
0.844
0.182
0.157
54.03
64.86
naisy
raindrop
rainy
NAPNet
DD-CUP
PrompeIR
27.14
0.859
0.147
48.26
IR-SDE
23.64
0.754
0.167
49.18
Figure 7: NAFNet with DA-CLIP for unified image restoration.
Ours
27.01
0.794
0.127
34.89
5.3 DISCUSSION AND ANALYSIS
We have two different embeddings from the DA-CLIP: an HQ content embedding and degradation embedding, which are both predicted from the LQ image. The former can be used for improving general image restoration performance, whereas the latter is predicted to classify degradation types for unified image restoration models. As can be observed in Figure 9a, separately applying either the degradation embedding or the HQ content embedding improves the unified restoration perfor- mance, and the performance is further improved by applying both embeddings as in our DA-CLIP. Moreover, directly embedding the ground truth HQ image and the real degradation type leads to an upper-bound performance of our method, as illustrated in Figure 9b.
We also compare using the HQ content image embedding from our DA-CLIP with the original image content embedding obtained from the OpenAl CLIP (Radford et al., 2021). For the unified image restoration task, observe in Figure 9c that using the original CLIP content embedding fails to substantially improve the performance. For the degradation-speecfii setting as shown in Figure 9d, also observe that DA-CLIP clearly outperforms the baseline of using the original CLIP.
As an alternative to using the HQ content and degradation embeddings of DA-CLIP (from the LQ image), the degradation type and HQ caption could be encoded directly using the CLIP text encoder. This alternative approach thus requires access to the ground truth degradation label and caption text for each LQ image. Surprisingly, we observe in Figure 9e that the two degradation embeddings give very similar performance. And Figure 9f further shows that caption embeddings are also help- ful for image restoration. Moreover, while using the ground truth text embeddings improves the baseline performance, it is slightly inferior compared to using our DA-CLIP image embeddings (Figure 9g), meaning that DA-CLIP has learned to accurately predict the degradation type and HQ content embeddings. Finally, analysis of the degradation prompt module is provided in Figure 9h, demonstrating that prompt learning also facilitates unified image restoration.
While we have demonstrated the effectiveness of our proposed DA-CLIP in various settings, it is worth acknowledging one potential limitation: increased model complexity and computational cost. As can be observed in Table 8 in the Appendix, DA-CLIP significantly increases the memory requirements compared to the baseline models (both NAFNet and IR-SDE). The test-time computa- tional cost (FLOPs and runtime) is however virtually unaffected.
8

---

Published as a conference paper at ICLR 2024
JPEG
Noisy
Raindrop
Shadow
Snowy
Inpainting
HQ
LQQ
AirNet
PromptR
IR-SDE
Ours
Figure 8: Comparison of our method with other approaches on the unified image restoration. All results in each row are produced by sending images with different degradations to the same model.
Unilfied restocation
Unitied image
Unified image restoration
Image deraining
25.0
(68)
Basilioo
22.5
20
DDSCLIPP
20
Booilto
Bauilien
18
DLL (Duzx)
DD.CLUPPP arehodings
CLato
Cootor
400
400
Training steps (k)
Trzining steps
Training steps
Training sleps
(a)
(b)
(c)
(d)
image restoration
Unitled image restoration
Unitted image costonation
Unified image restoration
PSNR (di)
22.5
24
22
22
20.0
Baaclae
Tex: degrolaton
20
Rediac
TTect dagnudatee "esti-gersbieee
Bocla:
ue unbeddings
Badia:
Preect degradatos
degraaiiies
200
400
600
Training steps
Trainnn
Training
Training
(c)
(f)
(h)
Figure 9: Training curves of model variations, demonstrating the effectiveness of our DA-CLIP.
6 CONCLUSION
This paper presents DA-CLIP to leverage large-scale pretrained vision-language models as uni- versal framework for image restoration. At the core of our approach controller that accurately predicts the degradation embeddings from LQ images and also controls the CLIP image encoder to output clean content embeddings. To train DA-CLIP, we construct mixed degradation dataset containing synthetic captions from HQ images. DA-CLIP is then integrated into downstream image restoration models using prompt learning module and a cross-attention mechanism. Experimental evaluation on both degradation-specific and unified image restoration tasks demonstrates that DA- CLIP consistently improves the restoration performance, across a variety of degradation types. On the other hand, notice that the current dataset makes it hard to restore multiple degradations in the same scene. In future work, we are interested in creating practical models that are more robust to real- world captured photos and able to fully restore images with mixed degradation types.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This research was supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation; by the project Deep Probabilistic Regression New Models and Learning Algorithms (contract number: 2021-04301) funded by the Swedish Research Council; and by the Kjell & Märla Beijer Foundation. The computations were enabled by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.
REFERENCES
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: dataset and study, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 126-1336 2017. 14
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023. 3
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 3
Dongdong Chen, Mingming He, Qingnan Fan, Jing Liao, Liheng Zhang, Dongdong Hou, Lu Yuan, and Gang Hua. Gated context aggregation network for image dehazing and deraining. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1375-1383. IEEE, 2019. 6
Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In European Conference Computer Vision, pp. 17-33. Springer, 2022. 7, 16, 17
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929 2020. 4
Xiuye Gu, Tsung-Yi Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations, 2021. 2
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to a local nash equilibrium. In Proceed- ings of Advances Neural Information Processing Systems (NeurIPS), volume 30, 2017. 6
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceed- ings of Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 6840- 6851, 2020. 5
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4916 PMLR, 2021. 3
Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang. Enlightengan: Deep light enhancement without paired supervision. IEEE Transactions on Image Processing, 30:2340-2533 2349, 2021. 6
Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jifí Matas. Deblur- gan: Blind motion deblurring using conditional adversarial networks. In Proceedings the the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 8183-8192, 2018. 2,
10

---

Published as a conference paper at ICLR 2024
Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. Deblurgan-v2 deblurring (orders-of maggniiude)) faster and better. In Proceedings of the IEEE/CVF International Con- ference Computer Vision (ICCV), pp. 5878-8887 2019.
Boyun Li, Xiao Liu, Peng Hu, Zhongqin Wu, Jiancheng Lv, and Xi Peng. All-in-one image restora- tion for unknown corruption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 77452-17488 2022a. 2. 6, 17
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888 12900.. PMLR, 2022b. 2, 3, 19
Siyuan Li, Iago Breno Araujo, Wenqi Ren, Zhangyang Wang. Eric K Tokuda, Roberto Hirata Junior, Roberto Cesar-Junior, Jiawan Zhang, Xiaojie Guo, and Xiaochun Cao. Single image deraining: A comprehensive benchmark analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3838 (888-,87,240 2019.
Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Ben Fei, Bo Dai, Wanli Ouyang, Yu Qiao, and Chao Dong. Difibir: Towards blind image restoration with generative diffusion prior. arXiv preprint arXiv:2308. 15070, 2023. 3
Xiaohong Liu, Yongrui Ma, Zhihao Shi, and Jun Chen. Griddehazenet: Attention-basel multi- scale network for image dehazing. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7314-7323, 2019. 6
Yun-Fu Liu, Da-Wei Jaw, Shih-Chia Huang, and Jenq-Neng Hwang. Desnownet: Context-aware deep network for snow removal. IEEE Transactions on Image Processing, 27(6):3064-300 3073, 2018. 14
llya Loshchilov and Frank Hutter. arXiv:1711.05101, 2017. 14
Decoupled weight decay regularization. arXiv preprint
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.
Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11461-11471, 2022. 14
Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Image restora- tion with mean-reverting stochastic differential equations. In Proceedings of the 40th Interna- tional Conference on Machine Learning, volume 202, pp. 17
PMLR, 2023a. 5, 6, 16,
Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Refusion: En- abling large-size realistic image restoration with latent-space diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1680-1691, 2023b.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the 18th IEEE International Conference Computer Vision (ICCV), volume 2. pp. 416-423. IEEE, 2001. 14
Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3883-3891, 2017. 6, 14
Minheng Ni, Xiaoming Li, and Wangmeng Zuo. NUWA-LIP: LLnguage-guudees image inpainting with defect-free VQGAN. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14 14183-14192, 2023. 2
Vaishnav Potlapalli, Syed Waqas Zamir, Salman Khan, and Fahad Shahbaz Khan. Promptir: Prompt- ing for all-in-one blind image restoration. Advances in Neural Information Processing Systems (NeurIPS), 2023. 2, 6, 17
11

---

Published as a conference paper at ICLR 2024
Rui Qian, Robby T Tan, Wenhan Yang, Jiajun Su, and Jiaying Liu. Attentive generative adversarial network for raindrop removal from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2482-2491, 2018. 14
Xu Qin, Zhilin Wang. Yuanchao Bai, Xiaodong Xie, and Huizhu Jia. FFA-Net: Feature fusion attention network for single image dehazing. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 11908-1181 19955, 2020. 6, 14
Liangqiong Qu Jiandong Tian, Shengfeng He, Yandong Tang, and Rynson WH Lau. Deshadownet: A multi-context embedding deep network for shadow removal. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 4067-4075, 2017. 14
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-88888 8763.873...03 PMLR, 2021. 2, 3,4,8
Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. Progressive image deraining networks: better and simpler baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3937-3946, 2019. 6,
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Puttern Recognition, pp. 10684-10695, 2022. 3,5
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 01225278-52944 2022. 14
Sheikh. Live image quality assessment database release edu/research/quality, 2005. 14
http://live. ece. utexas.
Yuda Song, Zhuqing He, Hui Qian, and Xin Du. Vision transformers for single image dehazing.
IEEE Transactions Image Processing, 32:1927-1941, 2023. 2,
Yonglong Tian, Dilip Krishman, and Phillip Isola. Contrastive multiview coding. In European
Conference on Computer Vision, pp. 776-794. Springer, 2020. 4
Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, and Lei Zhang. NTIRE 2017 challenge on single image super-resolution: methods and results. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 114-125, 2017.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxim: Multi-axis mlp for image processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5769-5780, 2022. 6
Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. arXiv preprint arXiv:2305.07013 2023. 3
Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pp. 1905-1914, 2021. 3
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4)6600- 612, 2004. 6
Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv: 1808 04560, 2018. 14
12

---

Published as a conference paper at ICLR 2024
Wenhui Wu, Jian Weng, Pingping Zhang, Xu Wang, Wenhan Yang, and Jianmin Jiang. Uretinex-ned net: Retinex-basse deep unfolding network for low-light image enhancement. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 5901-5910, 2022. 6
Liangbin Xie, Xintao Wang, Chao Dong, Zhongang Qi, and Ying Shan. Finding discriminative filters for specific degradations in blind super-resolution. Advances in Neural Information Processing Systems, 34:51-63 6-111,6622000 2021.
Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from single image. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1357-1366, 2017. 6, 14, 16
Wenhan Yang. Robby T Taa,, Jiashi Feng, Zongming Guo, Shuicheng Yan, and Jiaying Liu. Joint rain detection and removal from single image with contextualized deep networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(6):1377-1393, 2019. 6
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Learning enriched features for real image restoration and en- hancement. In Proceedings of the 16th European Conference on Computer Vision, pp. 492-511. Springer, 2020.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14821-14831, 2021. 6
Syed Waqas Zamir, Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming- Arora, Hsuan Yang. Restormer: efficient transformer for high-resolution image restoration. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5728-5739 2022. 2, 6, 16
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. arXiv preprint arXiv:2304.0063 2023.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a Gaussian denoiser: Residual learning of deep enn for image denoising. IEEE Transactions on Image Processing, 26 (7):3142-3155, 2017. 2
Kai Zhang. Jingyun Liang, Lue Van Gool, and Radu Timofte. Designing practical degradation model for deep blind image super-resolution. Conference Computer Vision, pp. 4791-4800, 2021. 3
Proceedings of the IEEF/CVE International
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint 2rXiv:2302.05553, 2023. 3,4
Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-fres free adaption of clip for few-shot classification. In European Conference on Computer Vision, pp. 493-510. Springer, 2022. 2
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 586-595, 2018. 6
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-2344 2022, 2, 3,
13

---

Published as a conference paper at ICLR 2024
A MORE DETAILS ABOUT DATASETS
In this section, we give more details about the mixed degradation dataset in Section 4. We collect images for 10 different image restoration tasks, including blurry, hazy, JPEG-coopressing, low- light, noisy, raindrop, rainy, shadowed, snowy, and inpainting, as shown in Figure 1. The details of these datasets are listed below:
Blurry: collected from the GoPro (Nah et al., 2017) dataset containing 2103 and 1111 training and testing images, respectively.
Hazy: collected from the RESIDE-6k (Qin et al., 2020) dataset which has mixed indoor and outdoor images with 6000 images for training and 1000 images for testing.
OPEG-compressingg the training dataset has 3440 images collected from DIV2K (Agusts- son & Timofte, 2017) and Flickr2K (Timofte et al., 2017). The testing dataset contains 29 images from LIVE1 (Sheikh, 2005). Moreover, all IQQ images are synthetic data with JPEG quality factor of 10.
Low-light: collected from the LOL (Wei et al., 2018) dataset containing 485 images for training and 15 images for testing.
Noisy: the training dataset is the same as that in JPEG-compressing but all LQ images are generated by adding Gaussian noise with noise level 50. The testing images are from CBSD68 (Martin et al., 2001) and also added that Gaussian noise.
Raindrop: collected from the RainDrop (Qian et al., 2018) dataset containing 861 images for training and images for testing.
Rainy: collected from the Rain100H (Yang et al., 2017) dataset containing 1800 images for training and 100 images for testing.
Shadowed: collection from the SRD (Qu et al., 2017) dataset containing 2680 images for training and 408 images for testing.
collected from the Snow 100K-L (Liu et al., 2018) dataset. Since the original dataset is too large (100K images), we only use subset which contains 1872 images for training and 601 images for testing.
Snowy:
Inpainting: we use CelebaHQ as the training dataset and divide 100 images with 100 thin masks from RePaint (Lugmayr et al., 2022) for testing.
We also provide several visual examples for each task for a better understanding of the 10 degrada- tions and datasets, as shown in Figure 10.
B IMPLEMENTATION DETAILS AND MORE ANALYSIS
B.1 IMPLEMENTATION DETAILS
The base CLIP model uses ViT-B-32 as the backbone for its image encoder, with weights pretrained on the LAION-2B dataset (Schuhmann et al., 2022). Built upon it, fine-tune the DA-CLIP on the mixed degradation dataset with a batch size of 3 136 (784 x 4) and learning rate 3 x 10 In preprocessing, all inputs are normalized in the range [0, 1] and resized to 224 x 224 with bicu- bic interpolation. We train the DA-CLIP model on four NVIDIA A100 GPUs for 50 epochs, in approximately 3 hours. For the restoration model, we use batch size of 16 and randomly crop images to 256 x 256 for data augmentation. The initial learning rate is 2 x 10-4 We use the AdamW (Loshchilov & Hutter, 2017) optimizer (81 0.9, 81 0.99) with cosine decay for a total of 700K iterations. All training is done using one A100 GPU for about 5 days.
B.2 ADDITIONAL ANALYSIS OF DA-CLIP
We provide the degradation classification results of three strategies: 1) retraining CLIP on our dataset; 2) fine-tuning CLIP from pretrained weights; and 3) adding controller but without zero- initializing its dense layers. The prompt is set to "a [degradation type] photo" for all models. The results reported in Table 4. Obviously, CLIP with finetuning significantly improves the results of
14

---

Published as a conference paper at ICLR 2024
Blurry
Hazy
JPEG
Low-light
Noisy
Raindrop
Rainy
Shadowed
Snowy
Inpainting
Example images with 10 image restoration tasks. For each task, the first row is the corrupted input and the second row the result produced by our unified image restoration model.
Figure 10:
Table 4: Accuracies of the degradation classification. The average accuracy for each method is provided in the last column. 'DA-CLIP w/o Zero' means adding a controller without zero-initialising its dense layers.
Model
Blurry
Hazy
JPEG
Low-light
Noisy
Raindrop
Rainy
Shadowed
Snowy
Inpainting
Average
OpenAT CLIP
80.3
72.9
20.7
13.3
1.7
80
29.9
0
26.7
Retrain CL.IP
65.3
97.8
65.5
100
86.2
99
94.6
100
89.6
CLIP finetune
95.6
100
100
100
100
100
100
100
100
99.6
DA CLIP w/o Zero
90.1
99
100
100
97.1
96.5
100
99.7
100
98.1
DA-CLIP (Ours)
91.6
100
100
100
100
100
100
100
100
99.2
direct retraining, demonstrating the effectiveness of large-scale vision-language model pretraining. Note that although fine-tuning CLIP achieves slightly better performance on blurry classification than DA-CLIP, it is not able to predict HQ content embeddings from the LQ image input, which leads to limited effects on downstream restoration models. Moreover, initializing dense layers to zero can further improve the accuracy of all datasets without adding additional costs.
Table 5: Evaluating unified models on the Rain dataset for OOD analysis.
Distortion
Perceptual
Method
PSNRT
SSIM
I.PISS).
FID
AirNet
30.07
0.935
0.114
45.56
PrompuI
32.77
0.950
0.081
43.31
IR-SDE
30.13
0.904
0.098
35.05
Ours
36.61
0.967
0.025
11.98
15

---

Published as a conference paper at ICLR 2024
Table Ablation experiments of applying DA-CLIP other MSE-based methods on unified image restoration. PromptIR is built upon Restormer but with additional degradation prompt modules.
Method
PSNR1
SSIM
LPIPS
FID!
NAFNet
26.34
0.847
0.159
55.68
Degradation embedding
27.02
0.856
0.146
48.27
Degradation and Content embedding
27.22
0.861
0.145
47.94
Restormer
26.43
0.850
0.157
54.03
PromptIR (Restormer degradation)
27.14
0.859
0.147
48.26
PromptR + Content embedding
27.26
0.861
0.145
47.75
Table 7: Results of integrating the DA-CLIP into NAFNet for degradation-sppeciii tasks.
Distection
Perceptual
Distortion
Perceptual
Method
Method
PSNRT
SSIM
LPIPS)
FID]
PSNRT
SSIM
LPIPS.
FID
NAFNet
31.49
0.903
0.087
31.05
NAFNet
23.09
0.839
0.122
57.45
NAFNet-DA-CLIP
31.68
0.907
0.086
31.02
NAFNeH+DA-CLPP
23.72
0.844
0.116
50.69
(a) Deraining results on the Rain 100H dataset.
(b) Low-light enhancement on the LOL dataset.
B.3 EXPERIMENTS ON OUT-OF-DISTRIBUTIO (OOD) DEGRADATION
To evaluate our model's generalization capability, apply our model on an additional light rain dataset 100L Yang et al. (2017). Since our mixed-degradation dataset only contains heavy rain images (from the Rain 100H dataset), Rain100L is out-of-distribution dataset with relatively low-level degradation. Table summarises the results. surprisingly well on this unseen dataset and surpasses all other unified approaches by significant margin. In addition, we also provide some real-world visual examples in Figure 17, showing that model seems generalize well compared the AirNet and PrompIR baselines.
observed that our method performs
B.4 IMPROVING MSE-BAN BASED METHOD WITH DA-CLIP
It is worth noting that DA-CLIP not only benefit the diffusion- based image restoration approach but also facilitate standard MSE-based model learning. Table 6 provides two ablation experiments NAFNet (Chen et al., 2022) and Restormer (7amir al., 2022). Note that Promp- tIR built upon Restormer but with additional degradation prompt modules. As can be seen, the degradation context useful for both methods, and adding additional content embedding can further improve their performance.
B.5 TRAINING CURVES ON SINGLE DEGRADATION TASKS
To further illustrate the effectiveness of our method, we compare the training curves between the baseline model IR-SDE (Luo al., 2023a) and that with DA-CLIP embeddings four dif- ferent degradation-speccif image restoration tasks: image deraining, low-light enhancement, image deblurring, and image dehazing. The results are shown in Figure 11, from which we can see the training with our DA-CLIP obviously performs better than the baseline model.
Image dervnnigg
Image dehluriing
debazing
25
32
26
Suxiinc
200
400
600
500
500
Triiiin steps (K)
Traming sieps
(k)
Training
Figure 11: Ablation studies on the degrallation-speciffe image restoration task.
16

---

Published as a conference paper at ICLR 2024
B.6 ANALYSIS ABOUT THE MODEL COMPLEXITY
We have shown the effectiveness of applying our DA-CLIP to various image restoration models and tasks. However, worth acknowledging one potential limitation: the model complexity is also increased along with the DA-CLIP inference and with additional prompt modules and cross- attention layers. The comparison of model complexities is shown in Table 8. DA-CLIP significantly increases the memory requirements compared to the baseline models (both NAFNet and IR-SDE). The test-time computational cost (FLOPs and runtime) is however virtually unaffected.
B.7 ANALYSIS OF PATCH SIZES
Intuitively, large patch sizes always contain more semantic information that might be important for guiding image restoration. We explore this property by training model with two different patch sizes on the unified image restoration task. As shown Figure 12, increasing the patch size from 128 128 to 256 x 256 improves the training process, which is consistent with our conjecture.
Table 8: Comparison the number of parameters, model computational efficiency. and inference time. The flops and inference time are computed on face inpainting images of size 256 x 256. Note that MAXIM implemented with the JAX GPU version.
METHOD
MAXIM
PROMPTIR
NAFNET
NAFNET DA-CLIP
IR-SDE
IR-SDE + DA-CLIP
#PARAM
14.1M
33M
67.8M
86. M
36.2M
48.9M 125.2M
FLOPS
216G
158G
63G
65G 0.5G
117G
118G 0.5G
RUNTIME
2.95
0.18
0.08s
0.08s
4.28
4.53s 0.06s
C
ADDITION EXPERIMENTAL RESULTS
In this section, we provide details and additional experimental results.
C.I DETAILED QUANTITATIVE RESULTS FOR UNIFIED IMAGE RESTORATION
We provide some examples show the degradation predic- tion on specific images with different corruptions in Figure 13. In Section 5.2 we have reported the average results and radar figures for different metrics. Here we also provide more de- tailed comparison between our method with other approaches for unified image restoration. The results on four metrics are shown Table 9, Table 10, Table 11, and Table 12. Ad- ditionally, we also provide detailed comparison between NAFNet (Chen et al., 2022) and its variant (with DA-CLIP) in all tables, which illustrates the potential of applying DA- CLIP to other general image restoration frameworks.
Unified image restoration
PSNR 225
23
Patch 128
Patch 256
200
400 Training steps (k)
600
Figure 12: Comparison of training curves with different patch sizes.
C.2 ADDITIONAL. VISUAL RESULTS
The additional visual results for unified image restoration are shown in Figure 14, Figure 16. Figure 15 shows the compari- son with Real-ESRGAN and StableSR on a compressed image and blurry image. Figure 17 also illustrates some examples of testing our method on real-world im- ages. We basically compare with PrompuR (Potlapalli et al., 2023) and IR-SDE (Luo et al., 2023a) but also add the results generated from AirNet (Li al., 2022a).
D LIMITATION ON MIXED DEGRADATIONS
Since the mixed degradation dataset contains a single degradation label for each image, our current model has not been trained restore multiple degradations in the same scene. For example, the
17

---

Published as a conference paper at ICLR 2024
raindrop image in Figure 8 contains a shadow area, but our model only removes the raindrop degra- dation. Also, due to this limitation, this paper can't well-explore and justify the effectiveness of linguistic components based mixed degradations.
In future work, we are very interested in 1) creating practical models that are able to process real- world degradations and more complex degradations; 2) exploring other pretrained VLM backbones for more robust visual eeprssenationseembedding. 3) exploring more interesting works for the linguistic side (from text encoder) such as instruction-based image restoration.
18

---

Published as a conference paper at ICLR 2024
Degradation Prediction (%)
chipenunt standing on rock
Degradation Prodiction (%)
benches and mees
09.5
8:2
17:4
19,1
Islarry
Netry
Hanny
Biurry
Notey
Keiny
Blury Image
Noisy Image
many lions walking the grass logether
Degradation Prediction
people walking down strect with trallic
Degradation Prodiction (%)
99.9
99.8
69.3
12.7
20.5
Blury
Nola
Hazy
Noisy
Raiay
Rainy
Hazy Image
CCLIP
DDACLIPP
CLIP
DACCLIP
Figure 13: Comparison of CLIP and DL-C for image degradation prediction. The captions above the corrupted images are generated using BLIP (Li al., 2022b) with corresponding clean images.
Table 9: Comparison of our method with other unified image restoration approaches on PSNR.
Model
Blury
Hazy
JPBG
Low-light
Noisy
Raindrop
Rainy
Shadowed
Snowy
Inpainting
Avergg
NAFNet
26.12
24.05
26.81
22.16
27.16
30.67
27.32
24.16
25.94
29.03
26.34
NANNGI-DA-CLIP
26.40
26.39
27.13
23.09
27.43
31.08
28.23
25.80
26.64
30.01
27.22
Restormer
26.34
23.75
26.90
22.17
27.25
30.85
27.91
23,33
25.98
29.88
26.43
PrompiR
26.50
25.19
26.95
23.14
27.56
31.35
29.24
24.06
27.23
30.22
27.14
27.56
Ours
27.03
29.53
23.70
22.09
24.36
30.81
29.41
27.27
26.83
28.94
27.01
Table 10: Comparison of our method with other unified image restoration approaches on SSIM.
Model
Bluryy
Hazy
JPBG
Low light
Noisy
Raindrop
Rainy
Shadowed
Snowy
Inpainting
Average
NAFNet
0.804
0.926
0,780
0.809
0,768
0.924
0,848
0.839
0.901
0.847
NAPNCI-DA-CLIP
0.816
0.948
0.798
0.823
0.777
0.928
0.869
0.849
0.880
0.914
0.861
Restormer
0.811
0.915
0.781
0.815
0.762
0.928
0.862
0.836
0.877
0.912
0,850
AirNet
0.805
0.916
0.783
0.781
0.769
0.926
0.867
0.832
0.846
0.911
0,844
PromptR
0.815
0.933
0.784
0.829
0.774
0.931
0.876
0.842
0.887
0.918
0.859
SDE
0.730
0.832
0.615
0.719
0,640
0.822
0,808
0.667
0.828
0.876
0.754
Ours
0.810
0.931
0.532
0.796
0.579
0.882
0.854
0.811
0.854
0.894
0.794
Table 11: Comparison of our method with other unified image restoration approaches LPIPS.
Model
Blurry
Hazy
JPBG
Low-light
Noisy
Raindrop
Rainy
Shadowed
Snowy
Inpainting
Average
NAFNet
0.284
0.043
0.303
0.158
0.216
0.082
0.180
0.138
0.096
0.085
0.159
NAPNet-DA CLIP
0.261
0.034
0.284
0.140
0.218
0.083
0.146
0.135
0.083
0.071
0.145
Restormer
0.282
0.054
0.300
0.156
0.215
0.083
0.170
0.145
0.072
0.157
AirNet
0.279
0.063
0,302
0.321
0.264
0.095
0.163
0.145
0.112
0.071
0.182
PromptIR
0.267
0.051
0.269
0.140
0.230
0.078
0.147
0.143
0.082
0.068
0.147
IR-SDE
0.198
0.168
0.246
0.185
0.232
0.113
0.142
0.223
0.107
0.065
0.167
0.140
0.037
0.317
0.114
0.272
0.068
0.085
0.118
0.072
0.047
0.127
Table 12: Comparison of our method with other unified image restoration approaches on FID.
Model
Rluryy
Hazy
JPEG
Low-light
Noisy
Raindrop
Rainy
Shadowed
Snowy
Inpainting
Averge
NAPNet
42.99
15.73
71.88
73.94
$2.08
$6.43
86.35
47.32
35.76
44,32
55.68
36.36
68.60
47.94
Restormer
39.08
15.34
72.68
78.22
87.14
50.97
78.16
48,33
33.45
36.96
54.03
AirNet
41.23
21.91
78.56
154,2
93.89
52.71
72.07
64.13
36,99
32.93
64.86
PromptTR
36.5
10.85
73.02
67.15
84.51
44.48
61.80 1.888
43.24
28.29
32.69
48.26
IR-SDE
29.79
23.16
61.85
66.42
79.38
50.22
63.07
50.71
34.63
32.61
49.18
Ours
14.13
5.66
42.05
52.23
64.71
38,91
52.78
25.48
27.26
28.73
34,39
19

---

Published as a conference paper at ICLR 2024
IQ Image
Blurry Input
AirNet
PromptIR
IR-SDE
Ours
Hazy Input
JPEG Input
Low-light Input
HQ Image
Noisy Input
PrompiIR
IR-SDE
Ours
Figure 14: Comparison of our method with other approaches on the unified image restoration.
LQ Image
Real-ESRGAN
StableSR
Ours
Figure 15: Comparison of our method with Real-ESRGAN and StableSR.
20

---

Published as a conference paper at ICLR 2024
Raindrop Input
Rainy Input
Shadowed Input
Snowy Input
HQ Image
Inpainting Input
PromptIR
IR-SDE
Ours
Figure 16: Comparison of method with other approaches the unified image restoration.
Real-world LQ
AirNet
PromptIR
Ours
Figure 17: Comparison of our method with other approaches real-world images.
21


Reviews:
[{"review_id": "0HgrRCzrkc", "reviewer": "Reviewer_j7ME", "summary": "The paper proposes a framework called degradation-aware CLIP (DA-CLIP) that combines large-scale pretrained vision-language models with image restoration networks. The authors address the issue of feature mismatching between corrupted inputs and clean captions in existing vision-language models (VLMs) by an Image Controller that adapts the VLM's image encoder to output high-quality content embeddings aligned with clean captions. The controller also predicts a degradation embedding to match the real degradation types. The paper presents the construction of a mixed degradation dataset for training DA-CLIP and demonstrates its effectiveness in both degradation-specific and unified image restoration tasks. The results show highly competitive performance across ten different degradation types.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes a novel framework, DA-CLIP, which combines large-scale pretrained vision-language models with image restoration networks.\n- This paper introduces an Image Controller that addresses the feature mismatching issue between corrupted inputs and clean captions in existing vision-language models. In addition, they introduce a prompt learning module to better utilize the degradation context for unified image restoration.\n- It demonstrates that DA-CLIP in both degradation-specific and unified image restoration tasks, achieving highly competitive performance across all ten degradation types.", "weaknesses": "- In Figure 1, DA-CLIP achieves surprisingly high accuracy in ten degradation types. How are these experiments set up? In contrast, CLIP performs poorly in many types. What prompts do the authors use for classifying degradations in CLIP?\n- In Figure 6, PromptIR is comparable or even better than the proposed DA-CLIP in most tasks on fidelity metrics.\n- In Table 2(c), the PSNR of DA-CLIP highly deviates from that of MAXIM. In addition, the results on task-specific restoration do not show a clear benefit of using a universal model for all tasks. It is believed that the merit of universal models is that different tasks can benefit each other, or at least be helpful in generalization to new domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7hq7sRekAF", "reviewer": "Reviewer_hnNQ", "summary": "This paper proposes a degradation-aware CLIP model. It is aligned with language through dual aspects of image content and degradation during training. This DA-CLIP can extract information not only about the image content but also about image degradation. The authors also combined DA-CLIP with image restoration, proposing what they call a \"universal\" image restoration method. This method is based on Diffusion-Based restoration techniques, but the DA-CLIP's results are used as controlled prompts for input. The authors have showcased many results.", "questions": "See Weakness", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "I hold a positive view on the idea of incorporating degradation information into CLIP.", "weaknesses": "My main concern with this paper is its task setting. First, \"Universal Image Restoration\" is a term that is not so easily justified. This paper simply brings together ten different image restoration tasks, which is closer to \"multi-task\" than the so-called \"universal\". For a large model, mixing these ten tasks in such a separate manner for training, the model would internally categorize the problems before handling them in a single-task manner [R1]. This would not endow the model with sufficient generalization capabilities. For instance, an image with both rain streaks and subsequent compression artifacts cannot be accurately restored. This is not \"universal\". Moreover, this paper seems to ignore a host of more \"universal\" solutions, such as Real ESRGAN, BSRGAN, StableSR, DiffBIR, etc. Merging degradations to achieve better generalization is a new direction (which is not so new anymore). But this paper barely discusses whether these methods are \"universal\" or not.\n\nSecondly, DA-CLIP's ability to predict degradation is not particularly special, considering the task setting only involves ten types of degradation; it can be said that almost any image restoration model trained on these degradations or any model that understands or classifies them would have this ability [R2]. I can only say that introducing degradation into CLIP is a very promising direction and could be very useful. However, the approach taken in this paper fails to reflect any significance in doing so. Due to the inherent issues with the task setting of the paper, the experimental part also fails to demonstrate the corresponding contributions.\n\n[R1] Finding Discriminative Filters for Specific Degradations in Blind Super-Resolution\n[R2] Discovering “Semantics” in Super-Resolution Networks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9eCh4Rv3hG", "reviewer": "Reviewer_3ZRs", "summary": "This paper presents a degradation-aware vision language model (DA-CLIP) to generate high-quality image representation and distinct the degradation types of low-quality inputs for all-in-one image restoration. The DA-CLIP model can be integrated into different image restoration networks to improve the performance. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method.", "questions": "1. Do both $e_c^I$ and $e_d^I$ have an important impact on the restoration performance? Intuitively, it is reasonable for diffusion-based models as $e_c^I$ controls the content and $e_d^I$ indicates the degradation. However, it seems not reasonable for mse-based models to use $e_c^I$ in the restoration process. \n2. What performance can be achieved by directly providing the semantic text prompt and the degradation type prompt to train the all-in-one image restoration diffusion model?\n3. What performance can be achieved by directly training a vanilla Restormer under the same all-in-one setting?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of constructing a vision-language model to restore clean semantic image representation and distinct degradation types of low-quality images is interesting.\n2. The method of using clean image representative and degradation prompt to instruct restoration networks for better performance is sound.\n3. The results look good, and the experimental analysis demonstrate the effectiveness of the DA-CLIP on all-in-one image restoration.\n4. The writing is well, and the paper is easy to read.", "weaknesses": "1. It is questionable that the caption embedding can provide a high-quality image representation supervision for the content embedding. $e_c^T$ can indeed provide a semantic supervision, but there is no guarantee that it is a clean image representation. Therefore, I think the claim that the image encoder with the controller outputs high-quality content features is not rigorous. It seems that $e_c^I$ mainly serves to provide semantic instruction for the restoration network, especially for diffusion-based models.\n2. The used experimental setup is too simple to demonstrate the superiority of this complex method. It is not difficult for a unified network, e.g., a vanilla version Restormer, to deal with the all-in-one image restoration setting with a specific degradation level for each degradation type.\n3. This paper do not provide the experiment about the generalization ability of the proposed method. I do some tests using the code provided by the authors, and the results show that the model cannot deal with out-of-distribution degradations as well as OOD degradation levels well. It is not surprising as the used degradation model is too simple. This also reflects that $e_c^I$ is not always a high-quality image representation.\n4. This method may be difficult to handle tasks with different degradation levels, because it is difficult to describe the specific degradation level in texts. Since the authors do not provide relevant experiments, the potential of this method to handle multiple degradation levels is still questionable. As far as the current results are concerned, the approach is not practical enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VaC7ZasBu1", "reviewer": "Reviewer_bHgA", "summary": "This paper presents a novel method for universal image restoration. The method, named as DA-CLIP, is based on the CLIP and diffusion model (IR-SDE). An additional controller is proposed to help predict high-quality content embedding and degradation type embedding. These two embeddings are then used through cross attention in a diffusion unet to help restore the degraded images. The proposed method is tested on the image denoising, inpainting, deblurring, etc. The experimental results show that the proposed method can achieve better performance than the state-of-the-art methods.", "questions": "#### **1. Why VLM like CLIP is necessary in this paper ?**\n\nAfter reading the paper, I think that the proposed method does not have much relationship with VLM like CLIP. The DA-CLIP serves as a degradation type classifier and a content classifier. It seems OK to replace CLIP with simple CNNs such as ResNet. Given that the degradation types are quite limited in this paper, a text-encoder like CLIP is easily to be replaced with simple classification. As for the captions, we may also simply replace it with just clean image embeddings to perform contrastive learning.   \n\n#### **2. Why diffusion model is used for restoration process ?**\n\nGiven the slow inference of diffusion model, it is not clear why diffusion model is used for restoration process. After all, the proposed method has little relationship with diffusion model, especially when the diffusion network is trained from scratch. Compared with `NAFNet+DA-CLIP`, the diffusion based backbone does not have much advantages.\n\n#### **Summary & Conclusion**\nIn summary, I think that using contrastive learning to learn degradation type embedding and content embedding for universal image restoration is a good idea. And the experiments are also quite comprehensive and effective. However, the proposed integration with VLM and diffusion models are not well justified. And I do not think it is appropriate to claim vision-language models as the main contribution and novelty of this paper.\n\nTo conclude, I would like to give borderline to this paper. However, there is no such options in the review form. So I choose to give a marginal accept. I hope the authors can address my concerns in the rebuttal.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper proposes a novel framework, DA-CLIP, to learn high-quality content and degradation embeddings through contrastive learning. \n1. The integration of the degradation and content prompts to universal image restoration appears effective and innovative.\n1. Extensive experiments on various tasks are conducted to show the effectiveness of the proposed method.", "weaknesses": "1. There are no justifications about what is embedded in the HQ content embedding. It is better to provide some comparisons with the HQ content embedding and the original image embedding.\n1. The effectiveness and necessity of the prompt learning module is not well discussed. It is better to provide some ablation studies to show the effectiveness of the prompt learning module compared with naive cross-attention.  \n1. The performance compared with `NAFNet+DA-CLIP` is not superior.  \n1. Although the paper discussed the computation complexity in supplementary material, it only provides #params and FLOPS. It is known that the FLOPS is not a good metric for the computation complexity, especially for diffusion models which require multiple iteration steps. The paper should provide the inference time for the proposed method, and better in main paper. \n1. Experiments on more complex degradation scenarios, involving multiple concurrent degradation types, would emphasize the model's robustness and versatility.\n1. A deeper discussion about the text encoder's role in the performance could lead to a better understanding of the proposed framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)