Title: DDR: Exploiting Deep Degradation Response as Flexible Image Descriptor
Venue: NeurIPS 2024
Accepted: yes
Keywords: Degradation Response

Image Discriptor

Image Deblur

Image Super-Resolution

Image Quality Assessment
Abstract: Image deep features extracted by pre-trained networks are known to contain rich and informative representations. In this paper, we present Deep Degradation Response (DDR), a method to quantify changes in image deep features under varying degradation conditions. Specifically, our approach facilitates flexible and adaptive degradation, enabling the controlled synthesis of image degradation through text-driven prompts. Extensive evaluations demonstrate the versatility of DDR as an image descriptor, with strong correlations observed with key image attributes such as complexity, colorfulness, sharpness, and overall quality. Moreover, we demonstrate the efficacy of DDR across a spectrum of applications. It excels as a blind image quality assessment metric, outperforming existing methodologies across multiple datasets. Additionally, DDR serves as an effective unsupervised learning objective in image restoration tasks, yielding notable advancements in image deblurring and single-image super-resolution. Our code is available at: https://github.com/eezkni/DDR.

DDR: Exploiting Deep Degradation Response as Flexible Image Descriptor

Abstract

Image deep features extracted by pre-trained networks are known to contain rich and informative representations. In this paper, we present Deep Degradation Response (DDR), a method to quantify changes in image deep features under varying degradation conditions. Specifically, our approach facilitates flexible and adaptive degradation, enabling the controlled synthesis of image degradation through text-driven prompts. Extensive evaluations demonstrate the versatility of DDR as an image descriptor, with strong correlations observed with key image attributes such as complexity, colorfulness, sharpness, and overall quality. Moreover, we demonstrate the efficacy of DDR across a spectrum of applications. It excels as a blind image quality assessment metric, outperforming existing methodologies across multiple datasets. Additionally, DDR serves as an effective unsupervised learning objective in image restoration tasks, yielding notable advancements in image deblurring and single-image super-resolution. Our code is available at: https://github.com/eezkni/DDR

1 Introduction

Deep features extracted by pre-trained neural networks are well-known for their capacity to encode rich and informative representations [1-4]. Extensive research efforts have aimed to quantify the information encoded within these deep features for use as image descriptors. For example, the distance between deep features has been employed as a metric for image quality assessment (IQA) in various studies [3, 5, 6]. Additionally, researchers have studied differences between images by comparing the distributions [2, 7] or frequency components [8, 9] of their deep features. Moreover, the statistical properties of deep features have been found to correlate with the style and texture of images in prior works [6, 10]. Recent research has also highlighted that the internal dissimilarity between deep features at different image scales can serve as a potent visual fingerprint [11].

This paper delves into an intriguing and unexplored property of image deep features: their response to degradation. Specifically, when subjecting images with diverse content and textures to various types of degradation, such as blur, noise, or JPEG compression, the deep features of these images exhibit varying degrees of change. This phenomenon is illustrated in Fig. 1, where Gaussian Blur is applied to images. The degrees of changes in feature space reflect the deep feature response to specific degradation. As shown in Tab. 1, a strong correlation exists between this response and the quality scores of blurred images. We validate that the response of deep features to degradation effectively

x

xd

l(x,xd)

0.351

0.349

0.222

0.193

0.048

0.042

175

DegradationLevel

low

Number

150

optimal

125

high

100

adaptive

Image !

75

50

25

0

ResponseValue

captures different image characteristics by varying the type of degradation. Therefore, we propose the Deep Degradation Response (DDR), which quantifies the response of image deep features to specific degradation types, serving as a powerful and flexible image descriptor.

One straightforward approach to compute the DDR is to apply handcrafted degradation to the image, extract degraded features from the degraded image, and then calculate the distance between these degraded features and the features of the original image. However, as shown in Fig. 2, adjusting the level of degradation applied to the image significantly affects the distribution of DDR. Therefore, meticulous adjustment of the degradation level is crucial to achieve optimal performance in various downstream tasks. To address this challenge, we propose a text-driven degradation fusing strategy. Inspired by manifold data augmentation algorithms [12, 13], we adaptively fuse text features representing specific degradation onto the original image features, resulting directly in degraded features. By manipulating the text, we can effectively control the type of degradation fused to the image features. This approach allows us to flexibly assess the response of image features to various degradation types, thereby enhancing adaptability across different downstream tasks.

We evaluate the performance of our proposed DDR across multiple downstream tasks. Firstly, we assess its effectiveness as an image quality descriptor on the opinion-unaware blind image quality assessment (OU-BIQA) task, where DDR demonstrates superior performance compared to

existing OU-BIQA methods across various datasets. Secondly, we employ DDR as an unsupervised learning objective, training image restoration models specifically to maximize the DDR of the output image, which includes tasks such as image deblurring and real-world single-image super-resolution. Incorporating DDR as an external training objective consistently improves performance in both tasks, highlighting the strength of DDR as a flexible and powerful image descriptor.

2 Related Works

Deep Feature Based Image Descriptors. Image descriptors aim to quantify fundamental characteristics of images, such as texture [22], color [23, 24], complexity [25], and quality [26, 27]. With the informative representations in deep features extracted by pre-trained networks, various efforts have been made to develop image descriptors based on these features. Many existing descriptors regress the deep features of an image to a score, training the model by minimizing the loss between these predicted scores and the ground truth scores labeled by humans [25, 20, 28]. However, these methods are somewhat inflexible for two reasons: (1) they rely on human-labeled opinion scores, and (2) they are designed to evaluate fixed image characteristics. In this paper, we propose a flexible alternative by measuring the degradation response of deep features.

Image Degradation Representation for Image Restoration. Various deep learning-based image restoration methods leverage image degradation representation to enhance model performance [2932]. For instance, some methods utilize contrastive-based [29] and codebook-based [30] techniques to encode various degradation types, enhancing the model's robustness to unknown forms of degradation. Moreover, other methods design degradation-aware modules to extract degradation representations from images and guide the removal of degradation [31, 32]. However, these methods depend on task-specific training. In contrast, the proposed DDR flexibly obtains representations for different types of degradation, making it an effective image descriptor for various image restoration tasks.

Multimodal Vision Models. Recent advancements in multimodal vision models, achieved through extensive training on paired image-text data [33-37], have significantly enhanced the capability of these models to understand and describe image textures using natural language [38]. Researchers have explored various methods to leverage this capability for modifying image texture attributes through language guidance. For instance, in language-guided image style transfer [39, 40], natural language descriptions are used to define the target texture style. Additionally, Moon et al. [12] introduced a manifold data augmentation technique that integrates language-guided attributes into image features. Building on these ideas, our work aims to adaptively fuse degradation information into image deep features using language guidance, thereby facilitating the measurement of our proposed DDR.

3 Deep Degradation Response as Flexible Image Descriptor

3.1 Deep Degradation Response

We define the Deep Degradation Response (DDR) as the measure of change in image deep features when specific types of degradation are introduced, which can be mathematically expressed as:

where d represents the type of degradation. M ( · , · ) denotes a disparity metric, such as L n distance or cosine distance. F = Φ v ( i ) represents the original image features extracted by a pre-trained network Φ v ( · ) , while F d denotes the degraded features.

The core of the proposed DDR lies in how to model F d . Anaive approach to this involves synthesizing degradation in the pixel domain , i.e. , generating degraded images. As shown in Fig. 3 (a), a handcrafted degradation process is applied to the image, leading to the creation of a degraded image i d . The extent of degradation is controlled by the parameter ω d . Then a pre-trained visual encoder Φ v ( · ) is utilized to extract the features of i d , generating degraded features. Therefore, the pixel space degradation synthesizing process can be formulated as:

where D ( · ) denotes the handcrafted degradation synthesis process. However, as depicted in Fig. 2, varying levels of degradation significantly affect DDR. For downstream tasks, it is imperative

𝐷(𝑖, 𝜔

#

)

Positive

Prompt

𝑃

#

&

Degraded

Prompt

𝑃

#

%

-

𝒯

#

+

𝒯

#

%

𝒯

#

&

ℱ

ℱ

#

ℱ

ℱ

#

(a) DDR with

synthetic pixel degradation

(b) DDR with

text-driven feature degradation

Visual Encoder

🧊

Degradation Adaptation

Visual Encoder

🧊

Text Encoder

🧊

Td-μ(Ta)

=o(F)

+μ(F)!

o(Ta)

--→DDRd(i) = M (F,Fa)←

~→DDRa(i) = M(F,Fa)

to meticulously determine the optimal ω d for different manual processes. This not only poses a substantial challenge but also diminishes the robustness of DDR as an image descriptor.

In this study, we propose a novel and efficient method for modeling F d by synthesizing degradation in the feature domain using text-driven prompts. Specifically, to construct the degradation representation, we first design a pair of prompts: one describing an image with a specific type of degradation and the other describing the same image without degradation. These prompts are then separately encoded using the text encoder Φ t ( · ) of CLIP [37], yielding text-driven degradation representations T -d and T + d , respectively. We obtain the degradation direction in the feature space by calculating the difference between these representations, as follows:

where T -d = Φ t ( P -d ) and T + d = Φ t ( P + d ) . P -d and P + d represent the degraded and clean prompts, respectively. However, due to the gap between text and image modality within the feature space [41, 42] of the CLIP model, we cannot effectively obtain the degraded image feature by directly fusing the features from different modalities. To address this challenge, we propose an adaptive degradation adaptation strategy by 'stylizing' the text-driven degradation representation using the image feature. Inspired by AdaIN [43], we propose to align the mean and variance of T d to match those of the image feature, which can be formulated as follows:

where ˆ T d denotes the adapted degradation representation. Finally, we fuse the image feature with ˆ T d , and the feature space text-driven degradation process can be represented as:

Our proposed degradation fusion method allows us to measure DDR across various types of degradation simply by modifying the text prompt, eliminating the need for handcrafted design processes. Additionally, our adaptation strategy enables the application of text-driven degradation to image features without adjusting any hyper-parameters. As shown in Fig. 2, in the LIVEitw [14] dataset, DDR with text-driven feature degradation method achieves a distribution similar to DDR with carefully adjusted optimal degradation level, demonstrating the flexibility of our method.

3.2 DDR as a Flexible Image Descriptor

By modifying the degradation type, DDR can capture different characteristics in natural images. We demonstrate this by measuring DDR with different degradation types across all images in the

High DDR

Low DDR

(a) Color

(b) Noise

(c) Blur

(d) Exposure

(e) Content

LIVEitw [14] dataset. Specifically, we set five pairs of prompts, representing five types of degradation, including color, noise, blur, exposure, and content. We employ a fixed prompt formatting for different types of degradation, as follows:

For example, when the degradation type is blur, the d -and d + are set as 'blurry' and 'sharp' respectively. The images with high and low DDR for each type of degradation are shown in Fig. 4. We observe a negative correlation between the DDR and the level of degradation within the image. For example, as demonstrated in Fig. 4(e), an image with a high DDR to content degradation retains clear content, while the corresponding image with a low DDR exhibits unrecognizable content.

To further quantify the correlation between DDR and other image characteristics, we calculate the Spearman's Rank Correlation Coefficient (SRCC) between DDR and four types of image characteristics. Specifically, we measure the complexity [25], colorfulness [23], and sharpness [44] of images in the CSIQ [45] dataset. Additionally, we use the Mean Opinion Score (MOS) of each image as its quality score. The results are presented in Tab. 2. It is interesting to note that there is a negative correlation between the complexity of an image and DDR. This suggests that more complex images are capable of enduring more degradation with a smaller degree of change in deep features. Furthermore, the DDR to color and blur degradations show the highest correlation with colorfulness and sharpness respectively. Overall, with different degradation types, DDR tends to emphasize different image characteristics. Therefore, DDR shows promise as a versatile image descriptor for diverse downstream tasks through simple prompt adjustments, including IQA and image restoration.

3.3 DDR as a Blind Image Quality Assessment Metric

DDR can function as an image quality descriptor. As shown in Tab. 2, there is a positive correlation between the quality score and the DDR of the image. In cases where image quality is predominantly affected by a specific degradation type, an image with a high DDR to this degradation would likely

obtain a higher quality. For instance, in Fig. 4, when the degradation type is blur, comparing the image with a high DDR to the image with a low DDR, it is evident that the former exhibits a sharper content with less blur. However, real-world images often feature a mix of degradations. To evaluate image quality in such scenarios, we formulate a set of degradations denoted as D and compute the mean DDR for each degradation in D . The blind image assessment metric based on DDR can thus be formulated as follows:

3.4 DDR as an Unsupervised Learning Objective

We can also utilize DDR as a learning objective in image restoration tasks, where the goal is to train a deep learning-based restoration model to predict a clean image from a degraded one. This is achieved by optimizing the restoration model to minimize the reconstruction loss function, which quantifies the difference between the pixel values of the model's output and the ground truth. In this work, we demonstrate that incorporating DDR as an external unsupervised learning objective can improve the optimization of the restoration models. Specifically, we measure the DDR of the model output and aim to simultaneously minimize the reconstruction loss while maximizing the DDR. The learning objective of the image restoration model is thus formulated as:

where R θ ( · ) is a restoration model parameterised by θ , and L rec ( · , · ) denotes the reconstruction loss, λ d is the weight of DDR in learning objective. Similarly, by adjusting the degradation prompt and combining different types of degradation, we can tailor the approach to various restoration tasks.

4 Experiments

4.1 Experiment Setting

To demonstrate the versatility and efficacy of DDR as an image descriptor, we conduct comprehensive experiments covering (1) opinion-unaware blind image quality assessment (OU-BIQA), which does not require training model with human-labeled Mean Opinion Score (MOS) values, and (2) image restoration tasks, including image deblurring and real-world image super-resolution.

Implementation Details. For different tasks, we tailor the degradation set D in Eq. 7 and Eq. 8 to focus on distinct image attributes. Specifically, in BIQA, we define D = { color , noise , blur , exposure } . Meanwhile, for image restoration tasks, we set D = { color , content , blur } . In all experiments related to image restoration, we empirically set the weight of DDR in the learning objective in Eq. 8 as λ d = 2 . 0 . We use the CLIP [37] ViT-B/32 model as the image feature extractor, and employ the cosine distance to quantify the disparity between original and degraded image features, which can be defined as follows:

Baseline Datasets. To evaluate the effectiveness of the proposed DDR as an image quality descriptor, we conduct extensive experiments on eight public IQA datasets, including CSIQ [45], TID2013 [46], KADID [47], KonIQ [48], LIVE in-the-wild [14], LIVE [21], CID2013 [49], and SPAQ [50], which encompass both synthetic and real-world degradation scenarios. For image deblurring, we train and test the model using the GoPro dataset [51] and RealBlur dataset [52], respectively. The GoPro dataset [51] consists of synthetic blurred images, while RealBlur [52] contains images with real-world motion blur. For SISR, we combine two real-world datasets together for training and testing, including the RealSR [53] and City100 [54] datasets.

Baseline Methods. For the OU-BIQA task, we compare DDR with representative and state-of-the-art opinion-unaware BIQA (OU-BIQA) methods, which do not require training with human-labeled MOS. The compared methods include NIQE [26], QAC [55], PIQE [56], LPSI [57], ILNIQE [15], diqIQ [58], SNP-NIQE [59], NPQI [60], and ContentSep [61]. Among all compared methods, DDR

is the only zero-shot method that does not require any training. For image restoration, we compare our proposed method, as illustrated in Eq. 8, with a combination of reconstruction loss and feature domain loss L f ( · , · ) , which quantifies the distance between deep features extracted from images. Generally, the reconstruction loss is combined with feature domain losses to enhance the overall quality of the restored image, forming the learning objective of the restoration model R θ ( · ) as follows:

where λ f is the weighting factor for L f ( · , · ) . In all experiments on image restoration, we utilize PSNR loss as the reconstruction loss. We consider four types of representative feature domain losses for comparison, including LPIPS [3], CTX [2], PDL [7], and FDL [8]. To ensure a fair comparison, we set λ f = 0 . 1 for FDL [8] and λ f = 1 . 0 for the other feature domain losses, ensuring that the magnitudes of the different feature domain losses are in a similar range.

Moreover, to fully assess the robustness of our proposed DDR across vaious architectural models, we conduct all image restoration experiments using two representative image restoration models: NAFNet [62] and Restormer [63]. NAFNet [62] is a convolutional neural network (CNN)-based model, while Restormer [63] is a Transformer [64]-based model. These models have demonstrated impressive performance in their respective tasks and are widely recognized as representative models in recent years. We empirically train the model at a resolution of 128 × 128 . For the learning rate, we adhere to the official settings for NAFNet and Restormer. Specifically, the initial learning rate for NAFNet is set to 1 e -3 , and for Restormer, it is set to 3 e -4 . We also adopted a cosine annealing strategy for both models.

4.2 Opinion-Unaware Blind Image Quality Assessment

Tab. 3 presents the results across all datasets. Our proposed DDR consistently outperforms all competing methods on datasets with both synthetic [45-47] and in-the-wild [14, 49, 48, 50] degradation, underscoring its robustness across diverse degradation types. Especially its substantial improvement in SRCC on the LIVE in-the-wild dataset, rising from 0.5060 to 0.6613, showcasing the effectiveness of DDR as an image quality descriptor for images with real-world degradation. Furthermore, comparing the SRCC performance in Tab. 3 and Tab. 2, it is obvious that on the CSIQ dataset [45], DDRs that integrate multiple types of degradation perform significantly better than DDRs that only focus on a single type of degradation. This underscores the superiority of DDR as a more comprehensive image quality descriptor simply by integrating multiple types of degradation.

4.3 Image Motion Deblurring

The objective of image deblurring is to restore a high-quality image with clear details. The quantitative analysis in Tab. 4 illustrates that our proposed DDR surpasses all compared loss functions across datasets with both synthetic and real-world blur. Compared to optimizing solely the PSNR loss, our approach achieves a notable enhancement in PSNR, with an increase of at least 0.16 dB across all models and datasets. These results suggest that maximizing the DDR of the predicted image results in higher fidelity and reduced degradation. This is further evident in the qualitative results shown in Fig. 5, where the PSNR loss alone produces blurry textures, and combining PSNR with feature

Blur

GT

PSNR

+LPIPS

+CTX

+PDL

+FDL

+DDR(ours)

domain losses introduces noticeable artifacts. In contrast, incorporating DDR substantially reduces artifacts, yielding predicted images with sharper and more natural textures.

4.4 Single Image Super Resolution

SISR is a task aimed at enhancing the resolution of a low-resolution image to match or surpass the quality of a high-resolution counterpart. In our study, we evaluate our proposed DDR method against state-of-the-art loss functions. Tab. 5 showcases the quantitative results on a real-world dataset by two representative models (NAFNet and Restormer). Our findings reveal that our method outperforms all competing methods in terms of PSNR. Particularly noteworthy is the improvement achieved with NAFNet, where the incorporation of DDR alongside the reconstruction loss elevates the PSNR from 27.08 to 27.31. Additionally, as depicted in Fig. 6, our method yields visual results with finer texture compared to those optimized solely for PSNR or combined with LPIPS.

LR

PSNR

+LPIPS

+DDR(ours)

4.5 Ablation Study

For image deblurring, we conduct a series of ablation experiments on the NAFNet using the GoPro dataset, with all results detailed in Table 6. Firstly, we adjusted D to evaluate the effect of the degradation set defined in Eq. 8. Notably, a decrease in performance is observed when any type of degradation is removed, suggesting that combining multiple types of degradation results in a more comprehensive image description. Secondly, we investigate the effect of λ d in Eq. 8. Minor fluctuations in performance are observed when adjusting λ d to 1.0 and 3.0, indicating the robustness of our method to this hyper-parameter. Next, we explore the effect of the visual feature extractor. Increasing the scale of Φ v in DDR does not lead to improved performance, suggesting that a larger visual model may not necessarily enhance the ability to understand low-level texture. Finally, we examine the impact of the adaptation strategy in Eq. 4. A significant drop in performance is observed when the adaptation is removed, highlighting the critical role of this strategy in DDR calculation.

For opinion-unaware blind image quality assessment task, we conduct ablation experiment on four datasets. As demonstrated in Tab. 7, comparing with measuring DDR to single type of degradation, combining mutiple degradation types consistently leads to significant performance improvement. Furthermore, we can observe a performance boost by utilizing degradation adaptation strategy, improving SRCC from 0.6074 to 0.8289 on CSIQ dataset.

5 Limitations and Discussion

In this section, we discuss the limitations of DDR and provide potential solutions to address them.

The ability of the visual feature extractor to understand low-level degradation. We currently employ the CLIP model's visual feature extractor to facilitate text-driven degradation fusion. These feature extractors may incline to focus on high-level information such as image content, while their ability to understand low-level degradation may be limited. This could impact the measurement of the degradation response. In future work, we plan to fine-tune the feature extractor on tasks such

as degradation classification or description, to enable it to extract more fine-grained degradation features.

The selection of degradation prompts for different downstream tasks. The suitable degradation prompts may vary for different downstream tasks. In future work, we hope to append learnable tokens in the degradation prompts, and fine-tune these tokens to better adapt our method to different tasks. Specifically, we can utilize the strategy such as adversarial training, training DDR as a discriminator. This is an interesting and promising direction for our further investigation.

6 Conclusions

This paper introduces a flexible and powerful image descriptor, which measures the response of image deep features to degradation. We propose a text-driven approach to adaptively fuse degradation into image features. Experimental results demonstrate that DDR achieves state-of-the-art performance in blind image quality assessment task, and optimizing DDR results in images with reduced distortion and improved overall quality in image restoration tasks. We believe that DDR can facilitate a better understanding and application of image deep features.

Acknowledgments This work was supported in part by the National Natural Science Foundation of China under Grant 62201387, in part by the Shanghai Pujiang Program under Grant 22PJ1413300, and in part by the Fundamental Research Funds for the Central Universities.

References

Taimoor Tariq, Okan Tarhan Tursun, Munchurl Kim, and Piotr Didyk. Why are deep representations good perceptual quality features? In European Conference on Computer Vision , pages 445-461, 2020.

Roey Mechrez, Itamar Talmi, and Lihi Zelnik-Manor. The contextual loss for image transformation with non-aligned data. In European Conference on Computer Vision , pages 768-783, 2018.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 586-595, 2018.

Assaf Shocher, Yossi Gandelsman, Inbar Mosseri, Michal Yarom, Michal Irani, William T Freeman, and Tali Dekel. Semantic pyramid for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7457-7466, 2020.

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision , pages 694-711, 2016.

Keyan Ding, Kede Ma, Shiqi Wang, and Eero P Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(5):2567-2581, 2020.

Mauricio Delbracio, Hossein Talebei, and Pevman Milanfar. Projected distribution loss for image enhancement. In IEEE International Conference on Computational Photography , pages 1-12, 2021.

Zhangkai Ni, Juncheng Wu, Zian Wang, Wenhan Yang, Hanli Wang, and Lin Ma. Misalignment-robust frequency distribution loss for image transformation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2910-2919, 2024.

Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Focal frequency loss for image reconstruction and synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13919-13929, 2021.

Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2414-2423, 2016.

Idan Kligvasser, Tamar Shaham, Yuval Bahat, and Tomer Michaeli. Deep self-dissimilarities as powerful visual fingerprints. In Advances in Neural Information Processing Systems , volume 34, pages 3939-3951, 2021.

Moon Ye-Bin, Jisoo Kim, Hongyeob Kim, Kilho Son, and Tae-Hyun Oh. Textmania: Enriching visual feature by text-driven manifold augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2526-2537, 2023.

Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning , pages 6438-6447, 2019.

Deepti Ghadiyaram and Alan C Bovik. Massive online crowdsourced study of subjective and objective picture quality. IEEE Transactions on Image Processing , 25(1):372-387, 2015.

Lin Zhang, Lei Zhang, and Alan C Bovik. A feature-enriched completely blind image quality evaluator. IEEE Transactions on Image Processing , 24(8):2579-2591, 2015.

Sebastian Bosse, Dominique Maniry, Klaus-Robert Müller, Thomas Wiegand, and Wojciech Samek. Deep neural networks for no-reference and full-reference image quality assessment. IEEE Transactions on image processing , 27(1):206-219, 2017.

Jongyoo Kim and Sanghoon Lee. Fully deep blind image quality predictor. IEEE Journal of selected topics in signal processing , 11(1):206-220, 2016.

Jingtao Xu, Peng Ye, Qiaohong Li, Haiqing Du, Yong Liu, and David Doermann. Blind image quality assessment based on high order statistics aggregation. IEEE Transactions on Image Processing , 25(9): 4444-4457, 2016.

Weixia Zhang, Kede Ma, Jia Yan, Dexiang Deng, and Zhou Wang. Blind image quality assessment using a deep bilinear convolutional neural network. IEEE Transactions on Circuits and Systems for Video Technology , 30(1):36-47, 2018.

Shaolin Su, Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge, Jinqiu Sun, and Yanning Zhang. Blindly assess image quality in the wild guided by a self-adaptive hyper network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3667-3676, 2020.

Hamid R Sheikh, Muhammad F Sabir, and Alan C Bovik. A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Transactions on image processing , 15(11):3440-3451, 2006.

Ruth Rosenholtz, Yuanzhen Li, and Lisa Nakano. Measuring visual clutter. Journal of Vision , 7:17-17, 2007.

David Hasler and Sabine E Suesstrunk. Measuring colorfulness in natural images. In Human Vision and Electronic Imaging VIII , volume 5007, pages 87-95, 2003.

Martin Solli and Reiner Lenz. Color harmony for image indexing. In IEEE 12th International Conference on Computer Vision Workshops , pages 1885-1892, 2009.

Tinglei Feng, Yingjie Zhai, Jufeng Yang, Jie Liang, Deng-Ping Fan, Jing Zhang, Ling Shao, and Dacheng Tao. IC9600: a benchmark dataset for automatic image complexity assessment. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(7):8577-8593, 2022.

Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making a 'completely blind' image quality analyzer. IEEE Signal processing letters , 20(3):209-212, 2012.

Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In The AAAI Conference on Artificial Intelligence , volume 37, pages 2555-2563, 2023.

Mengmeng Zhu, Guanqun Hou, Xinjia Chen, Jiaxing Xie, Haixian Lu, and Jun Che. Saliency-guided transformer network combined with local embedding for no-reference image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1953-1962, 2021.

Boyun Li, Xiao Liu, Peng Hu, Zhongqin Wu, Jiancheng Lv, and Xi Peng. All-in-one image restoration for unknown corruption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17452-17462, 2022.

Vaishnav Potlapalli, Syed Waqas Zamir, Salman H Khan, and Fahad Shahbaz Khan. Promptir: Prompting for all-in-one image restoration. Advances in Neural Information Processing Systems , 36, 2024.

Dasong Li, Yi Zhang, Ka Chun Cheung, Xiaogang Wang, Hongwei Qin, and Hongsheng Li. Learning degradation representations for image deblurring. In European Conference on Computer Vision , pages 736-753, 2022.

Longguang Wang, Yingqian Wang, Xiaoyu Dong, Qingyu Xu, Jungang Yang, Wei An, and Yulan Guo. Unsupervised degradation representation learning for blind super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10581-10590, 2021.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900, 2022.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning , pages 19730-19742, 2023.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35:23716-23736, 2022.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems , 36, 2024.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763, 2021.

Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. Q-bench: A benchmark for general-purpose foundation models on low-level vision. In The Twelfth International Conference on Learning Representations , pages 1-27, 2024.

Gihyun Kwon and Jong Chul Ye. Clipstyler: Image style transfer with a single text condition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1806218071, 2022.

Tsu-Jui Fu, Xin Eric Wang, and William Yang Wang. Language-driven artistic style transfer. In European Conference on Computer Vision , pages 717-734, 2022.

Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems , 35:17612-17625, 2022.

Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P Calmon, and Himabindu Lakkaraju. Interpreting clip with sparse linear concept embeddings (splice). arXiv preprint arXiv:2402.10376 , 2024.

Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1501-1510, 2017.

Niranjan D Narvekar and Lina J Karam. An improved no-reference sharpness metric based on the probability of blur detection. In Workshop on Video Processing and Quality Metrics , volume 6, 2010.

Eric C Larson and Damon M Chandler. Most apparent distortion: full-reference image quality assessment and the role of strategy. Journal of Electronic Imaging , 19:011006-011006, 2010.

Nikolay Ponomarenko, Lina Jin, Oleg Ieremeiev, Vladimir Lukin, Karen Egiazarian, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Federica Battisti, et al. Image database TID2013: Peculiarities, results and perspectives. Signal processing: Image communication , 30:57-77, 2015.

Hanhe Lin, Vlad Hosu, and Dietmar Saupe. KADID-10k: A large-scale artificially distorted iqa database. In 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX) , pages 1-3. IEEE, 2019.

Vlad Hosu, Hanhe Lin, Tamas Sziranyi, and Dietmar Saupe. KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment. IEEE Transactions on Image Processing , 29:4041-4056, 2020.

Toni Virtanen, Mikko Nuutinen, Mikko Vaahteranoksa, Pirkko Oittinen, and Jukka Häkkinen. CID2013: A database for evaluating no-reference image quality assessment algorithms. IEEE Transactions on Image Processing , 24(1):390-402, 2014.

Yuming Fang, Hanwei Zhu, Yan Zeng, Kede Ma, and Zhou Wang. Perceptual quality assessment of smartphone photography. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3677-3686, 2020.

Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3883-3891, 2017.

Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho. Real-world blur dataset for learning and benchmarking deblurring algorithms. In European Conference on Computer Vision , pages 184-201, 2020.

Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, and Lei Zhang. Toward real-world single image super-resolution: A new benchmark and a new model. In Proceedings of the IEEE/CVF international conference on computer vision , pages 3086-3095, 2019.

Chang Chen, Zhiwei Xiong, Xinmei Tian, Zheng-Jun Zha, and Feng Wu. Camera lens super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1652-1660, 2019.

Wufeng Xue, Lei Zhang, and Xuanqin Mou. Learning without human scores for blind image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 995-1002, 2013.

P Ganesan, BS Sathish, K Vasanth, M Vadivel, VG Sivakumar, and S Thulasiprasad. Color image quality assessment based on full reference and blind image quality measures. In Innovations in Electronics and Communication Engineering , pages 449-457, 2020.

Qingbo Wu, Zhou Wang, and Hongliang Li. A highly efficient method for blind image quality assessment. In IEEE International Conference on Image Processing , pages 339-343, 2015.

Kede Ma, Wentao Liu, Tongliang Liu, Zhou Wang, and Dacheng Tao. dipIQ: Blind image quality assessment by learning-to-rank discriminable image pairs. IEEE Transactions on Image Processing , 26(8): 3951-3964, 2017.

Yutao Liu, Ke Gu, Yongbing Zhang, Xiu Li, Guangtao Zhai, Debin Zhao, and Wen Gao. Unsupervised blind image quality evaluation via statistical measurements of structure, naturalness, and perception. IEEE Transactions on Circuits and Systems for Video Technology , 30(4):929-943, 2019.

Yutao Liu, Ke Gu, Xiu Li, and Yongbing Zhang. Blind image quality assessment by natural scene statistics and perceptual characteristics. ACM Transactions on Multimedia Computing, Communications, and Applications , 16(3):1-91, 2020.

Nithin C Babu, Vignesh Kannan, and Rajiv Soundararajan. No reference opinion unaware quality assessment of authentically distorted images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2459-2468, 2023.

Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In European Conference on Computer Vision , pages 17-33, 2022.

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5728-5739, 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2020.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600-612, 2004.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2017.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations , 2015.

A Appendix

A.1 Detailed Experiment Settings

A.1.1 Evaluation Metrics.

For BIQA, we select SRCC as evaluation metric to measure the correlation between predicted quality score and human opinion. For image restoration, we choose PSNR and Structural Similarity (SSIM) [65] as evaluation metrics to measure the fidelity and structure similarity of restored images, respectively.

A.1.2 Training Settings

For all experiment on image restoration, we employ AdamW [66] optimizer and set the batch size as 4. We train NAFNet [62] and Restormer [63] for 200,000 and 300,000 steps respectively, following their corresponding official settings. All experiments are conducted using one NVIDIA RTX 4090. Each setting in image restoration costs about 20 to 30 hours for training.

A.1.3 Explanation of Fig. 2

We measure the amount of images with different Degradation Response (DDR) values to represent the distribution of DDR. Specifically, we divided the range of DDR into multiple intervals. For each point on the curve, we measured the number of images whose DDR values fall within the corresponding interval. The horizontal axis in Fig. 2 represents the numerical values of DDR, while the vertical axis represents the number of images. By adjusting the levels of handcrafted degradation, DDR demonstrates varying performance on the Opinion-Unaware Blind Image Quality Assessment (OU-BIQA) task. We conducted experiments across a range of degradation levels, selecting the level with the best performance on OU-BIQA as "optimal". "Low" and "high" represent the lowest and highest degradation levels, respectively. When the degradation level is too low, there is only a subtle difference between the F and F d for all images. In contrast, when the degradation level is too high, most images demonstrate an overly strong response. Using our text-driven "adaptive" strategy, DDR demonstrates a similar value distribution and performance to the manually set "optimal" degradation level. This result shows the effectiveness and flexibility of the proposed method.

A.1.4 Degraded and Positive Prompt Pairs

Full degraded and positive prompt pairs are shown in Tab. 8. We set each prompt following the format in Eq. 6.

Standard Deviation

Mean

0.40

0.35

0.30

0.30

0.25

color

0.25

0.20

noise

0.20

blur

R

S

0.15

content

0.15

0.10

exposure

0.10

0.05

0.05

0.00

0.00

relu1_2

relu2_2

relu4_3

relu5_3

relu1_2

relu2_2

relu4_3

relu5_3

Layers

Layers

A.2 Additional Experiment Results

Here we present additional experiment results. Firstly, statistics of deep features is known to be correlated with multiple image characteristics such as texture and style [6, 10]. Therefore, it is

Degradation Type

Degraded Prompt

color

A

unnatural color

photo with low-quality.

noise

A

noise degraded

photo with low-quality.

blur

A

photo with low-quality.

exposure

A

unnatural exposure

photo with low-quality.

content

A

bad content

photo with low-quality.

Degradation Type

Positive Prompt

color

A

real color

photo with high-quality.

noise

A

clean

photo with high-quality.

blur

A

sharp

photo with high-quality.

exposure

A

natural exposure

photo with high-quality.

content

A

clear content

photo with high-quality.

interesting to investigate the correlation between DDR and deep feature statistics. Specifically, we extract features from five layers ( Relu _ 1 _ 1 , Relu _ 2 _ 1 , Relu _ 3 _ 1 , Relu _ 4 _ 1 , and Relu _ 5 _ 1 ) of pretrained VGG [67] network, and measure the mean and standard deviation of extracted features. Then, we utilize SRCC to quantify the correlation between DDR and these statistics. The results are shown in Fig. 7. We can observe that DDR to color degradation demonstrates similar correlation to statistics of feature from every layers. While for exposure and blur degradation, DDR shows significant higher correlation to mean and standard deviation of feature from Relu _ 1 _ 1 than subsequent layers. In contrast, for noise and content degradation, DDR shows higher correlation for Relu _ 1 _ 1 and Relu _ 5 _ 1 .

Secondly, we present additional qualitative results for image restoration tasks. Fig. 9 and Fig. 8 show the results in image deblurring on the realBlur [52] dataset and GoPro [51] dataset respectively. Moreover, the qualitative results in SISR on real-world dataset [53, 54] are demonstrated in Fig. 10.

Blur

GT

PSNR

+LPIPS

+CTX

+PDL

+FDL

+DDR(ours)

Blur

GT

PSNR

+LPIPS

+CTX

+PDL

+FDL

+DDR(ours)

Blur

GT

PSNR

+LPIPS

+CTX

+PDL

+FDL

+DDR(ours)

Blur

GT

PSNR

+LPIPS

+CTX

+PDL

+FDL

+DDR(ours)

回回回

可克包

Blur

GT

PSNR

+LPIPS

+CTX

+PDL

+FDL

+DDR(ours)

Blur

GT

PSNR

+LPIPS

+CTX

+PDL

+FDL

+DDR(ours)

616-

8

口

LR

PSNR

+LPIPS

+DDR(ours)

LR

PSNR

+LPIPS

+DDR(ours)

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have a 'Limitations" section in the supplementary material.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our paper clearly states all the information needed to reproduce the main experimental results of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will release our code after the paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all the training and test details in the main paper and supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our paper does not report error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Our paper provides sufficient information on computer resources in supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of this work because our work is a foundational research.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We submit a clean code of our work.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a low-level visual descriptor with text embeddings and explored its application on many low-level vision tasks.\n\nSTRENGTHS:\n1. The experiment is sufficient and detailed.\n2. The paper is well-written.\n\nWEAKNESSES:\n**Weaknesses of the methods.**\n\n1. This paper uses a text-based model to describe the low-level visual information of the images, which seems to be similar to Q-Bench [1]. I suggest that the author compare it with it and emphasize the differences between the method proposed and Q-Bench.\n2. Simple super-resolution networks can identify different low-level visual information [2,3], such as degradation types. The author should compare with this method. Also, the description of lines 69 and 70 could be appropriately modified based on this.\n3. The network settings for single-image super-resolution (SISR) are somewhat unreasonable. NAFNet and Restormer have a lot of downsampling and are not suitable as base models for SISR experiments. I suggest to use RCAN, SwinIR, and HAT.\n\n[1] Wu H, Zhang Z, Zhang E, et al. Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision.\n\n[2] Liu Y, Liu A, Gu J, et al. Discovering distinctive\" semantics\" in super-resolution networks.\n\n[3] Liu Y, He J, Gu J, et al. Degae: A new pretraining paradigm for low-level vision.\n\nQUESTIONS:\n1. The order of the figures and tables in this paper seems a little confusing and should be more carefully formatted. For example, the experimental results of SISR are not introduced until the end of page 8, but the result images are shown at the beginning of page 8.\n2. The experimental settings are not fully given, such as the learning rate and training resolution, which may lead to unfair comparison. For example, SISR models are often trained on 48 $\\times$ 48 resolution, are the NAFNet and Restormer also trained on 48 resolution for SISR tasks in this paper?\n\nLIMITATIONS:\nPlease see the weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Deep Degradation Response (DDR), a method to quantify changes in image deep features under varying degradation conditions. DDR facilitates flexible and adaptive degradation through text-driven prompts. It reports to excel in blind image quality assessment and image restoration tasks like deblurring and super-resolution. The paper compares the proposed DDR with the existing techniques across multiple datasets. The authors plan to release their code for public use.\n\nSTRENGTHS:\n1. DDR demonstrates effectiveness across multiple applications, including Blind Image Quality Assessment (BIQA) and image restoration. Its adaptability to different degradation scenarios is a significant advantage.\n2. The text-driven approach allows DDR to adjust degradation levels based on specific requirements, making it versatile and applicable to various use cases.\n\nWEAKNESSES:\n1. The performance on BIQA is not particularly competitive, and some of the latest IQA metrics, such as LIQE, UNIQUE, and TreS, are not included for comparison. Please refer to the paper “Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective”.\n2. In Equation 8, DDR is calculated using the restored image and its corresponding degradation. According to the results presented in Table 4, the proposed DDR performs better than PSNR+LPIPS. Since LPIPS directly minimizes the feature difference between the restored image and the original image, it should theoretically be more effective for image restoration. The paper does not explain why DDR outperforms the LPIPS loss. Moreover, in Table 6, the pre-trained model also influences the performance of DDR. Does this imply that the gains achieved by DDR might be due to a stronger backbone in comparison to LPIPS?\n3. The paper utilizes DDR as an image quality assessment metric but does not provide an explanation for why it could represent the quality of an image.\n\nQUESTIONS:\nGiven that the negative and positive values are fixed, T_d)is a constant vector for each degradation. DDR seems to measure the disparity after adding \\hat{T_d} to the feature. Does a high DDR indicate that the image distribution is robust to the feature interference caused by T_d?\n\nLIMITATIONS:\nThe authors address the limitations and potential societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a feature descriptor to assess low-level image quality degradations. Based on CLIP, the proposed method first encodes input image and its degraded version to features in CLIP space; the input image is encoded by CLIP image encoder, and the degraded image feature is generated by adding a textual feature of degradation to the image feature. The Deep Degradation Response (DDR) is measured by calculating the distance (seems that cosine is used) of two features. The authors demonstrated the effectiveness of the proposed descriptor with extensive experiments and analysis including SRCC test and applications to image deblurring and super-resolution.\n\nSTRENGTHS:\n1. The paper proposes to exploit CLIP feature to measure image quality under various degradations. Unlike other image quality works focusing on image-based approaches, this paper introduces a novel approach using textual features.\n2. Surprisingly, it seems that the proposed method works well without training CLIP with degradation prompts (I need clarification of it in Weakness). It demonstrates that CLIP can be used as a tool for image quality assessment.\n\nWEAKNESSES:\n1. Clarity: Some technical details are unclear, which limited the understanding of the proposed method.\n- I did not understand the distribution of DDR in Figure 2. Why is the distribution of \"adaptive\" better than \"low\" and \"high\"? What did the authors do specifically for \"optimal\"?\n- The authors mention in L84-85 that there are options in the disparity metric M, but in SRCC evaluation there exist a clear positive or negative direction of correlation. Since Ln and cosine metrics have different meaning for low/high values, more clarification is needed in the description of the method. It seems that the authors used cosine metric in their code.\n- It seems that the method simply used pre-trained CLIP features for both images and texts. Is it correct?\n- Among degradation types, Color and Content seem ambiguous. Do the authors have a clear definition of these categories?\n2. Even though the DDR can score good and bad quality images in terms of (color, noise, blur, ...), it seems difficult to measure the amount of degradation such as noise level. It makes me wonder if the DDR contains useful information to measure degradations, or simply focus on visually pleasing images because positive texts usually correspond to those images? This is a reasonable question as the authors are aware of the fact that the CLIP features may have bias to high-level features rather than low-level degradations.\n3. Continuing 2., although the authors framed the paper as image descriptor for degradations, I think the paper is more relevant to non-reference image quality assessment. Therefore, more prior works in this line of research need to be discussed in Sec. 2.\n4. Although it is mentioned in the limitation, the proposed method is evaluated on one type of sentence per each degradation. Can the authors justify why they chose these specific words? Due to the nature of texts, there would be similar words that share similar meanings. How robust is the proposed method against the choice of words?\n\nQUESTIONS:\nThe paper generally addresses an interesting problem of image quality descriptor using CLIP textual features, and the results look encouraging. However, it lacks the in-depth understanding of the CLIP and proposed method. I would like to hear the authors' answers to the points raised in Weaknesses before adjusting my rating.\n\nLIMITATIONS:\nThe authors described two limitations of the proposed method. A potential negative societal impact would be\n- The proposed method could be potentially used to discriminate a certain group of photos (e.g. race, gender, etc) by including those words in degradation prompts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}