Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image-Quality Metrics

Authors: Aleksandr Gushchin, Khaled Abud, Georgii Bychkov, Ekaterina Shumitskaya, Anna Chistyakova, Sergey Lavrushkin, Bader Rasheed, Kirill Malyshev, Dmitriy S. Vatolin, Anastasia Antsiferova
Primary area: datasets and benchmarks
Keywords: adversarial defenses, image quality assessment, adversarial attacks, image quality metrics, benchmark

Abstract:
Most modern image-quality-assessment (IQA) metrics are based on neural networks, which makes the adversarial robustness of these metrics a critical concern. This paper presents the first comprehensive study of IQA defense mechanisms in response to adversarial attacks on these metrics. We systematically evaluated 29 defense strategies - including adversarial purification, adversarial training, and certified robustness - and applied 14 adversarial attack algorithms in both adaptive and nonadaptive settings to compare these defenses on nine no-reference IQA metrics. Our analysis of the differences between defenses and their applicability to IQA metrics recognizes that a defense technique should preserve IQA scores and image quality. Our proposed benchmark aims to guide the development of IQA defense methods and can evaluate new methods; the latest results are at link hidden for blind review.

Decision: reject
Reviewer scores: 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730349356118

thread_type:
official_review

<thread_object>
creator: reviewer_vz76

<content>
summary:
This paper proposes an empirical investigation of the effectiveness of various defense techniques against adversarial attacks on image quality metrics.

strengths:
1. The paper is decently written.
2. The problem under investigation is of practice importance as well described in the Introduction.
3. This empirical study is comprehensive.

weaknesses:
1. A primary technical concern is label preservation (quality preservation in our context) amidst adversarial perturbations. This research evaluates various adversarial attacks, including those like MADC by Wang and Simoncelli, which may not preserve image quality after attacks. In such instances, the model is expected to provide a different quality prediction for the manipulated image, necessitating ground-truth quality assessment through human evaluation.

2. From a conceptual perspective, the reviewer wonders how to understand certified defenses that involve voting and medianing. In classification, it is not hard to comprehend that the robustness is gained through prediction ensembling, which is again under the assumption of label preservation (see Eq. (4)). However, the quality preservation assumption is clearly not true for quality assessment. For example, consider a test image with some Gaussian blur, to perform random smoothing, we shall add Gaussian noise to it according to Eq. (4). Then, the final score is the average quality estimates of a Gaussian blurred image and a Gaussian blurred and noised image (which may be of different quality), which makes less sense to the reviewer.

3. The observed effectiveness of the adversarial attacks (evidenced by an SRCC decrease from 0.611 to 0.477 in Table 3) appears inconsistent with prior research such as [Zhang et al., 2022a] (which reduces to random guessing). Given the limited success of these attacks, interpreting the defense results with similar SRCC values (ranging between 0.5 and 0.6) becomes challenging.

4. Recent NR-IQA models that integrate visual and language components have not been evaluated in this study.

5. The focus of this empirical study aligns more closely with image processing journals rather than a machine learning conference like ICLR, given that no new theories and algorithms are developed.

questions:
The authors should work more on Points 1, 2, and 3 in an attempt to raise the reviewer's rating.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730428521326

thread_type:
official_review

<thread_object>
creator: reviewer_srtc

<content>
summary:
This paper systematically evaluates the effectiveness of various defense strategies for NR-IQA models, including adversarial purification, adversarial training, and certified robustness. It also examines these defenses under both adaptive and non-adaptive attack scenarios. Overall, the experiments in this paper are thorough and comprehensive, but the paper's readability could still be enhanced.

strengths:
1. The research topic of this paper is interesting and promising.
2. The experiments in this paper are comprehensive and detailed.

weaknesses:
1. Most of the strategies are directly borrowed from classification tasks, with no new approaches tailored to the specific characteristics of IQA tasks.
2. In addition to PSNR and SSIM, incorporating additional quantitative indicators like L_2 and L_\infty would provide a more intuitive understanding of the differences between the original image and the purified image.
3. The article is less readable, and many tables contain abbreviations that are not defined.

questions:
1. The definition of attack in Equation 1 focuses solely on increasing the model's score. However, if an image already possesses the highest quality, what is the purpose of the attack on it? Why was the idea of decreasing the score of high-quality images/videos not considered?
2. Currently, the attack methods employed are those typically used in classification problems. It would be beneficial to consider incorporating some of the attack strategies for IQA that have been proposed in recent years.
3. Table 3 shows that many adversarial purification defenses exhibit strong defensive effects. However, these methods should be analyzed more thoroughly. For instance, purification techniques that modify the entire image, such as color quantization and median blur, should include more detailed indicators (L_2 and L_\infty) to better reflect the extent of image modification.
4. Most of the analysis in this paper primarily describes the data presented in the tables. It would be beneficial to include an in-depth analysis of the characteristics and connections among these various types of defense strategies, providing great guidance for future research.
5. The abbreviations in the table should be added with full spelling in the caption to help readers understand and prevent misunderstandings. 
6. Equation 2 misses the variable x’.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730614756103

thread_type:
official_review

<thread_object>
creator: reviewer_rz8e

<content>
summary:
The paper attempts to set up benchmarking for adversarial attacks and defenses in the context of No-Reference Image Quality Assessment algorithms. The coverage of the work seems to be good—29 defense strategies, 14 attack methods, and 9 IQA methods. Lots of experiments (as needed) and results are provided, as expected from a benchmarking paper.

strengths:
Major strengths : 

1. The paper presents the first comprehensive benchmark of 29 defense methods,14 attack methods, and 9 IQA methods for NR-IQA
2. Multi-dimensional approach to evaluation: robustness, preserving correlation with human perception (SRCC/PLCC), and quality of the image with respect to original (PSNR, SSIM)
3. Practical Contribution to Research and Industry -> Good work on setting up a public benchmark.

weaknesses:
Major weaknesses :

1. The paper is a valuable resource for the IQA community. But in terms of technical novelty, it would have been nice to have an attack/defense method with some motivation for the IQA task. I feel this paper would have been more suitable for a Benchmarking Track, but I understand ICLR lacks such a track and had to submit it to the main conference track
2. The appendix provides many results, but it is very difficult to connect them to the main text, which points to the paper's poor organization.
3. The authors should add LPIPS to the results in Table 3 (and other similar tables) along with PSNR and SSIM. PSNR is not a perceptual metric and can be reliable, leaving SSIM as the only metric. It is better to report both SSIM and LPIPS scores. 

Minor :

1. Paper formatting needs to improve, and content organization can also be better. For example, Fig 1 does not discuss page 8.
2. Section 3.1 under Adversarial defenses: It is better to divide the paragraphs into different types. This will make reading the paper much easier.

questions:
1. In section 3.2, "clustering the KonIQA dataset by spatiotemporal complexity." - Could you please explain the temporal aspect of images?
2. Given the high computational demands of certified defenses and 10 images being used? How would you expect the results to vary as you sample different sets of 10 images?
3. Logistics questions on the leaderboard :
   How do you plan to maintain the leaderboard, and will there be mechanisms for incorporating new defense/attack techniques over time?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620326522

thread_type:
official_review

<thread_object>
creator: reviewer_rxlb

<content>
summary:
The paper aims to benchmark and evaluate the robustness of 30 different adversarial defense methods against 14 adversarial attacks regarding IQA metrics. It emphasizes the need for effective defenses due to the unique challenges posed by preserving image quality while defending against adversarial perturbations. It presents a comprehensive analysis of the efficiency of various adversarial defense methods for image quality assessment (IQA) metric.

strengths:
(1) This paper gives a comprehensive comparison of multiple defense methods against IQA under a variety of attacks, and draws a few conclusions under different scenarios.
(2) The detailed analysis of the trade-offs between performance and robustness in various defense strategies offers practical guidance for researchers and developers.
(3) The inclusion of statistical tests and evaluations of quality scores adds robustness to the findings.

weaknesses:
(1)	Although the paper considers 30 different defense methods, it ignores some defense methods which are tailored for IQA methods specifically such as [1]. These methods should be discussed and compared in the paper, as the goal of this paper is to discuss the defense of IQA.
(2)	The paper has evaluated different defense methods under different indicators, showing a lot of charts, but it lacks the in-depth analysis about what is the reason behind the effectiveness of different defense methods which is important. For example, the defense performance on the KonIQ-1k dataset on the right of Figure 3 exceeds the other two datasets in multiple defense methods. What is the reason? Why do many attack methods in Figure 4 achieve the worst defense performance on FPR, in terms of R robustness?
(3)  More experimental details and analysis are expected. For example, in line 227, 50 images are selected from 1k images for attack, do different selections of attack images affect the performance of attack and defense? Does it affect the conclusion?

[1] Defense Against Adversarial Attacks on No-Reference Image Quality Models with Gradient Norm Regularization, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, June 2024

questions:
Please see Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732484838300

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Response

comment:
Dear Reviewers,
We sincerely thank you all for your thorough and constructive feedback. We appreciate the time spent reviewing our work.
We are grateful for the positive reception of our work. Multiple reviewers highlighted our study’s practical contribution to research and industry and its detailed and comprehensive analysis.

To address the main concerns raised in the reviews, we have made several improvements.
* Following the _RxLB_'s suggestion, we have evaluated defense method [1] and included its results in Figure 1 and Table 4. 
* We have conducted additional experiments to evaluate the impact of sampling 50 images from the dataset. The results of these experiments are included in Section A.2 in the Appendix. 
* Following the _sRtC_'s suggestion, we have added $MSE$ and $L_{inf}$ metrics to Table 3 to better measure effects on image quality.
* To enhance the readability of the Appendix, we have added section A.0 with a brief Appendix structure.
* We also have corrected some mistakes and typos pointed out by reviewers.
Thank you again for your valuable feedback!

[1] Defense Against Adversarial Attacks on No-Reference Image Quality Models with Gradient Norm Regularization, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, June 2024
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors present a comprehensive study benchmarking the robustness of 29 defense methods for 14 adversarial attacks against 9 different image quality assessment (IQA) metrics. While the study is comprehensive and of value to the IQA community, there is too little new insight generated to be a good fit for ICLR.

It is recommended to consider sending the work to an image processing journal or to a benchmarking track of a major conference.