Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image-Quality Metrics

## Abstract

Abstract In the field of Image Quality Assessment (IQA), the adversarial robustness of the metrics poses a critical concern. This paper presents a comprehensive benchmarking study of various defense mechanisms in response to the rise in adversarial attacks on IQA. We systematically evaluate 25 defense strategies, including adversarial purification, adversarial training, and certified robustness methods. We applied 14 adversarial attack algorithms of various types in both non-adaptive and adaptive settings and tested these defenses against them. We analyze the differences between defenses and their applicability to IQA tasks, considering that they should preserve IQA scores and image quality. The proposed benchmark aims to guide future developments and accepts submissions of new methods, with the latest results available online: https://videoprocessing.ai/benchmarks/iqa-defenses.html.

## 1 Introduction

Image quality assessment (IQA) metrics are essential for developing and evaluating image and video processing algorithms. Modern IQA metrics often utilize neural networks, allowing them to correlate highly with subjective quality assessments. However, neural networks are known to be vulnerable to input perturbations. Recent studies have explored the robustness of IQA metrics [, , , ], revealing that existing IQA metrics are not robust to adversarial attacks. Several scenarios exist where IQA metrics might be attacked intentionally or not. For instance, optimizing image quality for an IQA metric can reduce image quality []. In Super-Resolution algorithms, including IQA metrics in the loss function can generate visual artifacts  []. Intentional adversarial attacks on IQA metrics can manipulate search results, as some search engines, like Microsoft Bing, use IQA metrics to rank outputs []. Furthermore, IQA metrics are frequently used in public benchmarks to evaluate image processing algorithms (e.g., image compression, denoising). Algorithm developers can exploit vulnerabilities in IQA metrics to artificially inflate the perceived quality of their algorithms. An example is the libaom video compression codec, which includes a “–tune=vmaf” option to increase VMAF scores on compressed videos through sharpening filters [].

The researchers proposed different defense methods to increase the robustness of neural networks in various computer vision tasks, with several benchmarks available for object classification defense methods [, ]. However, only a few defense methods exist for IQA metrics. Compared to defenses for computer vision models, which only need to provide a correct class label, defending IQA metrics is more challenging. A successful defense for IQA metrics must meet two criteria: restoring the original IQA scores and the perceptual quality of the original image, which may have already been distorted before the attack. Given the variety of existing defense methods for computer vision tasks, it is reasonable to evaluate their effectiveness in defending IQA metrics.

This paper introduces the first benchmark that systematically evaluates defenses against adversarial attacks on IQA metrics, helping the community maintain a comprehensive view of the landscape of IQA defenses. An extensive benchmark enables developers and researchers to measure and compare the efficacy of different defense methods within a unified framework. It also promotes research transparency and continual improvement while encouraging the development of more advanced defenses. Our contributions include a new methodology for comparing the efficiency of IQA metrics’ adversarial defenses, a dataset of adversarial images for evaluating non-adaptive defenses, comprehensive experiments, an in-depth analysis of the results, and an online leaderboard.
Our methodology is the first to systematize defense methods applied to IQA metrics. We analyzed 25 defense methods, including both empirical and certified approaches, and evaluated their efficiency against adaptive and non-adaptive adversarial attacks. The benchmark is published online (^1^11[https://videoprocessing.ai/benchmarks/iqa-defenses.html](https://videoprocessing.ai/benchmarks/iqa-defenses.html)) alongside with the dataset, and we provide code for the proposed methodology in the GitHub repository (^2^22[https://github.com/msu-video-group/adversarial-defenses-for-iqa](https://github.com/msu-video-group/adversarial-defenses-for-iqa)).
We welcome and encourage submissions of new defense methods to enhance the benchmark continuously.

Figure: Figure 1: Adversarial defenses efficiency for IQA metrics in terms of (a) $D_{score}^{(D)}$ and (b) $SROCC_{adv}$. Bars and dots are for non-adaptive and adaptive attacks, respectively. We average the results by 7 IQA metrics and 14 attacks. For all defenses parameter sets with highest $SROCC_{adv}$ are selected.
Refer to caption: extracted/5771534/imgs/spider_plot.png

## 2 Related work

IQA metrics fall into full-reference (FR) and no-reference (NR). FR metrics evaluate the similarity between a reference image and its degraded version. NR metrics evaluate the quality of only a degraded image. NR metrics have broader applications than full-reference ones and are easier to use. Typically, NR metrics correlate less with subjective quality scores, although new neural-networks-based NR metrics successfully compete with FR metrics [].

Adversarial examples are samples that fool neural networks when the correct answer is obvious to humans. The most popular adversarial attacks generate high-frequency noise [, ]. An attack is referred to as restricted if a generated adversarial noise is norm-bounded and unrestricted otherwise. Adversarial examples generated by unrestricted attacks, for example, by semantic manipulation or color-filtering [, ], may look natural and non-suspicious to the human eye even with large and visible perturbations.
Adversarial attacks fall into categories white-box and black-box categories according to the attacker’s knowledge about the model. White-box attacks utilize gradients of the attacked models. However, there are situations where gradients are unavailable. In this case, black-box attacks remain applicable. Most of the existing adversarial attacks in the image domain were designed for image classification. Several white-box adversarial attacks [, , , , ] and at least one black-box attack [] have been proposed for IQA metrics.

Defense methods for neural networks fall into two main categories: certified and empirical. Certified methods provide deterministic or probabilistic robustness guarantees for particular types of perturbations, datasets, or model architectures. However, these methods are usually computationally complex and reduce the model’s general accuracy. The most well-known certified method is randomized smoothing []. Later, other variations have been proposed in  [, ]. Empirical methods do not provide robustness guarantees but require fewer computational resources. One of the most widely used empirical defense methods to improve the robustness of machine learning models is adversarial training [, ]. It involves generating new adversarial examples during training and updating the model’s weights based on these examples. This approach requires more computational time, but it creates a more robust model that is less vulnerable to attacks. However, vanilla adversarial training may lead to decreased model performance. Adversarial purification [] is an empirical method that removes adversarial perturbation from input data. Unlike adversarial training, this method does not change the neural network model but preprocess input data. Adversarial purification is model-agnostic and computationally efficient, however, it may not entirely eliminate advanced adversarial perturbations and can sometimes degrade image quality. Such methods include compression, spatial transformations, blurring, denoising, and other preprocessing techniques.

There are several benchmarks of adversarial attacks and defense efficiency in applications for object classification. The biggest are RobustBench [] with 95 robust models and Adversarial Robustness Benchmark [] with 13 defenses. In the field of image quality assessment, currently, there are no benchmarks for defense methods against adversarial attacks. MSU Metrics Robustness Benchmark [] compares IQA metrics’ robustness but without applying any defense methods.

## 3 Methodology

### 3.1 Problem definition

Adversarial attacks. In this work, we evaluate adversarial defenses within the NR IQA domain, because NR metrics are more susceptible to adversarial attacks than FR ones. Therefore, an attacked model, represented by an NR IQA metric takes a single image as an input and outputs an estimated image quality. Formally, the NR IQA metric is the mapping $f_{\omega}:X\rightarrow\mathbb{R}$ parameterized by the vector of weights $\omega$. Here, $X\subseteq[0,255]^{3\times H\times W}$ is the set of input images.

An adversarial attack $A:X\rightarrow X$ is the perturbation of the input image defined as:

$$ $\displaystyle A(x)=\underset{x^{\prime}:\rho(x^{\prime},x)\leq\varepsilon}{ \arg\max}\>L(f_{\omega}(x^{\prime})),$ (1) $$

where $L$ is a loss function that represents the model’s outputs on perturbed images, and $\rho$ is the distance function defined on $X$. We choose to increase the model’s values during the attack process since it reflects the motivation to conduct attacks in real-life applications.
For attacking IQA metrics, we define $L(x^{\prime},f_{\omega})=\frac{f_{\omega}(x^{\prime})}{\text{diam}(f_{\omega})}$, where $\text{diam}(f_{\omega})=\underset{x,z\in X}{\sup}\{|f_{\omega}(x)-f_{\omega}(z
)|\}$ represents the range of IQA metric’s values. The supplementary material contains ranges for IQA metrics used in this work.

Adversarial defenses. In our work, we consider three types of adversarial defenses.

Adversarial purification is an algorithm $P:X\rightarrow X$ that transforms the input image as follows:

$$ $\displaystyle P(x^{\prime})=\underset{x^{\prime\prime}\in X}{\arg\min}\>\left| f_{\omega}(x^{\prime\prime})-f_{\omega}(x)\right|+\lambda\rho(x^{\prime\prime} ,x),$ (2) $$

where $\lambda$ is the regularization term.

Adversarial training is formulated as the following min-max problem:

$$ $\displaystyle\min\limits_{\omega}\mathbb{E}_{(x,y)\sim\mathcal{D}}\Big{[}\max \limits_{\|\delta\|_{p}\leq\varepsilon}\mathcal{L}(f_{\omega}(x+\delta),y)\Big {]},$ (3) $$

where $\mathcal{D}$ is distribution of training data, $\mathcal{L}$ is a training loss function and $\varepsilon$ is allowable attack magnitude. In practice, instead of internal maximization, an adversarial attack is used.

Certified methods used in our paper are based on randomized smoothing [], denoised randomized smoothing [], diffusion-based randomized smoothing [, ] and median smoothing []. A basic idea behind randomized smoothing is the replacement of the original IQA metric with its smoothed version:

$$ $g(x)=\underset{\epsilon\sim\mathcal{N}(0,\sigma^{2})}{\mathbb{E}}f_{\omega}(x+\epsilon)$ (4) $$

where $\epsilon$ is a centered Gaussian random variable.

### 3.2 Adversarial attacks

This work considers two scenarios for adversarial attacks: non-adaptive and adaptive. In the first case, we generate a dataset of adversarial images using the
methodology described further. We integrate differentiable defenses into the attacked IQA metric in the second case. This setup allowed the adaptive attacks to access the IQA metrics’ and defense methods’ gradients, thereby simulating a more challenging defense case.

We selected several white-box attacks, both restricted and unrestricted, from the IQA robustness benchmark []. Table 1 briefly describes the selected attacks. To account for different possible scales of adversarial attacks, we executed each method with three distinct sets of hyperparameters corresponding to “weak”, “medium”, and “strong” attack scenarios. Figure 2 illustrates the procedure for selecting parameter values for these adversarial parameter sets. Supplementary material provides additional details.

**Table 1: List of adversarial attacks employed in our benchmark. WB and BB are white-box and black-box attack types. We adjust varied parameters to align attacks’ strengths and apply attacks as procedures.**
| Adversarial attack | Type | Restriction | Varied parameter | Short description |
| --- | --- | --- | --- | --- |
| I-FGSM [] | WB | $l_{\infty}$ | lr | Gradient descent to increase IQA metric |
| Optimised-UAP [] | WB | $l_{\infty}$ | amplitude | Universal perturbation generated with gradient descent |
| Korhonen et al. [] | WB | $l_{\infty}$ | lr | Sobel-filter-masked gradient descent |
| Zhang et al. [] | WB | $l_{\infty}$ | lr | Gradient descent with keeping FR metric (DISTS) |
| MADC [] | WB | $l_{\infty}$ | lr | Gradient projection onto a proxy FR metric (MSE) |
| cAdv [] | WB | SSIM | lr | Grad. descent with recolorization model |
| SSAH [] | WB | $l_{\infty}$ | lr | Grad. desc. with high-frequency distance minimization |
| stAdv | WB | $l_{\infty}$ | lr | Perturbs images spatially through a flow field |
| FACPA [] | WB | $l_{\infty}$ | amplitude | Perturbations generated using U-Net |
| NES [] | BB | $l_{\infty}$ | $\epsilon$ | Gradient descent with approximated gradient |
| Parsimonious [] | BB | PSNR | $\epsilon$ | Creating perturbations using discrete optimization |
| One Pixel [] | BB | $l_{0}$ | pixel count | Perturbing fixed number of pixels with diff. evolution |
| Square [] | BB | $l_{\infty}$ | $\epsilon$ | Square-formed perturbations created with random search |
| Patch-RS [] | BB | PSNR | $\epsilon$ | Adversarial patch found with random search |

Figure: Figure 2: Schematic procedure for selecting adversarial attacks’ parameters.
Refer to caption: extracted/5771534/imgs/parameters.png

### 3.3 Defense methods

To thoroughly investigate the effectiveness of defenses on IQA metrics, we explore three types of methods: adversarial purification, adversarial training, and methods of certified robustness.

**Table 2: List of compared adversarial purification methods.**
| Defense method | Type | Varied parameter | Short description |
| --- | --- | --- | --- |
| JPEG [] | Compression | q | JPEG compression algorithm |
| DiffJPEG [] | Compression | q | Differentiable JPEG |
| Color quantization [] | Compression | npp | Reducing the number of colors |
| Resize [] | Spat. transform. | scale | Changing the image size |
| Bilinear Upscale | Spat. transform. | scale | Resizing and upscaling to original size |
| Rotate | Spat. transform. | angle lim. | Image rotation |
| Crop [] | Spat. transform. | size | Cropping the image |
| Flip | Spat. transform. | — | Mirroring the image |
| Gaussian blur | Blurring | kernel size | Smoothing with a Gaussian filter |
| Median blur | Blurring | kernel size | Smoothing with a median filter |
| Unsharp masking | Preprocessing | kernel size | Unsharp masking |
| MPRNet [] | Denoising | — | 3-stage CNN for deblurring & denoising |
| Real-ESRGAN [] | Denoising | — | GAN-based model for super-resolution |
| DiffPure [] | Defense | t | Diffusion model for adversarial purification |
| DISCO [] | Defense | — | Resists attacks with different local structure |
| FCN [] | Defense | — | Neural filter to counter color attack cAdv |
| Random noise | Adding noise | — | Adding random noise |

Adversarial purification
Table 2 describes the selected purification methods. We used five sets of parameters for methods with parameters to vary defense strength. The supplementary material provides more information on the choice of defense parameters.

Adversarial training. For the IQA task, there are additional challenges in applying this method since adversarial samples generated during the training process are of lower subjective quality. Labeling such images with new scores manually is impractical, and using ground-truth labels from images before an attack is inaccurate. To overcome these limitations, we selected 6 training configurations for our experiments: 2 score penalizing strategies (using SSIM or LPIPS metrics) and 3 different attack magnitudes $\varepsilon=\textit{\{2, 4, 8\}/255}$. To create the adversarial images during the training process, we applied a 2-step APGD attack [].

Certified defenses.
In this work we used certified defenses based on smoothing, which is performing multiple runs of the original model. These defenses can be easily implemented with any given NR image quality metric and do not restrict the model architecture. We consider both classification-task and regression-task defenses. All these defenses generate noisy variations of input images and then pass them through the model. Before passing through the model, some of these methods use denoising to improve accuracy. See more details in Table 3.
To turn a regression quality metric into a classifier, we linearly divided its score range ($\text{diam}(f_{\omega})$) into ten segments, each corresponding to a specific class. According to our experiments, ten segments showed optimal correlations with subjective scores. Additional classes were added for values below or above this range, resulting in twelve classes of image quality. Given an input image, the results of the classification-based certified method are the quality class and the certified radius $R$. The method guarantees that the class remains unchanged for the input image within a $l_{2}$ ball of radius $R$.
The results of a regression-based certified method are the metric score and the certified delta. The method guarantees that, within a $l_{2}$ ball of radius $\epsilon$, the metric score changes by no more than delta. To make this value comparable across metrics, we define the certified relative delta by dividing the certified delta by $\text{diam}(f_{\omega})$.

**Table 3: List of certified defense methods employed in our benchmark. C and R are classification and regression tasks.**
| Defense method | Task | Short description |
| --- | --- | --- |
| Random. Smoothing (RS) [] | C | Noisy samples $\rightarrow$ Classifier $\rightarrow$ Majority voting |
| Denoised Random. Smoothing (DRS) [] | C | Noisy samples $\rightarrow$ Denoiser $\rightarrow$ Classifier $\rightarrow$ Majority voting |
| Diffusion DRS (DDRS) [] | C | Noisy samples $\rightarrow$ One-step diffusion $\rightarrow$ Classifier $\rightarrow$ Majority voting |
| DensePure (DP) [] | C | Noisy samples $\rightarrow$ Multi-step diffusion $\rightarrow$ Classifier $\rightarrow$ Majority voting |
| Median Smoothing (MS) [] | R | Noisy samples $\rightarrow$ Regressor $\rightarrow$ Median of scores |
| Denoised Median Smoothing (DMS) [] | R | Noisy samples $\rightarrow$ Denoiser $\rightarrow$ Regressor $\rightarrow$ Median of scores |

### 3.4 Experimental setup

Datasets.
We use a subset of 1000 images at 512$\times$384 resolution from the freely available to the research community KONIQ-10K IQA dataset [] to evaluate adversarial defense methods. We chose it due to its large, diverse collection of real-world images with subjective quality scores. The original set was partitioned into 10 clusters using K-Means [] based on 3 parameters: Spatial Information (SI), Colorfulness (CF), and Mean Opinion Scores (MOS). We selected 100 random images from each cluster, resulting in a diverse set of test images regarding quality and content. Using these images as the source, we generated an adversarial dataset of over 215,000 images (1000 $\times$ number of attacks $\times$ number of attacked NR IQA metrics $\times$ number of attack hyperparameter sets). Due to significantly higher computational complexity, we used a smaller set of 50 images (5 per cluster) for black-box attacks. For the same reason, we used a smaller set of 10 images for certified defenses to generate attacks.

IQA metrics. We chose 7 IQA metrics for our experiments according to the results of IQA adversarial robustness benchmark []: META-IQA [], MANIQA [], KONCEPT [], SPAQ [], PAQ2PIQ [], LINEARITY [] and FPR []. These metrics employ different CNN-backbones and have a wide range of $R_{score}$. A more detailed description is provided in the supplementary material. Due to the computational complexity of the adversarial training, we selected only two CNN-based NR IQA metrics for experiments with this method: LINEARITY and KONCEPT. The choice is based on these metrics’ high correlation with subjective quality scores and high vulnerability to adversarial attacks.

### 3.5 Evaluation metrics

Disparity scores ($D_{score},D^{(D)}_{score}\downarrow$) are used to measure the attack’s efficiency under adversarial purification defense:

$$ $D_{score}=\frac{1}{n}\sum_{i=1}^{n}\frac{\left|f_{\omega}(P(x_{i}^{\prime}))-f _{\omega}(x_{i})\right|}{\text{diam}(f_{\omega})}\times 100\%;\ D_{score}^{(D) }=\frac{1}{n}\sum_{i=1}^{n}\frac{\left|f_{\omega}(P(x_{i}^{\prime}))-f_{\omega }(P(x_{i}))\right|}{\text{diam}(f_{\omega})}\times 100\%,$ (5) $$

where scores denoted with superscript ^D are calculated for purified source images.

Quality score ($Q_{score}\uparrow$) measures the perceptual distance between purified and source image:

$$ $Q_{score}=\frac{1}{n}\sum_{i=1}^{n}\left(SSIM(P(x_{i}^{\prime}),x_{i})+\frac{ PSNR(P(x_{i}^{\prime}),x_{i})}{40}\right),$ (6) $$

We chose a combination of SSIM [] and PSNR metrics as they are simple and robust. Note that this method does not properly reflect visual differences between images in case of spatial transformations like image flipping, cropping, or rotating.

Spearman Rank-Order Correlation Coefficient (SROCC) assess the performance of an IQA metric in the presence of adversarial defense. We measure the correlation between ground-truth image quality scores $y$ and the IQA metric’s predictions. $SROCC_{clear}$ is calculated for purified non-adversarial images, while $SROCC_{attacked}$ is measured for purified adversarial images. These coefficients represent scenarios with and without using some detection method to identify adversarial attacks before passing them to purification defenses.

$$ $SROCC_{clear}=SROCC(\vv{y},f_{\omega}(P(\vv{x})));\ SROCC_{adv}=SROCC(\vv{y},f _{\omega}(P(\vv{x}^{\prime}))).$ (7) $$

For certified defense methods, we additionally measure the certified radius ($Cert.R\uparrow$), which indicates the amount the input image can be altered by without changing the class prediction; the percentage of abstentions ($Abst.\downarrow$), reported by classification-based methods when there is high uncertainty in the model’s prediction; and certified relative delta ($Cert.RD\uparrow$), which is the certified delta produced by the defense method divided by $\text{diam}(f_{\omega})$. This parameter characterizes how much a metric’s score can change within a fixed $l_{2}$ ball of norm $\epsilon$ around a given image $x$.

### 3.6 Implementation details

To ensure the reproducibility of all our results, we used a sophisticated end-to-end pipeline with a unified interface in the GitLab repository. We automated training and evaluation processes using GitLab CI/CD tools. Computers with the following configuration were used for calculations: 8 $\times$ NVIDIA Tesla A100 80 Gb GPU, Intel Xeon Processor (Ice Lake) 32-Core Processor @ 2.60 GHz. All calculations took a total of about 5000 GPU hours.

We utilized original open-source implementations wherever they were available for all adversarial attacks, defenses, and IQA metrics. Parameters for IQA metrics were set to default, and the supplementary material provides a list of parameters for attacks and defenses. We varied one parameter for each attack and defense while keeping the remaining parameters as in the original implementations. The supplementary material also includes links to the original repositories and a list of applied patches needed to enable gradients in some metrics.

## 4 Results

Adversarial purification defenses for IQA metrics. Table 4 presents the general results for evaluated defenses against a set of adversarial attacks applied to IQA metrics. Image flipping demonstrates one of the best performances in the non-adaptive case, even slightly increasing $SROCC_{clear}$ compared to the non-defended case. Given its low computational complexity, flipping can be effective when the attacker is unaware of the applied defense. However, it exhibits the worst results in the adaptive attack setting.
Simple spatial transformations are the most effective in the non-adaptive case. For adaptive cases, more complex methods that incorporate randomness, such as Rotate, Crop, Random Noise, DiffJPEG, Real-ESRGAN, and DiffPure, perform better as it is harder for the attack to adapt to these transformations.
Among methods designed to mitigate attack effects for classification models, DISCO performs well in non-adaptive settings but falls short in adaptive ones, while DiffPure exhibits the opposite performance pattern. FCN, which is specifically designed for color attacks, failed to perform well against our set of attacks and showed the worst results in adaptive settings. Methods incorporating blurring show low evaluation metrics for both cases, leading to the counterintuitive conclusion that simple high-frequency filtering does not effectively mitigate attack’s effects.
We noticed that the confidence intervals are large. We explain it by the high variability in defense performance across different IQA metrics and attack types. This variability shows the diverse nature of the metrics and attacks. Additionally, the variation in attack strengths and parameters further contributes to the wide confidence intervals, indicating that some defenses are highly vulnerable to specific attack parameters and settings while being more robust against others.
Figure 1 reports the comparison of evaluated criteria between non-adaptive and adaptive cases. The colors of the bars represent the quality of purified images. Adaptive attacks are significantly more successful than non-adaptive ones regarding defense efficiency. At the same time, we do not see such a phenomenon for $SROCC_{adv}$, which could be related to the more unpredictable behavior of adaptive attacks. In terms of a combination of criteria, in Figure 1 (a) the best defenses are DISCO and DiffPure, while in Figure 1 (b) leaders are Real-ESRGAN, MPRNet, DISCO and random noise.

**Table 4: Comparison of purification defenses. Evaluated metrics are averaged across all images and attacks. Defense parameters’ values with the highest $SROCC_{adv}$ are selected in non-adaptive case.**
| Defense | Common | Non-adaptive case | Adaptive case |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | $SROCC_{{clear}}\uparrow$ | Mean Time(ms)$\downarrow$ | $D_{{score}},\%\downarrow$ | $SROCC_{{adv}}\uparrow$ | $Q_{{score}}\uparrow$ | $D_{{score}},\%\downarrow$ | $SROCC_{{adv}}\uparrow$ |
| W/o Defense | 0.748±0.02 | 0 | 60.25±56.59 | 0.453±0.19 | 1.93±0.25 | — | — |
| Flip | 0.768±0.01 | 0.01±0.01 | 9.78±4.37 | 0.710±0.07 | 0.55±0.03 | 88.30±57.20 | 0.356±0.19 |
| Unsharp | 0.741±0.00 | 1.61±0.37 | 55.59±44.61 | 0.416±0.23 | 1.65±0.17 | 110.31±70.23 | 0.267±0.21 |
| JPEG [] | 0.735±0.02 | 328.18±122.40 | 10.62±2.18 | 0.718±0.04 | 1.62±0.11 | — | — |
| Color Quantization [] | 0.732±0.01 | 0.08±0.02 | 28.90±26.34 | 0.645±0.13 | 1.66±0.14 | — | — |
| DiffJPEG [] | 0.729±0.01 | 27.29±4.39 | 11.87±5.58 | 0.703±0.04 | 1.68±0.12 | 32.40±19.40 | 0.651±0.11 |
| Random Noise | 0.707±0.01 | 31.82±2.06 | 16.84±4.18 | 0.693±0.05 | 1.15±0.07 | 55.65±38.42 | 0.567±0.17 |
| Crop [] | 0.706±0.02 | 0.33±0.09 | 15.99±6.55 | 0.575±0.11 | 0.65±0.02 | 22.23±2.65 | 0.441±0.15 |
| Real-ESRGAN [] | 0.703±0.01 | 5.58±1.87 | 29.32±14.95 | 0.576±0.12 | 1.70±0.13 | 37.15±14.85 | 0.552±0.10 |
| FCN [] | 0.699±0.02 | 0.49±0.15 | 26.80±21.31 | 0.537±0.15 | 1.26±0.05 | 87.14±54.68 | 0.177±0.24 |
| DISCO [] | 0.688±0.00 | 72.10±5.68 | 8.59±1.84 | 0.677±0.03 | 1.63±0.12 | 55.27±39.56 | 0.505±0.11 |
| Resize [] | 0.688±0.02 | 0.38±0.11 | 14.09±8.02 | 0.598±0.08 | 1.73±0.15 | 81.06±50.42 | 0.241±0.17 |
| Rotate | 0.667±0.02 | 4.84±0.92 | 10.73±3.44 | 0.582±0.07 | 0.61±0.03 | 19.23±8.91 | 0.502±0.15 |
| MPRNet [] | 0.646±0.01 | 89.46±12.60 | 13.13±7.74 | 0.562±0.08 | 1.74±0.14 | 54.55±33.89 | 0.517±0.11 |
| Bilinear Upscale | 0.642±0.01 | 0.28±0.08 | 22.17±16.05 | 0.396±0.20 | 1.77±0.16 | 54.38±34.19 | 0.261±0.18 |
| Median Blur | 0.637±0.01 | 0.25±0.08 | 16.93±10.01 | 0.424±0.18 | 1.69±0.15 | 59.49±35.23 | 0.416±0.16 |
| Gaussian Blur | 0.569±0.01 | 1.54±0.35 | 18.28±7.22 | 0.347±0.20 | 1.74±0.15 | 49.17±29.66 | 0.278±0.18 |
| DiffPure [] | 0.566±0.01 | 2626.68±263.66 | 17.47±1.75 | 0.549±0.05 | 1.52±0.10 | 27.56±15.41 | 0.466±0.07 |

Adversarial training for IQA metrics. Models trained with a strategy where SSIM penalizes MOS scores for perturbed images and attacks are restricted by ($\varepsilon=2/255$) showed the best results. Increasing the perturbation budget during training reduces general robustness but improves $SROCC_{clear}$. Compared to the LPIPS-based penalization strategy, SSIM-based penalization for ground-truth quality labels provides higher $SROCC_{adv}$. The supplementary material contains tables with detailed results.

**Table 5: Certified defenses comparison on 10 images for each attack case. All results are averaged by attacks and IQA metrics. RS, DRS, DDRS, and DP methods are classification-based. We additionally measured the certified radius and number of abstains for them. For regression-based methods (MS and DMS) we measured certified relative delta.**
| Defense | $D_{score}^{(D)}\downarrow$,% | $SROCC_{adv}\uparrow$ | $SROCC_{clear}\uparrow$ | $Cert.R\uparrow$ | $Abst.\downarrow$,% | $Cert.RD\uparrow$,% |
| --- | --- | --- | --- | --- | --- | --- |
| RS [] | 2.51$\pm$2.03 | 0.59$\pm$0.16 | 0.63$\pm$0.17 | 0.19$\pm$0.04 | 5.27$\pm$3.73 | - |
| DRS [] | 11.34$\pm$7.62 | 0.59$\pm$0.16 | 0.78$\pm$0.13 | 0.20$\pm$0.04 | 6.42$\pm$4.58 | - |
| DDRS [] | 1.21$\pm$0.75 | 0.76$\pm$0.11 | 0.78$\pm$0.10 | 0.19$\pm$0.03 | 10.34$\pm$7.33 | - |
| DP [] | 0.97$\pm$0.39 | 0.76$\pm$0.12 | 0.78$\pm$0.11 | 0.18$\pm$0.03 | 6.77$\pm$3.53 | - |
| MS [] | 10.21$\pm$9.65 | 0.64$\pm$0.09 | 0.67$\pm$0.11 | - | - | 1.34$\pm$0.49 |
| DMS [] | 12.57$\pm$11.39 | 0.75$\pm$0.11 | 0.79$\pm$0.12 | - | - | 1.31$\pm$0.38 |

Certified defenses for IQA metrics.
From the results of certified defenses (Table 5), we can conclude that defenses involving diffusion denoising (DDRS, DP) show better results than other methods. However, these methods have a higher number of abstains. We also noticed that all methods provide almost the same theoretical guarantees ($Cert.R\uparrow$ for classification and $Cert.RD\uparrow$ for regression). A significant drawback of all certified defenses is their high computational complexity. Additionally, these methods guarantee theoretical restrictions on a given dataset, but these restrictions cannot be transferred to unseen data. Certified methods generally produce better $D_{score}$ and $SROCC$ than purification methods.

Figure: Figure 3: Purification defenses efficiency for IQA metrics in terms of (a) $D_{score}$ and (b) $D_{score}^{D}$ and $SROCC_{clear}$. Legend for defenses is shared for both figures. (a) shows performance for selected defenses for each attack separately, (b) reports performance for defenses with different parameters.
Refer to caption: extracted/5771534/imgs/two_new_plots.png

Attacks of different types and strengths.
The supplementary material contains results of adversarial purification methods against attacks of varying strengths. It highlights defenses’ performance when applied to original and attacked images.
Although the leading methods in terms of defense effectiveness ($D_{score}^{D}$) and similarity to original images ($Q_{score}$) remain consistent regardless of attack strength, higher attack strengths generally reduce the effectiveness of defenses.
Overall, we observed no changes in the behavior of defenses varying attacks’ strength.
Figure 3 (a) reports $D_{score}$ for selected purification methods on different attacks. Attacks’ labels are colored based on their type, and the legend is shared for both figures. The main conclusion is that the efficiency of white-box attacks and complex purification defenses (DISCO, DiffPure) drops more significantly than that of black-box attacks. $D_{score}$ is highly depended on attack method, thus it is essential to evaluate defense methods against a wide range of attacks. The dispersion of values across different attacks explains the large confidence intervals in Table 4. Figure 3 (b) shows defenses’ performance with different parameters.
Supplementary material provides more results, tables, figures, and discussions.

## 5 Conclusion

This paper analyzed the efficiency of 25 adversarial defense methods for IQA metrics. We used a wide range of 14 adversarial attacks with varied hyperparameters to cover a large set of possible perturbations. We also considered adaptive and non-adaptive cases, when an attack is and is not aware of adversarial defense. Among the defenses, we compared both certified and empirical types of attacks, such as adversarial training and adversarial purification.
Our main findings are that no defenses are effective across all scenarios. Secondly, when choosing an appropriate defense mechanism, one needs to take into account the robustness of the employed IQA metric: adversarial attacks can significantly affect the model’s performance. Thirdly, to thoroughly examine defenses’ performance, they need to be tested against a wide range of attacks.
We published the results as online leaderboards and a dataset of adversarial images used in non-adaptive attack scenarios. This dataset and the benchmark can be helpful for researchers and companies who want to make their metrics more robust to potential attacks. Although the proposed benchmark can identify the most effective defense methods against adversarial attacks on IQA metrics, it can also pinpoint attacks most resistant to these defenses, which can be considered to have a potential negative social impact. By publishing our findings, we highlight the necessity of further research on improving defense methods in applications of image quality assessment.

Limitations.
While the proposed framework for benchmarking defenses against adversarial attacks on IQA metrics offers significant contributions, we acknowledge some limitations that need to be addressed in future work. Firstly, using a single dataset type may introduce biases, as defenses might perform well on specific image types but not on others. Robustness evaluation depends significantly on the quality and complexity of the testing dataset. Secondly, the framework handles attacks with a single parameter, but future versions should address attacks with multiple parameters. Thirdly, the transferability of adversarial attacks among different defenses is not evaluated, which should be included to provide insights into generalizability. Lastly, the current ranking methodology is straightforward and might not capture the complexity of different evaluation metrics; thus, a more rigorous statistical framework is needed to reflect the overall performance of defense mechanisms. A discussion of more detailed limitations can be found in the supplementary materials.

## Acknowledgments and Disclosure of Funding

The authors would like to thank the CMC Faculty of MSU,
especially A.V. Gulyaev for providing computing
resources, which enabled us to undertake some of the
calculations for this paper. We also would like to express our gratitude to Mikhail Pautov for discussing the results of this research.

The work was supported by a grant for research centers in the field of artificial intelligence, provided by the Analytical Center in accordance with the subsidy agreement (agreement identifier 000000D730321P5Q0002) and the agreement with the Ivannikov Institute for System Programming of dated November 2, 2021 No. 70-2021-00142.

## Checklist

- 1.
For all authors…
(a)
Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b)
Did you describe the limitations of your work?
[Yes] See Section 5.
(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 5.
(d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
- 2.
If you are including theoretical results…
(a)
Did you state the full set of assumptions of all theoretical results?
[N/A]
(b)
Did you include complete proofs of all theoretical results?
[N/A]
- 3.
If you ran experiments (e.g. for benchmarks)…
(a)
Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the supplementary material.
(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 3.6.
(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 3.6.
- 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…
(a)
If your work uses existing assets, did you cite the creators?
[Yes] See Section 3.4.
(b)
Did you mention the license of the assets?
[Yes]
(c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes] See Section 3.4.
(d)
Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e)
Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
- 5.
If you used crowdsourcing or conducted research with human subjects…
(a)
Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b)
Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
(c)
Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see Weakness.
soundness: 3
strengths: (1) This paper gives a comprehensive comparison of multiple defense methods against IQA under a variety of attacks, and draws a few conclusions under different scenarios.
(2) The detailed analysis of the trade-offs between performance and robustness in various defense strategies offers practical guidance for researchers and developers.
(3) The inclusion of statistical tests and evaluations of quality scores adds robustness to the findings.
summary: The paper aims to benchmark and evaluate the robustness of 30 different adversarial defense methods against 14 adversarial attacks regarding IQA metrics. It emphasizes the need for effective defenses due to the unique challenges posed by preserving image quality while defending against adversarial perturbations. It presents a comprehensive analysis of the efficiency of various adversarial defense methods for image quality assessment (IQA) metric.
weaknesses: (1)	Although the paper considers 30 different defense methods, it ignores some defense methods which are tailored for IQA methods specifically such as [1]. These methods should be discussed and compared in the paper, as the goal of this paper is to discuss the defense of IQA.
(2)	The paper has evaluated different defense methods under different indicators, showing a lot of charts, but it lacks the in-depth analysis about what is the reason behind the effectiveness of different defense methods which is important. For example, the defense performance on the KonIQ-1k dataset on the right of Figure 3 exceeds the other two datasets in multiple defense methods. What is the reason? Why do many attack methods in Figure 4 achieve the worst defense performance on FPR, in terms of R robustness?
(3)  More experimental details and analysis are expected. For example, in line 227, 50 images are selected from 1k images for attack, do different selections of attack images affect the performance of attack and defense? Does it affect the conclusion?

[1] Defense Against Adversarial Attacks on No-Reference Image Quality Models with Gradient Norm Regularization, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, June 2024

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 1
questions: 1. In section 3.2, "clustering the KonIQA dataset by spatiotemporal complexity." - Could you please explain the temporal aspect of images?
2. Given the high computational demands of certified defenses and 10 images being used? How would you expect the results to vary as you sample different sets of 10 images?
3. Logistics questions on the leaderboard :
   How do you plan to maintain the leaderboard, and will there be mechanisms for incorporating new defense/attack techniques over time?
soundness: 3
strengths: Major strengths : 

1. The paper presents the first comprehensive benchmark of 29 defense methods,14 attack methods, and 9 IQA methods for NR-IQA
2. Multi-dimensional approach to evaluation: robustness, preserving correlation with human perception (SRCC/PLCC), and quality of the image with respect to original (PSNR, SSIM)
3. Practical Contribution to Research and Industry -> Good work on setting up a public benchmark.
summary: The paper attempts to set up benchmarking for adversarial attacks and defenses in the context of No-Reference Image Quality Assessment algorithms. The coverage of the work seems to be good—29 defense strategies, 14 attack methods, and 9 IQA methods. Lots of experiments (as needed) and results are provided, as expected from a benchmarking paper.
weaknesses: Major weaknesses :

1. The paper is a valuable resource for the IQA community. But in terms of technical novelty, it would have been nice to have an attack/defense method with some motivation for the IQA task. I feel this paper would have been more suitable for a Benchmarking Track, but I understand ICLR lacks such a track and had to submit it to the main conference track
2. The appendix provides many results, but it is very difficult to connect them to the main text, which points to the paper's poor organization.
3. The authors should add LPIPS to the results in Table 3 (and other similar tables) along with PSNR and SSIM. PSNR is not a perceptual metric and can be reliable, leaving SSIM as the only metric. It is better to report both SSIM and LPIPS scores. 

Minor :

1. Paper formatting needs to improve, and content organization can also be better. For example, Fig 1 does not discuss page 8.
2. Section 3.1 under Adversarial defenses: It is better to divide the paragraphs into different types. This will make reading the paper much easier.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. The definition of attack in Equation 1 focuses solely on increasing the model's score. However, if an image already possesses the highest quality, what is the purpose of the attack on it? Why was the idea of decreasing the score of high-quality images/videos not considered?
2. Currently, the attack methods employed are those typically used in classification problems. It would be beneficial to consider incorporating some of the attack strategies for IQA that have been proposed in recent years.
3. Table 3 shows that many adversarial purification defenses exhibit strong defensive effects. However, these methods should be analyzed more thoroughly. For instance, purification techniques that modify the entire image, such as color quantization and median blur, should include more detailed indicators (L_2 and L_\infty) to better reflect the extent of image modification.
4. Most of the analysis in this paper primarily describes the data presented in the tables. It would be beneficial to include an in-depth analysis of the characteristics and connections among these various types of defense strategies, providing great guidance for future research.
5. The abbreviations in the table should be added with full spelling in the caption to help readers understand and prevent misunderstandings. 
6. Equation 2 misses the variable x’.
soundness: 3
strengths: 1. The research topic of this paper is interesting and promising.
2. The experiments in this paper are comprehensive and detailed.
summary: This paper systematically evaluates the effectiveness of various defense strategies for NR-IQA models, including adversarial purification, adversarial training, and certified robustness. It also examines these defenses under both adaptive and non-adaptive attack scenarios. Overall, the experiments in this paper are thorough and comprehensive, but the paper's readability could still be enhanced.
weaknesses: 1. Most of the strategies are directly borrowed from classification tasks, with no new approaches tailored to the specific characteristics of IQA tasks.
2. In addition to PSNR and SSIM, incorporating additional quantitative indicators like L_2 and L_\infty would provide a more intuitive understanding of the differences between the original image and the purified image.
3. The article is less readable, and many tables contain abbreviations that are not defined.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: The authors should work more on Points 1, 2, and 3 in an attempt to raise the reviewer's rating.
soundness: 2
strengths: 1. The paper is decently written.
2. The problem under investigation is of practice importance as well described in the Introduction.
3. This empirical study is comprehensive.
summary: This paper proposes an empirical investigation of the effectiveness of various defense techniques against adversarial attacks on image quality metrics.
weaknesses: 1. A primary technical concern is label preservation (quality preservation in our context) amidst adversarial perturbations. This research evaluates various adversarial attacks, including those like MADC by Wang and Simoncelli, which may not preserve image quality after attacks. In such instances, the model is expected to provide a different quality prediction for the manipulated image, necessitating ground-truth quality assessment through human evaluation.

2. From a conceptual perspective, the reviewer wonders how to understand certified defenses that involve voting and medianing. In classification, it is not hard to comprehend that the robustness is gained through prediction ensembling, which is again under the assumption of label preservation (see Eq. (4)). However, the quality preservation assumption is clearly not true for quality assessment. For example, consider a test image with some Gaussian blur, to perform random smoothing, we shall add Gaussian noise to it according to Eq. (4). Then, the final score is the average quality estimates of a Gaussian blurred image and a Gaussian blurred and noised image (which may be of different quality), which makes less sense to the reviewer.

3. The observed effectiveness of the adversarial attacks (evidenced by an SRCC decrease from 0.611 to 0.477 in Table 3) appears inconsistent with prior research such as [Zhang et al., 2022a] (which reduces to random guessing). Given the limited success of these attacks, interpreting the defense results with similar SRCC values (ranging between 0.5 and 0.6) becomes challenging.

4. Recent NR-IQA models that integrate visual and language components have not been evaluated in this study.

5. The focus of this empirical study aligns more closely with image processing journals rather than a machine learning conference like ICLR, given that no new theories and algorithms are developed.

Meta-review: The authors present a comprehensive study benchmarking the robustness of 29 defense methods for 14 adversarial attacks against 9 different image quality assessment (IQA) metrics. While the study is comprehensive and of value to the IQA community, there is too little new insight generated to be a good fit for ICLR.

It is recommended to consider sending the work to an image processing journal or to a benchmarking track of a major conference.

Decision: Reject