Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metrics
Authors: Aleksandr Gushchin
Khaled Abud
Georgii Bychkov
Ekaterina Shumitskaya
Anna Chistyakova
Sergey Lavrushkin
Bader Rasheed
Kirill Malyshev
Dmitriy S. Vatolin
Anastasia Antsiferova
Venue: ICML 2025 poster (2025)
Keywords: adversarial defenses
image quality assessment
adversarial attacks
image quality metrics
benchmark
Primary area: general_machine_learning->evaluation
TL;DR: This paper presents a new benchmark of defense methods against adversarial attacks on image quality metrics

Abstract: Modern neural-network-based Image Quality Assessment (IQA) metrics are vulnerable to adversarial attacks, which can be exploited to manipulate search engine rankings, benchmark results, and content quality assessments, raising concerns about the reliability of IQA metrics in critical applications. This paper presents the first comprehensive study of IQA defense mechanisms in response to adversarial attacks on these metrics to pave the way for safer use of IQA metrics. We systematically evaluated 30 defense strategies, including purification, training-based, and certified methods --- and applied 14 adversarial attacks in adaptive and non-adaptive settings to compare these defenses on 9 no-reference IQA metrics. Our proposed benchmark aims to guide the development of IQA defense methods and is open to submissions; the latest results and code are at https://msu-video-group.github.io/adversarial-defenses-for-iqa/.

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
Aleksandr Gushchin 123 Khaled Abud 23 Georgi Bychkov 123 Ekaterina Shumitskaya 123 Anna Chistyakova" 13 Sergey Lavrushkin 12 Bader Rasheed 4 Kirill Malyshev Dmitriy S. Vatolin 123 Anastasia Antsiferova 124
Abstract
of such vulnerabilities of IQA models (Antsiferova et al., 2024; Meftah et al., 2023; Zhang al., 2024; Ghildyal & Liu, 2023). Adversarial attacks on IQA metrics are per- turbations that mislead the metric's score, making quality assessment invalid. Such attacks can manipulate image search results, as search engines (e.g., Microsoft's Bing) rely on IQA metrics rank results (Bing, 2013). Further- more, since IQA metries serve in public benchmarks and comparisons (Huang et al. 2024; Wu et al. 2024, etc.) to eval- uate image/video processing and compression algorithms, competitors can exploit the vulnerabilities of the metric to artificially inflate the quality of their algorithm. Several works showed that optimizing image restoration for modern vulnerable IQA metrics can reduce actual image quality (Ding al., 2021) or generate visual artifacts (Kashkarov et al., 2024). For these reasons, it becomes necessary to study and design methods to improve the robustness of IQA models.
Modern neural-networr-basde Image Quality As- sessment (IQA) metrics are vulnerable to adver- sarial attacks, which can be exploited to manip- ulate search engine rankings, benchmark results, and content quality assessments, raising concerns about the reliability of IQA metrics in critical applications. This paper presents the first com- prehensive study of IQA defense mechanisms in response to adversarial attacks these metrics to pave the way for safer use of IQA metrics. We systematically evaluated 30 defense strategies, in- cluding purification, training-basse methods - and applied 14 adversarial attacks in adaptive and non-adaptive settings compare these defenses on 9 no-reference IQA metrics. Our proposed benchmark aims to guide the de- velopment of IQA defense methods and is open to submissions; the latest results and code are at https://msu-viddee group group.github. veraaaasa defenses-for-igee
and certified
Although researchers have proposed various defense meth- ods to enhance the robustness of neural networks in different applications, few defenses have been developed explicitly for IQA metrics, and there are currently comprehensive benchmarks for this task. Defending IQA metrics poses unique challenges compared to object classification: de- fenses must restore the original IQA scores and their cor- relation with subjective evaluations while preserving the perceptual quality of attacked images.
1. Introduction
Image quality assessment (IQA) metrics are essential to de- velop and evaluate image and video processing algorithms. Modern IQA metries based on neural networks are highly correlated with subjective assessments. However, neural networks are proven to be vulnerable adversarial pertur- bations(Kurake) et al., 2018), which have led to exploration
This paper introduces the first benchmark that systemati- cally evaluates defenses against adversarial attacks IQA metrics. The evaluation scheme is presented in Figure 1. Our contributions include novel methodology for measur- ing and comparing defenses for the IQA tasks, addressing a critical gap in the field, comprehensive experiments, an extensive subjective study with 60,000s responses, an in- depth analysis of the results, and an online leaderboard. We also publish novel dataset of adversarial images, which can be used for adversarial training, evaluating non-adaptive defenses, and training methods for attack detection. Our methodology is the first to systematize defenses for IQA metrics, analyzing 30 defense algorithms (both empirical
'ISP RAS Research Center for Trusted Artificial In- telligence, Moscow, Russia "MSU Institute for Artificial Intelligence, Moscow. Russia 'Lomonosov Moscow State University, Moscow, Russia "Innopolis University, Innop- olis, Russia. Correspondence to: Aleksandr Gushchin <alexander. galexanderggushhhiin gushchin @ graphics.es. es.muu.uu>. Anastasia Antsifer- ova <aantsiferovaa grapiirrovvvrrggrrisii....
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
Datasets
14 Adversarial Attacks attack strength levels White-Box
Evaluation
KonlQ-10K" KADID-10K"
NIPS2017 AGIQA-3K
Adversaria Efficiency Statistical Tests
Computational Camplexity
Perceptual Quality
(-Subjective Study.
Adversarial Training methods
HFCSM
Gradient Norm (NT)
APGD x6 variations
Uimeurity
Lineartyy 78
* subset
5 Black-Box
17 Purification methods
Certified methods
Randomised Smoothing (R5)
9 Target IQA Models
Denoised RS
Diffusion dencised RS
Linearity
CLIP-IQA=
Meta-IQA
DensePure
Koncept
FPR
MANIOA
TOPIQ
PaG-2-PIQ
SPAQ
Lincority.
Parsimonious
Inmaaitty.
MPRNNt
Donoisod MS
Smoothing
Figure 1. Evaluation scheme. The benchmark consists of four parts: datasets, IQA models, adversarial attacks, and adversarial defenses of three types: Adversarial Training, Purification and Certified methads.
and certified) and evaluating their effectiveness against 14 adversarial attacks. We address both adaptive and non- adaptive attack scenarios, depending on the attacker's aware- ness of the defense. The benchmark is available online at https://videoprocessngggg www.aaanncnaansiiaaeefeeeeeeeeee html along with the code for the proposed methodology, IQA models, adversarial attacks, and defense methods in the GitHub repository. This unified framework enables re- searchers to measure and compare defense performance, and we welcome submissions of new methods.
and black-box attacks remain applicable. Several white-box adversarial attacks (Shumitskaya al., 2024; Zhang et al., 2022b; Wang & Simoncelli, 2008; Shumitskaya et al., 2023) and at least two black-box attacks (Ran al., 2024; Yang et al., 2024) are designed specifically for IQA metrics.
Defense methods for neural networks come in certified and empirical types. Certified methods provide deterministic or probabilistic robustness guarantees for particular pertur- bations, datasets, or model architectures. However, these methods are usually computationally complex and reduce the model's general accuracy. One of the most well-known certified methods is randomized smoothing (Cohen et al., 2019). Later variations appeared in (Salman et al., 2020; Chen et al., 2022b), and included a denoiser to improve the defended model's performance. Empirical methods lack robustness guarantees but require fewer computational re- sources. widely used empirical defense method is ad- versarial training (AT) (Wong et al., 2020; Singh et al., 2023), which additionally trains the model on adversar- ial examples. Vanilla adversarial training, however, may decrease model performance. Adjusting subjective scores during training has been suggested to prevent performance degradation (Chistyakova et al., 2024). In another study, ( l-regularization of the gradient norm (NT) was used to improve the robustness of NR IQA models against adver- sarial attacks (Liu et al., 2024). Adversarial purification is an empirical method that removes adversarial perturbations by processing input data. Although adversarial purifica- tion is model-agnostic and computationally efficient, may fail to eliminate advanced adversarial perturbations and can degrade image quality. Examples of such methods range from compression and spatial transformations specialized methods (c.g. DifiPure (Nic et al., 2022)).
2. Related work
Existing comparisons of defense methods mostly focus on object classification (Croce et al., 2021; Dong et al., 2020), leaving a significant gap evaluating defenses for image quality assessment (IQA) metrics. While (Antsiferova et al., 2024) investigates IQA robustness under adversarial attacks, it does not explore defense strategies. Current trends in IQA metrics development emphasize creating task-specific metrics to achieve better correlations, as different IQA tasks (e.g.. user- or Al-generated content, artificial distortions caused by image- processinn algorithms, etc.) require a slightly different approach. This makes the development of universally efficient and robust IQA metric impractical. To address this, we present systematic comparison of defense methods for IQA tasks, enabling researchers to enhance existing models. Our benchmark advances prior work by incorporating attack parameter selection, diverse datasets, and large-scale subjective evaluations, offering a comprehensive framework for assessing defense efficacy.
Adversarial attacks fall into two main categories depending on the attacker's knowledge of the model: "white box" or "black box" (Huang et al., 2017). White-box attacks em- ploy intrinsic characteristies of the attacked models (c.g. gradients); however, in some situations, it is unavailable,
2

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
3. Methodology
3.2. Adversarial attacks
3.1. Problem definition
Accurately evaluating defense mechanisms requires testing under conditions that closely reflect real-world applications. To address this, consider both adversarial attack scenar- ios: non-adaptive and adaptive. In the first case, the attack method targets only the IQA model itself and does not take the defense into account. In the second, we incorporate dif- ferentiable defense into the attacked IQA metric, enabling adaptive attacks leverage gradients from both the metric and the defense mechanism. We selected 14 white- and black-box attacks of diverse types, including methods tai- lored specifically for IQA task. Table 7 in the Appendix describes these attacks.
Adversarial attacks. This work evaluates adversarial de- fenses for no-reference (NR) IQA metrics because they have more comprehensive range of applications and are more vulnerable to attacks (Ghildyal & Liu, 2023). In this setting, an attacked model, represented by NR IQA metric, takes a single image as input and estimates im- age quality. Formally, the NR IQA metric is the mapping fw X + IR, parameterized by the vector of weights w. Here, X E [o, || is the input image. An adversarial attack X + X is the perturbation of the input image defined as
A Attacks hyperparameters. Recent work (Dong et al., 2020) reveals that defense rankings are sensitive to attack parame- ters. This instability underscores the necessity of evaluating defenses across diverse attack configurations. To account for this, we execute each attack method with three hyper- parameter sets corresponding to "weak", "medium", and "strong" perturbation budget. Parameter selection was per- formed by linear approximation for the target attack budgets on a small subset. We chose a subset of 50 images used for attack alignment via clustering the KonIQ- 10k dataset by spatial complexity (SI), colorfulness, and ground-truth quality (MOS). Appendix A.3.2 contains a list of chosen parameter sets, alongside the procedure scheme in Fig.7.
A(z) = arg max ((f((t))),
(1)
where L is a loss function that represents the model's out- puts for perturbed images and p(+--) is the distance function defined on X x X. We increase IQA scores during the attack to reflect real-life applications (decreasing scores is nearly identical task (Antsiferova et al., 2024)). For IQA metric attacks, we define (z")) f_(x') where dfa = sup (1f((z) - fw { represents the range s.sēX of IQA metric values.
dfam(f..)
Adversarial defenses.
Adversarial purification is an algorithm P: X + X that aims to transform the input image according to the following optimization problem:
3.3. Adversarial defenses
To thoroughly investigate the effectiveness of IQA metric defenses, we explored three method types: adversarial pu- rification, adversarial training, and certified robustness.
min (sa(()
a))
Adversarial purification mitigates threats by preprocess- ing input images before IQA calculation. These methods are efficient and offer a flexible trade-off between attack mitigation and image quality. The top part of Table 1 de- scribes the selected adversarial purification techniques. We used five parameter sets to vary the defense strength, e.g., scaling ratio, blurring kernel size, and number of diffusion steps. The Appendix A.3.5 provides a list of used defense parameters and their selection methodology.
where s' is the adversarial image, À controls regularization.
Adversarial training is formulated as the following problem:
min E D
[(J + +0),)) |1
max
(3)
where D is the distribution of training data, L is a training loss function, E is the attack magnitude, y is image quality of x. In practice, adversarial training uses an adversarial attack rather than internal maximization.
Adversarial training fine-tunes a model on adversarial ex- amples to enhance its robustness. IQA presents additional challenges in applying adversarial training since adversarial examples don't 't preserve ground truth labels (MOS). Man- ually assigning such images with subjective scores is im- practical, and using ground-trutt -truth labels from clean images is inaccurate, as attacks could alter perceived quality. We evaluate the method from (Chistyakova et al., 2024) with different parameters and NT method (Liu al., 2024) that employ gradient normalization during training. Both these methods are specifically designed for the IQA task.
Certified methods used in our paper are based on random- ized smoothing (Cohen et al., 2019), denoised randomized smoothing (Salman et al., 2020), diffusion-based random- ized smoothing (Carlini et al., 2022; Chen et al., 2022b) and median smoothing (Chiang et al., 2020). Randomized smoothing replaces the original IQA metric (x) with a smoothed version g(x), adding Gaussian noise
g(x) = E
Jux
(4)
3

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
Table 1. Evaluated defense methods in our benchmark by types (purification, adversarial training, certified methods).
Defense method
Short description
Defense method
Short description
Gunssian
Smwooth Gaussian filter
Adremanial training
Median blur
Smooth with median filte
JPEG (Givo al., 2018)
JPEG compresioon algorithm Reduce the number colors Diflerentiable IPEG
Color quaniizaion et 2018)
Classic adv, training (Chistyakova al., 2024)
Model fine tuning adv, img.
Adversatial purficttion
Unsharp masking
DiaUPEG (Reich ei al., 2024)
Unsharp mask
Gradicnt Norn optinication (Liu et al., 2024)
Perform gradicnt normalization during training
FCN (Gushchin al., 2024) Flip
Noural filter to counter color attack
Mirrar the image
Bilinear Upscale
Resize and upscale to original size
Resize (Ciso et al., 2018)
Change the image size
Randoen Rotate
Iange cotation
Random, Sunothing (Cohen 2019)
Nuisy samp. -veli--voting
Randoen Crop (Gun et al., 2018)
Crop the image
Cortifted
Denoised RS (DRS) (Salman et al., 2020)
Noisy sump--denmiser-- clf. clf.-vxoting
Randomn noise
Adi random noise
Diffassio DRS (DDRS) (Carlini at al., 2022)
Noisy samp. o s1-step diffus. self. -woing
MPRNet (Zamir al., 2021)
3-stuge CNN denoising
DensePure (Chen 2022h)
Noisy samp. -t--tpp -ilfiin-til.--oxxinng
Real-ESRCa ESRGAN (Wang et al., 2021)
GAN
GAN-based based super-res. denoising
Median Smoothhing (MS) (Chäsng et al., 2020)
Noisy samp. -trog--tooiin
DISCO (Ho & Vasconcelos, 2022)
Enc. floc. imphiit module denoising
Denoised MS (DMS) (Chiang et al., 2020)
Noisy samp. >denoiser oreg. median
DifPure (Nie ol., 2022)
Ditlusion denoising
Certified defenses provide theoretical guarantees for the attacked model. The most common certified defenses are based on randomized smoothing. They can be applied to any IQA metric without restricting the model architecture. Certified defense methods generate noisy variations of the input images, which then pass through the model. Before passing them through the model, some methods apply de- noising to boost accuracy. Table 1 provides further details. For each certified defense method, we generated 1000 noisy images as input for the metric. Currently, most smoothing methods are developed for classification, and one study (Chi- ang et al., 2020) investigated smoothing for regression. To apply the classification- based smoothing for IQA metrics, we are converting the IQA metric into a multiclass classifi- cation model with ordered classes (Hammoudeh & Lowd, 2023). Despite the challenge of IQA metrics discretization, classifier-based smoothing methods can yield impressive results for our task since they are more extensively studied. For a classification--ass quality class and certified radius R; for a regression- based method, it's metric score and certified delta. For more details refer to Sections 3.5 and A.3.5.
images. Due to high computational complexity, we used 50 images from each dataset for black-box attacks and certified defenses. Appendix A.3.1 presents statistical tests verifying that our sampling procedure is valid and representative of the entire dataset.
IQA metrics. Based on the results of the IQA adversarial robustness benchmark (Antsiferova et al., 2024), we chose 9 NR IQA metries: Meta-IQA (Zhu et al., 2020), MANIQA (Yang et al., 2022), CLIP-IQA+ (Wang et al., 2023), TOPIQ (Chen et al., 2024), Koncept (Hosu et al., 2020), SPAQ (Fang et al., 2020), PAQ2PIQ (Ying et al., 2020), Linearity (Li et al., 2020), and FPR (Chen et al., 2022a). These met- ries employ different convolutional and transformer-based architectures and have wide robustness Recore range. The Appendix A.3.5 provides more detailed description. Be- cause of adversarial training's ' computational complexity, we selected only two NR IQA metrics, Linearity and Koncept, for their high correlation with subjective scores.
certified method, the output is a
3.5. Evaluation metrics
Robustness scores. Recore (Zhang et al., 2022a) and R' aim to assess model robustness by measuring rel- ative score changes before and after attacks on dataset of N images. Recore takes into consideration the maximum allowable quality-predicte change:
3.4. Experimental setup
Datasets. To thoroughly evaluate adversarial defenses we use four datasets. KonIQ-10k (10,073 images) (Hosu et al., 2020) and KADID-10K (10,125 images) (Lin et al., 2019) contain various natural images with multiple distortions. NIPS 2017: Adversarial I Learning Development Set (2017, Competition Page. 1,000 images) is designed for evaluating adversarial attacks against image classifiers. AGIQA-3K dataset (Li et al., 2024) contains 2,982 Al Al-generated images for different quality levels. We randomly sampled 1,000 im- ages from KonlQ-10k and KADID-IOK datasets to balance computational efficiency and dataset diversity. We included each distortion type and strength and sampled 8 out of 81 original images from KADID, resulting in 1000 distorted
1
N
(mxx (31 -
- 82)
Recore
log
N
[fw( - L (P((
(5)
where 24 is the source image, ar is the attacked version of Z4. fw(+) is the IQA model, and 81 and Ba are the maximum and minimum of IQA scores in the dataset. In addition, we propose a variation of this metric called R(2) which differs only in applying purification P(-) to x., and x/. A larger value means better robustness.
4

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
E-41255, SIM
c+52255 LPPS
DI/PEG
SSIM
c-42255.
Nose
Ad f/l RE
S5M
dv eeverr Ceverraarall ar
LIPSS
2/55
DITP.N
EBROAN
DITIPEO
CUUU
DISCO
3400Y
2125 SSM
ESRCAN
C020256
D.PPur
cation
MS
Upsca
Biincer
66 11 CO
D68
Rotate
RS
Bandom
DMB
12
Modian
MPRNN4
FRl
aall
DDRS
B
R5
Upacalo
Eliinar
Meddan
DMS
DIBCO
CP
Wio
Adaptive Cara
Blr
Crop
Redibo
Color
Han-adaptive
Res
FCN
Unshorp
JPEG
Co Quonttaaiio
10"
16"
10"
10"
10"
16"
PSNRT
Mean Time(ma)
Figure 2. Adversarial defenses efficiency for IQA metries in terms RROCCade (left) and Dicore (right). Bars and dots for adaptive and non-adaptive attacks, respectively. Each dot represents the result for each preset of defense. Red dots represent a selected preset for the adaptive case. Results are averaged across 9 IQA metrics and 14 attacks.
We propose D Dscore and D(D) to measure adversarial pu- rification's ability to reduce the discrepancy between the IQA scores of the original and purified images:
ing principle is that the defense mechanism should restore the IQA score and preserve the image's perceptual quality. More complex IQA metrics (such LPIPS) cannot be used in this environment due to possible transferabilities of ad- versarial attacks. We also conducted subjective survey to better evaluate the quality of defended images.
100
(P(a()()))) - (mil)
Dscore
diam (Fw)
D(D) acooee
100
(P(x)) - fo (P(x.))
Performance scores. We use SROCC with MOS values T to assess an IQA metric's performance in the presence of adversarial defense. PLCC results are similar to SROCC and provided in the Appendix A.6.
72
diam
where scores denoted with the superscript are for purified source images, P represents the purification method. Lower scores indicate better defense effectivvness, they reflect smaller disparities between the IQA metrics of defended and original images. The metrics quantify how well defense can restore the IQA scores of adversarial images to match their original values.
SROCC =
(P((T))
(8)
SROCC = SROCC(T.L((())))
(9)
3.6. Implementation details
We used a sophisticated end-to-end automated training and evaluation pipeline using GitL CI/CD tools to ensure all our results reproducible. All calculations required ap- proximately 25,000 GPU-hours. Timing benchmarks were performed on a dedicated server with NVIDIA Tesla A100 80 Gb GPU, Intel Xeon Processor (Ice Lake) 32-Core Pro- cessor @ 2.60 GHz.
For certified defense methods, we additionally measured the certified radius (Cert.R 1), which indicates how much the input image can undergo alteration without changing the class prediction; the percentage of abstentions (Abst. J), reported by classification--asee methods when their pre- dictions are highly uncertain; and certified relative delta (Cert.RD 1), which is the certified delta, produced by the defense method, divided by diam(fw). This parameter char- acterizes how much metric score can change in a fixed la ball of norm around given image x.
When available, we used original open-source implementa- tions for all adversarial attacks, defenses, and IQA metrics. For each attack and defense, varied main parameter commonly associated with the attack strength while keeping the remaining parameters consistent with their orig- inal implementations (see Table 1). The Appendix A.3.3, A.3.5 provides a list of parameters for attacks and defenses and links the original repositories.
Quality scores. We use PSNR, R. SSIM et al., 2004), MSE and L to measure the perceptual similarity between purified images and their original images, reflecting the preservation of visual quality pest-defense. The underly-
5

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
Table Comparison of purification defenses. Results are averaged for images, attacks, IQA models for non won-adaative/adaatiie cases.
Time (ms)
5ROCCader
SROCCa 1
Dacore (P)
Rusore
RSd
S81M
MSE 4, x
Lend
Wo Defense
0.511/0.511
0,413/0.413
66.68/66.88
0.56/0.56
44.61/44.61
0,94N0.94
2.51/2.51
0,09/0.09
Flip
0.05
0,593X0,587
0,555/0440
7.91/67 41
1,17/0.45
10.76/10.76
0.28/0.29
110.47/100.80
0
Color Quantization
0.07
0.587/
0.532//
27.38/
0.83/
32.54/
0.86/
2.84/
0.11/
Median Blur
0.11
0.551/0.531
0,431/0.424
15.14/99.95
0.9200.50
31.38/1.80
0.86/0.87
4,48/3.11
0.51/0.51
Bilincar Upscale
0.15
0,569/0,479
0,452/0.355
18. 13/4.3
0.86/0,58
52.82/28.68
0.91 83
3,50/4.23
0.35/0.48
Crop
0.16
0,58700.431
0,508/0.355
66666499
0.92/0
11.53/11.00
0.33/0.37
89.94/105.34 34
0,94/0.93
Rosize
0.19
0,597/0,511
0,549/0.353
10,56/54.31
1.020.42
32. 32.11/29.38
0.90
3.83/3,90
0.37/0.45
FCN
0.52
0,478/0.310
39
0.800044
20.89/2077 78
0. 0,78/0.77
13.24/13.35
0.54/0.55
Unsharp
0.78
0.61 595
0.422/0.370
43.22/80.24 24
0.52/0.32
30. 37/29.7777
0.87/0.86
3/81/303
0.33/0.35
Gaussian Blur
0.99
0,552X0.522
0,423/0.376
15.75/45.67
0.84/0.5
32.22/$5.30
0,90//990
3.83/2.72
0.34/0.35
Rotate
2.14
0,56000,585
0,501/0,169
6.64/16.24
L0990.89
11.56/14.65
0.31/0.42
96.44/54.03
0.97/0.96
Real ESRGAN
5.89
0,552X0.501
0,50330.436
9.47//3.13
0.66/0.58
30.32/30.47
0.89/1.88
3. 3.97/2.99 98
0.43/0.44
DamPEG
8.31
0.625/0.610
0.60%
12.99/29.81
L0770.71
34.33/31.33
0.91/0.87
3.04/2.61
0.26/0.33
Random Noise
8.29
0.556/0 594
0.539/0 508
10. 10.14/44.84 84
0,87/0,59
25.42/35.87 87
0.54/0.90 90
4,78/1.79
0,30
MPRNer
65.79
0.565/0.565
08535/0.488
12.1 12.14/45.00
0.9100.53
32.21/32.32
0 0.8810889
4.23/2.91
0,37/0 36
DISCO
139.60
0.58540.562
0/581/0.476
3.51/47.91
L,440.50
29. 12/29.08
0.86/0.86
4,34/3.11
0.43/0.43
JPEG
227.34
0.622/
0.605/
13.07/
L077
34.25/
0.90/
3.03/
0.26/
DiffPure
691.42
0,496/0.487
0.485/0470
2.01/22.96
0. 0.79/0.75
27.59/30.1 11
0.79/0.86
5,34/3.4
0.48/0.43
4. Results
Compared to purification methods, adversarial training pro- vides superior correlations but shows worse Dscore and doesn't 't purify images. Certified methods deliver the best overall combination of correlations, Dscore and Recore, but are highly impractical due to computational overhead.
In all tables and figures, for non-adaptive cases, we report the results of defenses with a hyperparameter set that pro- vides the best SROCCadv- Table 2 shows overall results for adversarial purification defenses, and Table 3 for adversarial training and certified methods.
Parameter Variations for Defenses and Attacks. Figure (a) illustrates how the parameters of adversarial purification methods impact the trade-off between robustness and perfor- mance in non-adaptive scenario. Strong defenses, located in the lower-left corner of the scatter plot, nearly restore IQA metric scores to their pre-attack values but significantly reduce correlations with subjective quality, making them impractical for real-world applications. The red line high- lights the Pareto-optimal front, which includes strong JPEG compression, weak DiffPure, Gaussian blur, and DISCO.
Adversarial perturbations generally consist of high- frequency noise, making compression- based defenses partic- ularly effective. DifUJPEG leads in terms of several evalua- tion metries among purifications with the best SROCCadvv Decore: and Recore- JPEG and DiffJPEG remove high- frequency noise alongside adversarial perturbations while preserving the structural information of a clean image, as the perturbation has far more complex and unnatural rep- resentation than the natural high-freque freeuery components. Thus, the developers of purification methods should analyze how the high-frequency components of an image and per- turbations differ. DISCO, which uses an encoder-decoder architecture similar to compression, leverages learned fea- tures of clean images to project images back onto the nat- ural manifold. Denoising methods, such as MPRNet and Real-ESRGAN, show average performance, as they were trained on noise of a more uncomplicated nature. At the same time, adversarial perturbations possess more complex high-frequency structures. Fine-tuning these methods on adversarial perturbations is promising direction for future research. Diffusion--based models offer high variability in strength, allowing precise tuning for specific adversarial attack budgets. On the other hand, DiffPure introduces its own processing artifacts, causing the worst correlations and lower image quality of defended images. This highlights a key difference between applying diffusion-based defenses in classification tasks (where they faae-of-the-ar meth- ods) and IQA tasks, underscoring the need for task-specific adaptations.
The comparison results for different attack parameters, as presented in Appendix A.6 (Table 11), show that increasing attack strength leads to decreased defense success. Notably, defenses in non-adaptive case remain stable across different attack strengths, while in the adaptive case they are highly sensitive to attack intensity. Strong attacks cause significant correlation decrease, making most defenses impractical in real-world applications, with DiffPure and DiffJPEG being notable exceptions. Importantly, the relative ranking of defenses remains mostly consistent across attack strengths.
Inference computational complexity. Tables and 3 present time costs across defenses. Certified methods excel in defense efficiency but are computationally intensive, with the fastest certified method being 4x slower than the slowest purification onc. Adversarial training requires no inference overhead. For porification-baaeee defenses, computational demands vary. Basic preprocessing (blurring and rotation) has minimal overhead, while diffusion-basse methods (c.g.. DiffPuree, are much slower due to multiple denoising steps.
6

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
Defense parameters
BORGAN
ESPGAN
R
Strongeat
Strong
Drop
Rcobe
Crop
Rotalo
Medium
Weak
Wankest
NITON
DIS00
WW.
JPDG
Web
Vecar
Nacar
SROCC
Devemue
far
Quntization
Upocas
Estor Quanization
Crop
FON
MP
.PCC
Fanston Nooss
Resdet
-Ussharp WNo Defenso
Datasel
KADID1K
KON OTK
AGIGASK
NPS
(b)
Figure 3. (a) Scatter plot of purification defense parameter contigurations in the non-adaptive case, with the red line indicating Pareto- optimal defenses. (b) Performance across lest datasets in terms of D (left) and SROCCodo (right).
Table 3. Comparison of adversarial training (AT) (left) and certified defenses (right). C: classificationn based methods, R: regression-based. For AT methods, APGD is an attack used for fine-tuming 2/4/8 perturbation budget. LPIPS/SSIM are FR metries for MOS adjustment.
Adaptive atneks 1800 imnges KoulQ-I0L 10k, Koncqptel incuity
Non-adeptive stncks 10 images from KoulQ--II IQA metriss
SROCCa..
SROCC...
D
Rearr
Cert Defense
Time(ms]
SROCCa..
SROCC...
R.
Cent. / //Cert.RD.
LPIPS
20.70
1.10
11080
0.706
0
4PGI-LPIPS-4
0.66
0.76
30.59
0.77
DRS
15320
11882
0712
16:57
2.01
0.175/
APGD-LPIPS-R
0.547
45.61
0.60
DDRS (C)
39800
0.819
0.702
1.20
6.21
0.174/
APGD SSIM 2
0.30
0.763
17.11
1.23
DP (C)
82130
0.823
0.815
1.09
5.20
0.102/ 30
APGI-SSIM-4
0,52
0.625
39.55
0:30
MS (R)
2830
01753
01881
3.80
1.02
B/77707
APGD-SSIM-S
0.573
0.582
45.38
0.64
DMS (R)
5970
0.875
0.822
4.70
1.59
0/ 1.440
NT (LiU et 2024)
0.315
0.649
35.42
0.81
Defenses against adaptive and non-adaptive attacks. Figure and Table 2 compare defense performance against non-adaptive and adaptive attacks. Adversarial training was measured only in adaptive setting, while certified defenses were only for non-adaptive scenario. By design, adaptive attacks are significantly more successful, so D'D) robust- ness bars are higher than markers on Figure 2. Furthermore, the SROCCade of defended IQA metrics is lower, due to the more unpredictable behavior of adaptive attacks. Simple spatial transformations (Flip, Resize) and frequency filter- ing (Gaussian Blur, Median Blur) are effective in the non- adaptive case but insufficient for adaptive one. In adaptive case, Crop, Rotate and DiffPure excel in (DD) and Racore+ suggesting high randomness is crucial for effective defense. The combination of randomness and geometric transforma- tions particularly mitigates perturbations. Flip and Random Rotate are great examples: the first lacks randomness, and adaptive attacks easily surpass it, while Random Rotate re- duces attack effectiveness since the angle differs between attack calculation and inference. Specialized defenses can demonstrate high effectiveness in the non-adaptive case but exhibit unpredictable performance against adaptive attacks. For instance, DISCO ranks top-3 by Rycore in the non- adaptive case but plummets under adaptive attacks, whereas DiffPure maintains its top position in both scenarios.
Defenses for regular/AI-gen image content Figure 3 (b) compares Dacore and SROCC on different datasets, including three natural-scene image datasets and the AI- generated images from AGIQA-3K. There is no significant difference in defense efficiency for most methods between datasets, but some advanced defenses based on neural net- works (Real-ESRGAN, DISCO) have larger discrepancies. This highlights the critical importance of dataset coverage of the target data domain during training. As shown in Figure (b, right), correlations depend highly on the dataset. On average, SROCCade on KonIQ- is significantly higher than on KADID and AGIQA-3K, similar to results in Table 19 in the Appendix regarding SROCCd This can be due to two factors: a) Several IQA models (e.g.. TOPIQ and CLIP-IQA+) were trained on the KonIQ-10k dataset or its subsets, giving them a natural advantage. b) Certain IQA models, such as MetalQA and PAQ2PIQ, generally achieve higher correlation values on KonIQ-10k, as reported in their respective studies, suggesting an inherent dataset bias.
Guarantees of the defenses. Among all the methods com- pared, only certified methods provide theoretically reliable predictions. Table 3 presents the results for the certified defenses. Compared to more sophisticated methods, sim- ple randomized smoothing showed the highest certified ra- dius. Among regression- based methods, Denoised Median
7

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
SPAD
Linwanty
SPAG
Linsonty
Perceptual quality of defended images. Most presented purification defenses aim to restore the original content of the image, but inevitably introduce artifacts. The most noticeable ones include loss of details (DISCO, MPR- Net), altering the image content (Real-ESRGAN, Diff- Pure), blurring (DiffPure, blur), and compression artifacts (JPEG/DifUJPEG, Color Quantization). Figures 9 and in the Appendix A.5 show examples of images with such artifacts. Table 2 shows quantitative results of perceptual quality, Attacked images turned out to be closer to clean im- ages than purified ones. PSN R and SSTM M cannot account for geometric transformations and, thus, are meaningless for Flip, Rotate, and Crop.
Kimeapi
Kanepp
PAAZPIO
PAADPIO
Spope
PPR
FPR
CUINOA
CLIP CLIP-OA+
MANAAA
TOPIO/NR)
TOP (c)
Defense
FCN
Color Cuantzation
JPEG
RASHESSRGAN
MPRNel
Fondomm No
DISCO
Vio Deferso
To assess perceptual quality of defended images we con- ducted large-scale crowd-source subjective study with 2,700+ participants who provided 60,000+ responses (de- tails in Appendix A.2). The results highlight Real-ESRGAN as the top method, achieving perceptual quality that sur- passes even the "Woo defense" case in quality. This method treats adversarial perturbations as degradations, effectively removing them while preserving natural image content. Real-ESRGAN has architecture and training objective that naturally suppress high-frequency noise, a common feature of adversarial attacks, and demonstrate unique ability to mitigate unrestricted attack perturbations - an area where traditional denoising methods falter. The model's relatively lower performance on objective metrics can be attributed to aliasing artifacts introduced during downsampling opera- tions, residual noise, and subtle change in contrast. Subjec- tive evaluation reveals that Gaussian blur and DiffJPEG per- form significantly worse than their objective metrics suggest. While Gaussian blur fails to suppress both high-frequency noise and unrestricted attack artifacts, DiffJPEG introduces strong blocking artifacts and texture loss. DiffPure not only fails to suppress high-frequency noise, but also distorts tex- tures and blurs image, resulting low subjective scores. These findings highlight the importance of subjective eval- uations: objective metries such as SSIM and PSNR can misrepresent perceived quality and thus lead to misleading leaderboard rankings.
Figure 4. R score (a) and SROCCotaa (b) different IQA met- rics of some purification defenses.
Smoothing showed the lowest certified relative delta. Al- though there are no theoretical restrictions using certified defenses without denoising, our experiments indicate that the denoising step is crucial to obtain a defended model that produces high SROCC for the IQA task. Another finding is that, despite the questionable applicability of randomized smoothing- based defenses to IQA, they remain effective, as the certified radii are sufficiently high and the number of abstentions is relatively low.
Defenses for different IQA metries' architectures. The chosen IQA metrics fall into categories by their back- bones: CNN-based (Meta-IQA, Koncept, SPAQ, PAQ2PIQ. Linearity), transformer- based (MANIQA, CLIP-IQA+, TOPIQ), and custom (FPR). Figure 4 shows Rycore and SROCCotear for these metrics in non-adaptive scenario.
Transformer-based metrics have greater Racore robustness even without defense. Their self-attention mechanisms en- able comprehensive global contextual analysis, capturing imdge-wide dependencies beyond local features. This ar- chitectural feature provides an intrinsic resilience to sub- tle adversary perturbations, resulting in a lower robustness increase when defenses are applied. For other architec- tures, most defenses increased the robustness. DISCO im- proved the robustness of all metrics, but the effect was much stronger on CNN-based metrics. Defended transformer- based metrics showed a higher correlation decrease than metrics of other architectures. Note that custom architec- tures can be highly vulnerable (FPR model shows the worst Recore). This vulnerability is likely caused by its atypical architecture with Siamese network and attempt to "hal- lucinate" the features of the pseudo-reference image from distorted one. These results correlate with previous research (Antsiferova et al., 2024). In general, all tested defenses do not impose restrictions on the architecture of the models; however, some defenses perform better for specific metries.
Statistical tests: A one-sided Wilcoxon signed-rank test with Bonferroni correction (see Appendix A.4) further proves the statistical significance of the results obtained in our comparison and confirm that DISCO, DiffPure and DiffJPEG defenses significantly outperform all others in both D and Recore with consistently high pairwise win-rate percentages across datasets. In the adaptive setting, APGI-SSIM-22 adversarial training also demonstrates sta- tistically significant robustness advantage over competing defenses.
8

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
5. Conclusion
References
This paper introduces the first comprehensive benchmark for evaluating defenses against adversarial attacks on neural network-bass basedd Image Quality Assessment (IQA) metrics, addressing critical gap in the field. By systematically analyzing 30 defense strategies across 14 attack methods and 9 IQA metrics, we provide framework to guide the development of secure and reliable IQA models.
Agustsson, E. and Timofte, R. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
Andriushchenko, M., Croce, F., Flammarion, N., and Hein, M. Square attack: query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision, pp. 484-501. Springer, 2020.
Our results highlight superiority of compression- based de- fenses due to their ability to remove high- figh-frequency adver- sarial perturbations while preserving the structural details. This suggests that future defenses should focus on better distinguishing genuine fygguerry details from adver-
Antsiferova, A., Abud, K., Gushchin. A.. Shumitskaya, E., Lavrushkin, S., and Vatolin, D. Comparing the robustness of modern no-reference image-and video-quality metrics to adversarial attacks. In Proceedings of the AAAI Confer- ence on Artificial Intellience, volume 38, pp. 700-708, 2024.
sarial noise. Diffusion-based defenses, such as DiffPure, demonstrate strong performance in classification tasks but struggle in IQA tasks, emphasizing the need for task specifc adaptations of defenses and the importance of diverse and representative training datasets. Furthermore, the results highlight the critical role of randomness in mitigating at- tacks in the adaptive case.
Bhattad, A., Chong, M. Liang, K., Li, B., and Forsyth, D. A. Unrestricted adversarial examples via semantic manipulation. arXiv preprint
2019.
Bing, M. A behind the scenes look at how bing improving image search quality. https://blogs. binss.iii.oooo....oo.. com/search- at- muiliggiinshhhss200030022222bbb whwe-sceees-oollee 2013.
We conducted large-scale subjective study with 60,000+ responses to evaluate the perceptual quality of purification defenses and collect Mean Opinion Score values for a new dataset, which serves as a valuable resource for adversarial training and detection of adversarial perturbations.
Carlini, N., Tramer, F., Dvijotham, K. D., Rice, L., Sun, M., and Kolter, J. Z. (certified!!) adversarial robustness for free! In The Eleventh International Conference on Learning Representations, 2022.
Robust IQA metrics are essential for applications such as search engine optimization, video processing, and bench- marking, where adversarial vulnerabilities threaten trust and fairness. By publishing a dataset, detailed results, and an online leaderboard, we establish transparent and practical foundation to the research community and industry, enabling the development of more secure IQA models.
Chen, B., Zhu, L., Kong, C., Zhu, H., Wang. S., and Li, Z. No-reference image quality assessment by hallucinating pristine features. IEEE Transactions on Image Processing. 31:6139-611 6151, 2022a.
Impact Statement
Chen, C., Mo, J., Hou, J.,, Wu, H., Liao, L., Sun, W., Yan, Q. and Lin, W. Topiq: A top-down approach from se- manties to distortions for image quality assessment. IEEE Transactions on Image Processing, 2024.
This paper presents work whose goal is to advance the field of Machine Learning. There are potential societal conse- quences, none which we feel must be highlighted here.
Chen, Z., Jin, K., Wang, J., Nie, W., Liu, M., Anandku- mar, A., Li, B., and Song. D. Densepure: Understanding diffusion models towards adversarial robustness. In Work- shop on Trustworthy and Socially Responsible Machine Learning, NeurlPS 2022, 2022b.
ACKNOWLEDGMENTS
This work was supported by a grant, provided by the Min- istry of Economic Development of the Russian Federation in accordance with the subsidy agreement (agreement iden- tifier M00000C313925P4G0000 and the agreement with the Ivannikov Institute for System Programming of the Russian Academy of Sciences dated June 20, 2025 No. 139-15-2025- 011.
Chiang, P.-y., Curry, M., Abdelkader, A., Kumar, A., Dick- erson, J., and Goldstein, T. Detection as regression: Certi- fied object detection with median smoothing- Advances Neural Information Processing Systems, 33:1275-1286. 2020.
The research was carried out using the MSU-270 supercom- puter of Lomonosov Moscow State University. We also would like to express our gratitude to Mikhail Pautov for discussing the results of this research.
Chistyakova, A., Antsiferova, A., Khrebtov, M., Lavrushkin, S., Arkhipenko, K., Vatolin, D., and Turdakov, D. Increas- ing the robustness of image quality assessment models

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
through adversarial training. Technologies, 12:220, 11 2024. doi: 10.3990tachnollogies2110000
Hammoudeh, Z. and Lowd, D. Reducing certified regression certified classification for general poisoning attacks. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 484-523, 2023. doi:
Cohen, J., Rosenfeld, E., and Kolter, Z. Certified adver- sarial robustness via randomized smoothing. In Interna- tional Conference on Machine Learning. pp. 1310-1320. PMLR, 2019.
10.1 109999 0020445555.223004444
Ho, C.-H. and Vasconcelos, N. Disco: Adversarial defense with local implicit functions. Advances in Neural Infor- mation Processing Systems, 35:23818 - 2022.
Competition Page. Nips 2017: Adversarial learning develop- ment set. https://wwww.i kaggle. com/lataaeetsgoooglee brain// set, 2017.
Hosu, V., Lin, H.,, Sziranyi, T., and Saupe, D. Koniq- 10: 10k:
An ecologically valid database for deep learning of blind image quality assessment. IEEE Transactions on Image Processing, 29:4041-4056 2020.
Croce, F. and Hein, M. Reliable evaluation of ad adversarial robustness with an ensemble of diverse parameter-fiee attacks, 2020.
Croce, F., Andriushchenko, M., Sehwag, V., Debenedetti, E., Flammarion, N., Chiang, M., Mittal, P., and Hein, M. Robustbench: a standardized adversarial robustness benchmark. In Thirty-fifth Conference on Neural Infor- mation Processing Systems Dutasets and Benchmarks Track, 2021. URL https://opemeeiive..e///rommmmi net/forummid- SSKZPJC17B.
Huang, S., Papernot, N., Goodfellow, I., Duan, Y., and
Abbeel, P. Adversarial attacks neural network policies. arXiv preprint arXiv: 022842284 2017.
Huang, Z., He, Y., Yu, J., Zhang, F., Si, C.,, Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al. Vbench: Compre- hensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21815 2024.
Croce, F., Andriushchenko, M., Singh, N. D., Flammarion, N., and Hein, M. Sparse-rs: a versatile framework for query-efficient sparse black-box adversarial attacks. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pp. 6437-6445, 2022.
Ilyas, A., Engstrom, L., Athalye, A., and Lin, J. Black-box adversarial attacks with limited queries and information. International Conference on Machine Learning, pp. 2137-2146. PMLR, 2018.
Ding, K., Ma, K., Wang, S., and Simoncelli, E, P. Compari- son of full-reference image quality models for optimiza- tion of image processing systems. International Journal of Computer Vision, 129(4): (9944)2258-122 12-1, 2021.
Kashkarov, E., Chistov, E., Molodetskikh, I., and Vatolin, D. Can no-reference quality-assessmeent methods serve perceptual losses for super-resolution?, 2024.
Dong, Y., Fu, Q.-A,, Yang. X., Pang. T., Su, H., Xiao, Z., and Zhu, J. Benchmarking adversarial robustness on image classification. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 321-331, 2020.
Korhonen, and You, Adversarial attacks against blind image quality assessment models. In Proceedings of the 2nd Workshop on Quality of Experience in Visual Multimedia Applications, pp. 3-11, 2022.
Kruskal, W. H. and Wallis, W. A. Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association, 47(260):583-62 1952. doi: 10. 10.1188006621499.1952.0444333333 URL 01621459. 111.599.9952.111884.4
Fang, Y., Zhu, H., Zeng, Y., Ma, K., and Wang, Z. Percep- tual quality assessment of smartphone photography. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3677-3686, 2020.
Ghildyal, A. and Liu, Attacking perceptual simi- larity metries. pre-print, 05 2023. doi: 10.48550/ arXiv.2305.0888 0084000
Kurakin, A., Goodfellow, I. J., and Bengio. S. Adversarial examples in the physical world. In Artificial Intelligence Safety and Security, pp. 99-112. Chapman and Hall/CRC. 2018.
Guo, C., Rana, M., Cisse, M., and van der Maaten, L. Coun- tering adversarial images using input transformations, 2018.
Li, C., Zhang, Z., Wu, H., Sun, W., Min, X., Liu, X., Zhai, G., and Lin, W. Agiqa-3k: An open database for ai- generated image quality assessment. JEEE Transactions on Circuits and Systems for Video Techmology. 34(8): 6833-6846, 2024. doi: 10. 1011/CS/TTTTTC 2023333322233310220
Gushchin, A., Chistyakova, A., Minashkin, V., Antsifer- ova, A., and Vatolin, D. Adversarial purification for no-reference image-quality metrics: applicability study and new methods, 2024.
10

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
Li, D., Jiang, T., and Jiang, M. Norm-in-norm loss with faster convergence and better performance for image qual- ity assessment. In Proceedings of the 28th ACM Interna- tional Conference on Multimedia, pp. 789-797, 2020.
Shumitskaya, E., Antsiferova, A., and Vatolin, D. S. Fast adversarial cnn-based perturbation attack on no-reference image- and video-quality metrics. In Maughan, K., Liu, R., and Burns, T. F. (eds.), The First Tiny Papers Track at ICLR 2023, Tiny Papers @ ICLR 2023, Kigali, Rwanda, May 5, 2023, OpenReview.net, 2023, URL
Lin, H., Hosu, V., and Saupe, D. Kadid- 10k: A large-scale artificially distorted iqa database. In 2019 Tenth Interna- tional Conference on Quality of Multimedia Experience (QoMEX), pp. 1-3. IEEE, 2019.
Shumitskaya, E., Antsiferova, A., and Vatolin, D. Towards adversarial robustness verification of no-reference image- and video-quality metrics. Computer Vision and Image Understanding, 240: 240:10391: 103913, 2024.
Liu, Y., Yang. C., Li, D., Ding, J., and Jiang, T. De- fense against adversarial attacks on no-reference image quality models with gradient norm regularization. In 2024 IEFF/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pp. 25554-25563, 2024, doi: 10.1 10///VPR5522333 2024.02244.....
Singh, N. D., Croce, F., and Hein, M. Revisiting adversarial training for imagenet: Architectures, training and gener- alization across threat models. ArXiv, abs/2303.01870 2023. URL 257353366.
Lloyd, S. Least squares quantization in pem. IEEE Trans- actions on Information Theory, 28(2): 137-137, 1982.
Su, J., Vargas, D. V., and Sakurai, K. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Compulation, 23(5):828-84) 2019.
Luo, C., Lin, Q., Xie, W., Wu, B., Xie, J., and Shen, L. Frequency-driee driven imperceptible adversarial attack on se- mantic similarity. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 15315-15324, 2022.
Wang, J., Chan, K. C., and Loy, C. C. Exploring clip for as- sessing the look and feel of images. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp.
Meftah, H. F. B., Fezza, S. A., Hamidouche, W., and Déforges, O. Evaluating the vulnerability of deep learning-based image quality assessment methods to ad- versarial attacks. In 2023 11th European Workshop on Visual Information Processing (EUVIP), pp. 1-6. IEEE, 2023.
2023.
Wang. X., Xie, L.,, Dong, C., and Shan, Y. Real-esrgan: Training real-world blind super-resolution with pure syn- thetic data. In International Conference Computer Vision Workshops (ICCVW), 2021.
Moon, S., An, G., and Song, H. O. Parsimonious black-box adversarial attacks via efficient combinatorial optimiza- tion. In International Conference on Machine Learning, pp. 4636-4645. PMLR, 2019.
Wang. Z. and Simoncelli, E. Maximum differentiation (mad) competition: A methodology for comparing com- putational models of perceptual quantities. Journal of Vision, 8(12):8-8, 8, 2008
Nie, W., Guo, B., Huang, Y., Xiao, C., Vahdat, A., and Anandkumar, A. Diffusion models for adversarial purifi- cation, 2022.
Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-6) 612,0 2004.
Ran, Y., Zhang, A.-X., Li, M., Tang, W., and Wang, Y.- G. Black-box adversarial attacks against image quality assessment models. arXiv preprint 1xXiv:2402.17333 2024.
Wong, E., Rice, L., and Kolter, J. Z. Fast is better than free: Revisiting adversarial training. ArXiv, abs/2001 103994,, 2020. URL https://api.semmniisss www.amaiiiccoooaarorrrorrrrssess org/CorppsII: 210164926.
Reich, C., Debnath, B., Patel, D., and Chakradhar, S. Differentiable jpeg: The devil is in the details. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, January 2024. doi: URL http::/dx.doi.. .or/
Wu, R., Yang. T., Sun, L., Zhang, Z., Li, S., and Zhang, L. Seesr: Towards semantics sewaet aware real-world image super- resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 25456 25467, 2024.
10. 109//aavx7701..02244 10. N0.1109/WACVV7771..2222. 000408.
Salman, H., Sun, M., Yang, G., Kapoor, A., and Kolter, J. Z. Denoised smoothing: A provable defense for pretrained classifiers. Advances in Neural Information Processing Systems, 33:21945-21197, 2020.
Xu, W., Evans, D., and Qi. Y. Feature squeezing: Detect- ing adversarial examples in deep neural networks. In Proceedings 2018 Network and Distributed System Se- curity Symposium, NDSS 2018. Internet Society, 2018

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
doi: 10. (..14222ndss.2218822111 URL http://xx.oioorrrg 10.14722/dss.2188.23/11
Yang. C., Liu, Y., Li, D., and Jiang. T. Exploring vulnera- bilities of no-reference image quality assessment models: A query-based black-box method. IEEE Transactions on Circuits and Systems for Video Technology, 2024.
Yang. S., Wu, T., Shi, S., Lao, S., Gong, Y., Cao, M., Wang, J., and Yang, Y. Maniqa: Multi-dimension attention network for no-reference image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1191-1200, 2022.
Ying, Z.,, Niu, H., Gupta, P., Mahajan, D., Ghadiyaram, D., and Bovik, From patches to pictures (paq-2-piq): Map- ping the perceptual space of picture quality. In Proceed- ings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3575-3585, 2020.
Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, F. S., Yang, M.-H., and Shao, L. Multi-stage progressive image restoration. In CVPR, 2021.
Zhang, A., Ran, Y., Tang, W., and Wang, Y-GG. Vulnerabili- ties in video quality assessment models: The challenge of adversarial attacks. Advances in Neural Information Processing Systems, 36, 2024.
Zhang, W., Li, D., Min, X., Zhai, G., Guo, G., Yang, X., and Ma, K. Perceptual attacks of no-reference image quality models with human toop- pre-print, 10 2022a. doi: 10.48550/arXiv wttp://aaai........... 2210.00933.
Zhang, W., Li, D.. Min, X., Zhai, G., Guo, G., Yang, X., and Ma, K. Perceptual attacks of no-reference image quality models with human-in-the-loop. Advances Neural Information Processing Systemx, 35:2916-2929, 2022b.
Zhu, H., Li, L., Wu, J., Dong, W., and Shi, G. Metaiqa: Deep meta-learning for no-reference image quality as- sessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14143- 14152, 2020.
12

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
A. Appendix
A.1. Limitations
While the proposed framework for benchmarking defenses against adversarial attacks on IQA metrics offers significant contributions, we acknowledge the existence of the following limitations to be addressed in future work:
1. Handling Multiple Parameter Attacks: The current framework deals mainly with attacks that have a single parameter. However, Some attacks might have multiple parameters control their strength, complicating the evaluation process. Moreover, group of Boundary Attacks adapts their parameters according the response of the attacked model, which poses additional challenge in fixed axed-parameter setting. Future versions will include methods for dealing with different types of evolving attacks, possibly through dynamic parameter optimization techniques
2. Transferability of Adversarial Attacks:
There might be defenses that better generalize to attacks produced on other defenses. Currently, the framework does not evaluate the transferability of adversarial attacks among different defenses. Future versions should provide insights into the generalizability and robustness of the defense.
3. Simplified Ranking Methodology: The current framework employs a straightforward ranking methodology that may not fully capture the complexity and the existence of different evaluation metries with varying importance levels depending on the attack used for testing. Different evaluation measures can be assigned different weights based on their importance and relevance the attack. This system allows for composite score that reflects the overall performance of a defense mechanism. To provide nuanced assessment of metric robustness, more rigorous statistical framework for ranking metrics will be employed in future versions.
Addressing these limitations in future work will ensure the framework's 's robustness and adaptability in diverse and realistic scenarios.
A.2. Subjective study
To assess the perceptual quality of defended images we have conducted large-scale crowd-sourced subjective study on the Subjectify.us, a platform that employs the Bradley-Terry model to convert pairwise comparisons into numerical scores. We selected 5 attack methods in adaptive setting (Zhang et. al.-DISTS, IFGSM, Korhonen et. al., UAP), 4 IQA models (Lincarity, MANIQA, TOPIQ, CLIP-IQA+), and 12 purification defenses. Non dinfdifferentiable methods (Color quantization, JPEG) and defenses with obvious perceptual differences (FCN, Crop, Flip, Rotate) were excluded from the study. The attacks were applied in an adaptive setting 20 randomly selected source images, resulting in 5,200 images to compare (4 IQA model 5 attacks (12 defenses
W/o defense) 20 source images 5200).
Each participant was given this instruction: "You will be shown one original image and two modified ones. For each pair of selected images, select the one that seems to you to be of higher quality, more realistic, and closer to the original image".
Because the number of pairwise comparisons grows quadratically with the number of images to compare, we divided the dataset into 400 subjective evaluations, each focusing on single source image, attack, and IQA model combination. Each pair received least 10 votes, with responses from participants failing verification questions being discarded. Bradley-Terry scores were computed for each defense method per evaluation and averaged across 400 subjective evaluations determine the final rankings.
Figure 5 illustrates the results of this experiment. Subjective scores are along X-axis, Rscore are across Y-axis.
A.3. Details of methodology
A.3.1. DATASETS
In Table 4 we provide information about the datasets used in our study.
We employ 4 datasets: KonIQ-10k (10,073 images) (Hosu et al., 2020) and KADID (10, images) (Lin et al., 2019).
NIPS 2017: Adversarial Learning Development Set (1000 images) (2017, (Competition Page)), AGIQA-3K (2982 images) (Li et al., 2024). KonIQ-10k, KADID and AGIQA-3K contain MOS scores that is used in evaluating correlations with grouhd-truth labels. The datasets were chosen to represent different distortion and generation types.
13

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
DMPPG
0.7
0.6
Blinear Upscale
Defense
ESRAAE
MPRNct
Caussion Blur
0.5
DISCO
Undharg
Clarr
Secter
Subjective score
Figure 5. Results of subjective evaluation. Subjective scores are along X-axis, Recore are across Y-axis.
Table 4. List of datasets used in our benchmark. These dataset consist of clean images. Based on them, we constructed new dataset of adversarial images with MOSes, that consists of 5,200 images.
Dataset
Size
Resolution
Suhjective ratings
Short description
KonIQ-10k
1,000 (out 10,073)
512 x 384
120,000
Provides wide range of real-world photos with authentic distortions
KADID-1Ok
8 out of 81 original images
512 x 384
30,000
Large-scale dataset with wide variety of content and artificial distortions Competition on adversarial examples and defenses in the NIPS 2017 AGIs from seattoorrgrossoondffffiiee
NIPS 2017
1,000
299 x 299
AGIQA-3K 2,982
512 x 512
125,244
model with subjective scores
We use subsets of 10K (Hosu et al., 2020) and KADID (Lin et al., 2019) to evaluate adversarial defense methods. Both subsets of KonIQ-10K and KADID contain 1000 images at 512x384 resolution. KADID subset contains all distortions from 8 images, resulting in 1000 total images, that contains all distortions and do not decrease its diversity. Original KonIQ- 10K dataset was partitioned into 10 clusters using K-Means (Lloyd, 1982) based on 3 parameters: Spatial Information (SI) Colorfulness (CF), and Mean Opinion Scores (MOS). We selected 100 random images from each cluster, resulting in a diverse of 1000 test images regarding quality and content. Due significantly higher computational complexity, we used smaller set of 50 images for black-box attacks. They were sampled images from each out of 10 clusters using K-Means (Lloyd, 1982) based on 3 parameters: Spatial Information (SI), Colorfulness (CF), and Mean Opinion Scores (MOS) where possible. For the same reason, we used smaller set of 10 images for certified defenses to generate attacks. To evaluate the impact of sampling this procedure was repeated 10 times, focusing on purification methods and black-box attacks to accelerate calculations. We used default parameters for defense methods and 3 presets from the main paper for attacks. For each IQA model, attack and defense method, we calculated four scores per sample: D Decores S SROCCdear. SROCCadv. and SSIM.
Figure 6 illustrates the distribution of these scores for each sample for KonIQ-10k dataset. The results show that the distributions are nearly identical across all samples and metrics, with consistent mean values. To assess the differences between the means of distributions, we computed the mean for each distribution and score, yielding a list of 10 mean values per score. Then, we calculated the mean and variance of these values across the 10 samples. These values can be found in Table 5. To verify these findings statistically, we performed a Kruskal-Wall test (Kruskal & Wallis, 1952) for each metric across the 10 samples. The p-values are shown Table These p-values indicate no significant differences between the samples, confirming that the sampling procedure does not introduce variability into the evaluation results. This consistency strengthens our conclusions and ensures that the findings robust across different random subsets of the dataset.
A.3.2. IQA METRICS
We define metric range as diam(fu) = sup fu())) = upper - lower, where upper is called the upper metric xx
bound and lower- lower metric bound. To calculate these bounds, we used the DIV2K valid HR subset from the DIV2K dataset (Agustsson & Timofte, 2017). The upper bound is set to the highest metric value across the chosen subset, while the lower bound is set to the minimum value between the lowest metric value on subset images compressed with JPEG with quality of 10 and sampled random noise of the image subset size.
14

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
0.1
Velue
BROCC
SROCCWWr
SSIM
Figure 6. effect of sampling images on results. Each box represents one experiment.
Table 5. Validity of sampling methodology. To validate our sampling strategy we compare results of defenses on 10 different samplings of 50 images. Results the Kruskal-War -allis test suggests that samplings have no significant differences.
Variance of means
p-value after
Score
Mean
for each sample
Kruskal- Wallis test
Decore
0.1533
0.000044
0,5425
SROCCude
0.4681
0.00043
0.1449
S SROCCalaar
0.7343
0.00066
0.1958
SSIM
0.7953
0.000034
0.1 138
Table 6. List of NR IQA metrics used in benehmark.
Metric
Racove
Backbone
Number of parumeters
Input transformations
Bounds
Code
Meta-IQA (Zhu et al., 2020)
1.168
ResNet-18
13.2M
ImageNet Normalization
0.00/1.00
Github
MANIQA (Yang al., 2022)
0.986
VIT-B/8
135 35..2M
224 x 224 crop
0.00/1.00 .0
Github
Koncept (Hosu
2020)
0.584
InceptionResNer22
59.82M
Normalization (0.5, 0.5) 224 224 crop
26.40/66.87 87
Github
SPAQ (Fang et al., 2020)
0.493
ResNer-50
23.5M
21.75/77.75
Github
PAQ2PIQ (Ying et al., 2020)
0.449
ResNet-18
11M
58.38/84.17
Github
Linearity (Li et al., 2020)
0.267
ResNeXt-101
90M
ImageNet Normalization
25.78/83.23
Github
Splitting into fixed-sized patches
FPR (Chen al., 2022a)
-0.229
Custom
16.6M
47.22/77.05
Github
CLIP-IQA+ (Wang et al., 2023)
0.713
CL.IP
244M
0.00/1.00
Github
TOPIQ (Chen al., 2024)
Transformer
45M
0.22/0.82
Github
We report metric ranges and parameters in Table 6. The Rscore is taken from (Antsiferova et al., 2024).
A.3.3. USED ADVERSARIAL ATTACKS
Early methods for attacking IQA metries aimed to stress-test performance. (Wang & Simoncelli, 2008) introduced the MADC method, which uses gradient projection onto a proxy FR metric to compare accuracy. Later, (Kurakin al., 2018) proposed I-FGSM, iteratively adding gradients to the image, but this caused visible distortions. (Korhonen & You, 2022) addressed this by targeting high-textured regions using Sobel-fiter-bbsed weighting (Korhonen et al.). while (Zhang et al., 2022b) incorporated FR IQA metrics like DISTS and LPIPS into loss functions (Zhang et al.). SSAH by (Luo et al., 2022) limited attacks to high frequencies, and (Bhattad et al., 2019) introduced cAdv, operating in LAB color space. Efficient universal perturbation methods, such as UAP and FACPA, climinate backpropagation during inference, with FACPA further optimizing for high-resolution data.
15

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
Table List of adversarial attacks used in our benchmark. WB and BB white-hox and black-hox attack types. We adjust varied parameters to align attacks' strengths.
Adversarial attack
Type
Restriction
Varied parameter
Short description
I-FGSM (Kurakin et al., 2018)
WB
1oo
Ir
Grad. descent increase IQA metric Universal perturb. via grad. descent Sobel-fittr-masker gradient descent Grad. descent with saving DISTS Grad. project. onto MSF Grad. descent with recolorization Grad. descent with high-freq. min.
Optimised- -aiessddd------- (Shumitskaya et al., 2024)
WB
160
amplitude Ir
Korhonen et al. (Korhonen You, 2022)
WB
loo
Zhang et al. (Zhang et al., 2022b)
WB
Ioo
Ir
MADC (Wang& & Simoncelli, 2008)
WB
Ico
cAdv (Bhattad al., 2019)
WB
SSIM
Ir
SSAH (Luo et al., 2022)
WB
1o0
Ir
FACPA (Shumitskuya al., 2023)
WB
100 lou PSNR
amplitude c
Perturb. generated using U-Net Grad. descent with approx. gradient Perturbs using discrete optimization Perturbs pixels with diff. evolution Square-like perturb. via rand. search Finds adv. patch via random search
NES (Ilyas et al., 2018)
BB
Parsimonious (Moon al., 2019)
BB
One Pixel (Su et al., 2019)
BB
to
pixel count
Square (Andriushchenko et al., 2020)
1s0
Patch-RS (Croce al., 2022)
BB
PSNR
For black-box attacks, we adopted efficient methods originally designed for image classifiers. NES (Alyas et al., 2018)) estimates gradients using natural evolutionary strategies, while the Parsimonious attack ((Moon et al., 2019)) identifies sparse pixel-level perturbations through hierarchical updates. Square attack ((Andriushchenko al., 2020)) applies square patches in random search algorithm. Representing sparse attacks, Patch-RS ((Croce et al., 2022)) uses random search to place patches. Finally, One Pixel ((Su et al., 2019)) uses the Differential Evolution algorithm alter minimal pixels.
Descriptions of these methods and varied parameters are detailed in Table 7.
A.3.4. CHOOSING PARAMETERS FOR ATTACKS
To align attacks by strength across different metrics and defenses, we developed the methodology illustrated in Figure 7. For each attack, a restriction metric was selected: PSN R, SSIM or Loc. While Loe is suitable for most attacks, SSIM is more appropriate for color-based attacks. For unrestricted attacks, such as Parsimonious and One Pixel, PSN is the most suitable. For these restriction metrics we defined three sets of target values corresponding to "weak", "medium" and "strong" attacks. Specifically;
For Loos the values 255 255 and 255 were chosen to be consistent with previous studics
For SSTM, 0.75, 0.8 and 0.9 were chosen
For PSNR, R, the values 25, 30 and 35 were used
For each attack, a single parameter corresponding attack strength was varied (see Table 7). We computed the mean attack strength on the KonIQ- 10k dataset for 10 values of this parameter and performed a linear approximation to match the target strength values.
A.3.5. EVALUATED DEFENSES
Purification
According to (Guo al., 2018), several standard image preprocessing techniques can be used defenses against additive adversarial noise. These methods include compression (JPEG, DiffJPEG (Reich al., 2024) color quantization (Xu et al., 2018)), spatial transformations (Resize, Rotate, Crop, Flip), blurring (Median blur, Gaussian blur, etc.), Unsharp masking, and others. Although not originally designed for adversarial defense, studies have demonstrated that these methods can be effective.
noise, denoising techniques can be particularly useful. The Multi- Multi-Stage Progressive Image Restoration Network (MPRNet (Zamir al., 2021)) is three-stage convolutional neural
Since adversarial perturbations often consist
16

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
Preliminarity
Attack evaluation
Approximation
Choose attacks' budget metric
Attacking model with defined grid of parameters
For each of three desired perturbation budget values:
IQA metric
MMNN, S8/M,
1. Choose two closest parameters
For each parameter value: calculate attack intensity with budget metric
2. Find desired attack parameter via linear approximation
Three perturbation budgets corresponding budget metric
Attack method
get attack
Findl attack parameters
Attacks' one parameter vary
Dataset 50 mages
Choose grid for this parameter
Porameher value
Pansmater ve2
Figure 7. Procedure for selecting adversarial attack parameters.
Table 8. List of compared adversarial Purification methods.
Defense method
Type
Varied parameter
Varied parameter values
Fixed parameters
Code
JPEG (Gao al., 2018)
Compression
10, 30, 50, 70, 90
Di (Reich et al., 2024)
Compression
10, 30, 50, 70, 70,90
Github
Color quantization (Xu 2018)
Compression
upp
2.5, 16, 20,
Resize (Guo al., 2018)
Spat. transform.
scale
0.25, 0.5, 0.75,
Bilinear Upscale
Spat. transform.
scale
0.1, 0.25, 0.5. 0,75,
Rotate
Sput, transform.
angle lim.
10, 15, 20, 30, 50
Crop (Guo et al., 2018)
Spat. transform.
size
32, 64, 128, 256, 288
Flip
Spat. transform.
Gaussian blur
Blurring
kernel size
3. 11 7. 11 3,
sigma=0. 0.15 *kemel size+ 0.35
Median blur
Blurring
kernel size
Unsharp masking
Preprocessing
kernel size
sigma=1, amount=1
MPRNet (Zamir et al., 2021)
Denoising
Github
outscaleml, tile=0, tile pad==100110, pre pad=O)
Real-ESRGAN (Wang et al., 2021)
Denoising
Github
DitPPure (Nie et al., 2023)
Defense
5, 10, 20, 30, 50
1 delta= 15, diffission type=ddpm, sample step=1
Github
DISCO (Ho & Vasconcelos, 2022)
Defense
Github
FCN (Gushchin et al., 2024)
Defense
Github
Random noise
Adding noise
network for image deblurring, deraining, and denoising. The first two stages use an encoder-decoder architecture for multi-scale contextual information, while the final stage operates at the original resolution to preserve details. MPRNet features supervised attention modules and cross-stage feature fusion for effective information transfer. Real-ESRGAN (Wang et al., 2021), a GAN-based model with several residual dense blocks for super-resolution, is trained with synthetic data and can be used for adversarial denoising.
We also included methods designed specifically as adversarial defenses for image classifiers IQA models. DiffPure (Nie et al., 2022) employs diffusion models to purify adversarial images by introducing small amount of noise through forward diffusion, and then reversing the process to recover clean image. DISCO (Ho & Vasconcelos, 2022) is an image purification method aimed at enhancing classification robustness. It employs local implicit functions to ensure small perturbations do not significantly alter local data representations. By maintaining these robust local representations, DISCO effectively resists adversarial perturbations that do not align with the data's local structure, Some adversarial attacks, such as color-based modifications, are not bounded. Standard denoising approaches are ineffective against these. (Gushchin et al., 2024) proposed neural filter FCN, counter color attack cAdv on image quality metrics. FCN features a compact, fully convolutional architecture with three hidden layers of 64, 32, and 3 filters
We report the list of parameters for defenses and fixed values for non-varied parameters in Table 8.
Adversarial training We applied the method proposed in (Chistyakova et al., 2024). More specifically, we fine-tuned Linearity and Koncept IQA models using the original images and attacked images in a 1:1 ratio from the original KonIQ- 10K training dataset for 30 epochs. During the training process, we used a 2- step APGD attack (Croce & Hein, 2020) to generate
17

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
the attacked images. This method uses an adaptive step that allows a small number of iterations to achieve strong adversarial examples and reduce computational time. The goal of the attack during the training process is to increase model loss. We adjusted the MOS values based on the FR metric scores. For given original image with MOS y, we obtain the adjusted MOS for the attacked image x' as follows:
y- - 2') y' y
(10)
We have considered LPIPS and 1 SSIM as M. To evaluate the impact of attack magnitude during training we chose 3 different attack magnitudes e = (2, 4, 8//55
We also evaluated method proposed in (Liu et al., 2024). Specifically, the approach introduces a gradient norm regularization term into the training objective to enhance the robustness of NR IQA models. The regularization term penalizes the T1 norm of the gradient of the model's 's quality predictions with respect to the input image, thereby encouraging smoother model behavior and reducing vulnerability to adversarial perturbations. Formally, the loss function is modified as follows:
Ctotal = (f. 2) + AIIV
(11)
where f denotes the NR IQA model, x is the input image, LIqA (f. x)) represents the original IQA loss, and À is a hyperparameter balancing the trade-off between task performance and robustness. For our experiments, we set X = 0.0005, as suggested in the original paper. The remaining training hyperparameters were aligned with those used for adversarial training.
Both these methods are tailored for IQA task.
Certified methods
Description. (Cohen et al., 2019) proposed the Randomized Smoothing (RS) method to transform any classifier that performs well under Gaussian noise into a new classifier that is certifiably robust to adversarial perturbations under the lz norm. The overall process of this defense can be described as follows: given an input image, the algorithm samples N noisy variations of this image using a Gaussian noise model with certain a. These images are then passed through the backbone classification model, and the most frequently predicted class is given as the final answer. This approach results in an algorithm that provides provable answer for the model within a l2 ball. The radius of this ball is caleulated based on the difference between the most popular and the second most popular classes across the sampled images used for answer selection. The main disadvantage of the previous approach is that running the classifier on noisy data causes drop in model accuracy, as it was not trained to handle such data. To address this issue, (Salman et al., 2020) extended randomized smoothing to Denoised Randomized Smoothing (DRS) by denoising the noisy image before passing it to the model. Since the noise model is known, training an effective denoiser for a given a is relatively straightforward. (Carlini et al., 2022) extended the approach of (Salman et al., 2020) by replacing the denoiser with a pre-trained denoising diffusion probabilistic model (Diffusion Denoised Randomized Smoothing (DDRS)). They used only one diffusion step because it demonstrated high speed and relatively good quality. (Chen et al., 2022b) proposed the DensePure (DP) method that involves multiple runs of denoising via the reverse process of the diffusion model (using different random seeds) to generate multiple samples. These samples are then passed through the classifier, and the final prediction is made using majority voting.
(Chiang et al., 2020) proposed a method to certify regression models. Instead of using the most popular class within the l2 ball, they utilize the median of function values. They also theoretically demonstrated that using the median is better than the mean. We denote this method as Median Smoothing (MS). They further extended the method to Denoised Median Smoothing (DMS) by adding denoising step before model prediction to improve accuracy.
Parameters selection. Given an input image, the results of the dlassificttion-aasee certified method are the metric score and the certified radius R. The method guarantees that the class remains unchanged for the input image within a la ball of radius R. All classification-baat 1000, a = 0.001. Here, o is the standard deviation of the Gaussian noise used for sampling, No is the number of samples for class selection, N is the number for class certification, and alpha is the probability of class change within the le ball of the predicted certified radius/ R.
certified methods were run with the following parameters: a = 0.12, No = 100, N =
Given an input image, the results of a regression-based certified method are the metric score and the certified delta. The method guarantees that, within a la ball of radius e, the metric score changes by no more than delta. To make this value comparable across metrics, we define the certified relative delta by dividing the certified delta by the metric range. All regression- eegression-based certified methods were run with the following parameters: o = 0.12, e = 0.05, N = 1000, a = 0.001.
Scripts for running all these methods are available on GitHub.
18

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
Table 9. Experiment to determine the optimal number of classes N for regression metric discretization.
SROCColear t (no Monte-Carlo sampling)
Cert.R t
N
(with Monte-Carlo sampling)
3
0.49
0.249
5
0.53
0.248
10
0.56
0.206
15
0,56
0,160
20
0,56
0.142
8
0.56
0
Table 10. Wilcoxon tests nonadaptive use case of purification defenses on KoniQ dataset for D Each cell value represents the percentage of experiments in which defense denoted in row statistically performs better in terms of D Dloor than the defense in corresponding column with #0055
Deiieee
Bilinzz
Upscale
Unstarp
Redino
Romen
Chop
Maalien Iller
JP9G
Connoian Blux
Coloe Quantication
DellPPrn
Randoo Noise
Flip
MIP9NN11
PCN
Raal- ESRGAN
DISCII
Detense
Wio
DimPeG
65.35 2455
75 01.5
25.550 55,365
8.55%
$3,0555
55.46%
9.60%
90.325
$8,1250
0.855
3,70%
15.059;
31.05%
45.87%
27.07%
1425%
85.08%
Bilinz Upante
6.84%
62.35 39%
13.39%
6.22%
0.92%
5.20%
4.44%
0.00%
25,775
0,0052
1.99
8.26%
21.08%
16.81%
7,125
0.025
83.48%
Uesherp
010055
1,71%
5.135
0.2835
01855
0.85%
01055
1.14%
2.365
0,0556
0.055
8,35
0.00%
0005
1555
Resine
11,855
$4.475
79.30%
5.2055
40.1755
90.4%5
40.17%
45 8777
57 57.89%
28.359
11.87%
42.74%
47 57.545
51%
00055
81 81.30%
Rouae
56.41%
30.9455
90.6275
46.15%
09.29%
63.53%
63.55%
72.36%
13.96%
49,007
51 51.0%
10,26%
1771775
Crop
33.05%
601.07%
17.38%
8.55%
44.44%
36.88
53,9775
1033%
29.325
11,005
37.04%
09.0%
41.67%
10,377
35.32%
Medias
22.7995
58.69%
29.345
63:0735
15.85%
27.35%
47.29%
8.8%%
90.0815
JPGG
0.0015
21.94%
8.83%
33.35
59.545
3.999
25.6
0.85%
85.597
Queniiaaioo
2.2855
1.955
6.94%%
0.09%
DettPure
MAM374
95.75%
60.97%
34.35%
75.545%
16-617
44.73%
66.67%
75.21%
67.53%
39.8V2
15.16%
Random
73.95%
64.1166
68.07%
71.3276
7.9866
81.91%
File
35.611
51.57%
66.
34.26%
8.55%
36.185
45,550
24.95
435775
54-4255
9.6955
22.793
43.30%
47.58%
33.90%
541%
MPRNer
28,497
53.28%
135%
23.95%
10.20%
27.6456
700
44,1625
2.85%
10.263
15.3905
39.62%
25.92%
2.28%
62.68%5
FCN
0.97%
42.45%
82.91%
18.22%
4.22%
21 21.4%
39-2225
1.4250
5418
4.56%
24.79%
15.35%
2.85%
ESRGAN
20.015
19.54%
42.17% 17%
25.038
33.08
37.61%
36-385
02 02-3995
10.265
20.948
19.37%
42.17%
$1.03RR
19.50%
83.195
DISCO
7849254
$3.9155
87 79%
70.37%
91 387
99 9656
26.00%
78, 19
76.85%
83 48%
25.30%
68.40'
43.0075
05.533
72.08%
58.4255
94.59%
Win Diefenso
0.003
0.0055
69.
0.00%5
0.28%
845
0.00%
0.00%
0.00
0.0055
0.0056
0.00%%
0.765E
1.99%%
0.007%
0.0075
0.005
Classifier-based methods application. To discretize a regression quality metric for classification methods, we divided the metric range into N segments, each corresponding specific class. We also added additional classes for metric values that fall below or above the calculated range, ensuring that every metric value is assigned to class. This resulted in (N + 2)-class metric-classifier. Note that these classes are ordered, with higher class values indicating better quality. Thus, we can measure the quality of the classifier metric in the same way as the regression metric- using relative gain and correlations with subjective scores.
We conducted additional experiments to determine the optimal value of N on PAQ2PIQ NR metric. The main challenge balancing the trade-off between SROCCelear and Cert.R. As the number of classes increases, SROCCdear also increases, but Cert.R decreases. This occurs because higher number of classes makes it easier cross class borders during Monte Carlo sampling. Table 9 presents the results of our experiment, indicating that when the number of classes is set to ten, SROCCatar on the discrete metric without Monte Carlo sampling is optimal. Additionally, we measured Cert.R for this number of classes and discovered that Cert. Rt does not significantly decrease for N = 10. Therefore, we chose N = 10 to discretize NR metric values in the main experiments of this paper.
A.4. Statistical tests
We applied the one-sided Wilcoxon Signed Rank Test to assess the statistical significance of defense comparisons, as it is non-parametric and suited for paired samples without assuming normality (which often the case for attacked IQA models' values) - ideal for adversarial robustness analysis. This test evaluates whether one defense consistently outperforms another in terms of D Decore and other relevant metries. Results for different datasets are provided in Tables 10, 15 (non-adaptive) and 16, 17 (adaptive).
Pairwise comparisons yield percentages indicating how often defense statistically outperforms another under similar adversarial conditions. Stronger defenses (i.e. those yielding high percentages across multiple pairwise comparisons), such as DiftPure and DISCO, show higher effeeciveness due to their sophisticated adversarial mitigation techniques, while simpler transformations like blurring or resizing often struggle against stronger attacks.
To ensure reliability, we applied the Bonferroni correction, which controls the family-wise error rate across thousands of
19

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
comparisons (17+7 empirical defenses, 14 attacks, intensity scales, and 9 IQA models). This conservative adjustment minimizes false positives, reinforcing the significance of the results.
A.5. Examples of attacks and defenses
We show examples of attacks and defenses with corresponding metric values in Figure 8. We chose the PAQ2PIQ metric and several types of defenses. The central part of the image is zoomed to show the effects of the defenses and attacks.
We show image artifacts of presented defenses in Figure 9. The attacks were performed on MANIQA metric. We demonstrate that most defenses have artifacts. Most of them include: removing details of the original image (DISCO, MPRNet), altering the image content (Real-ESRGAN, DiffPure), reducing the image clarity (DiffPure, blur defenses), changing image color (FCN), and compression artifacts (JPEG/DifNJPEG, Color Quantization).
A.6. Additional results
We show performances for evaluated defenses in tables below. Confidence intervals in some table are large due to the fact that we calculate an average score across a large pool of IQA metrris/attuckk/dattsttss. To statistically check what defense is better we provide results of statistical tests A.4.
We analyze how much does SROCC and PLCC correlations are differ in table 20. It reports that ranks in both cases identical. Thus, in other tables report only SROCC.
Table 11 shows how well defenses can respond to attack of different strengths. In summary, results do not change much across strengths.
Tables 12 and 13 present results of purification defenses on different IQA metrics. We can see that correlations are highly dependent on IQA metric, while top methods in terms of robustness (D Dcoore Recore) are similar across different IQA models.
Table 14 shows scores for different attack types. The experiment showed that undefended images are closer to the original than defended by any defense. The results are the same on all attack types. JPEG and DiffJPEG show greater S S while Color Quantization has better PSNR with the original images.
Table 18 reports results for grouped purification defenses. The results demonstrate that compression one of the best in all metrics except R
Table 21 reports some score for certified methods. It shows that RS has bigger Cert.R and Abst. among all classification- based defenses, while MS is the best among regression-bast defenses.
Figure 10 pictures tradeoffs for Purification-baxe defenses. shows the importance of tuning defense parameters for defenses with different parameters.
Figure 1I illustrates difference between Dccore D.Dr (left) and Racore and R!D (right). The main finding is that there high correlation within these pairs of scores with rare exceptions like Random Crop.
20

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
I492M10:66.348
IMQ2/90-2.665
1402PG9:7.262
IAQ2MQQ81.24
17.27
No daferse
PAOZRO.72.CC
PAOO2PI2
PAQ2PIQ
Cropped attackett
PAQ2P10:68.62
7002RO:73.194
6800P12.68.071
PAG2PIQ 145
69.21
FiP
JPEG
8402RQ:61668
66
6402222.600522
68.22
RVQ2P12
Bilewar Upocaate
1M92RG-64.856
PA02NO:71.CO
PAG2NG.28.6K
MAG2PIG
D8000
FAD2RQ:71.7
PNO2PIC.8..0
RMG2RG.77.3W
PAO2PIC 74.36
Cohor Qtanttratton
PAQ2HQ:64.77
7O0330.70.316
PN02RQ .99
742270.44.711
7006610-72.650
Iutate
CAdY
Parsimorious
HFGSM
Kott-oen
Figure 8. Examples of attacks and defenses PAQ2PIQ metric. The central part of the image is zoomed to show defense effects.
21

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
Resize
CNN-attackJPEG
Real-ESRGAN
Sottee
MPANet
CNN attackkandddo naise
MeaC RAA-RAAN
CAdv:DiffPure
Korhonen et al. Gaussian Blur
BUCKLI
CADERNET SAUVIGN
VINTAGE 2009 SONOMA VA
OLD HILL RANCH VINGYA
Karhanen Di:DiffPure
CNN attack:Colar Quantizatian
CAdv.DISCO
CAdv:Bllnnar Upscale
UAP:DIffPure
LFGSM.DifPPre
CNN- MoV-attack
MADC.DISCO
Komonnen et al.:FNN
Korhonen RSaGE
Korhonen Cl.Co Quantiration
CNW-attack:Median
CAdvIFCN
Korhonen al..Unsharp
SSAH
Figure Examples of artifacts caused by various defenses when MANIQA metric is attacked. We selectively key parts of the images to highlight the details.
22

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
23
Defense paramelers
Defense parameters
30
Strangast
Strongest
Strong
Strong
Medium
22
Medium
Week
Weak
20
Weskes!
(0)
Weakest
P5N
36
SSIM
Delerse
Upscale
DRUPEG DMP ure
Gaussian
Notoe
Rotste
Color Quaniizatio
JPEG
ReaHESRIGAN
Unshorp
Crop
FGN
MPRNel
Resiee
We Detense
DISCO
Fipp
Median Blur
Figure 10. D (V) R(t) (left) and ( tradeoff's for Purification based defenses in non-adaptive scenario averaged across KonIQ-1Ok, KADID and AGIQA-3K datasets. Red line denotes the Pareto Optimal front.
Table 11. Comparison of purification defenses by different attack strength. Evaluated metrics are averaged across all images, attacks and quality metrics on KonIQ and KADID dataset.
Weak
Medium Ryooee 1
Strong
Dacoee
Racore 1
SROCC.at
Duore
SROCC, 1
Dacore 4
Russe f
SROCC. ?
Wio Defense
33.11
0.822/
0,522/
45.88
0.639 /
0,470/
66,83/
0.502
0.401 /-
Bilinear Upscale
0,909 0.674
0.499/0.385
18.44 38.96
0.556
0.354
24.16 53.77
0.748
0.461
0.411 0.331
Gaussian Blur
23.10/27.56
0.878 0.655
0,465 / 0.421
16.52/ 38.85
0.836/ 0.518
0.437 0.337
20.13 1/3/58 58.06
0.761 00336
0,400 0.255
Resize
9.99 43.42
1.118/0.55 0.525
0.594 0.411
11.78/ / 58.27
1.053 /0.404
0.582/ / 0.347
14.15 81.16
0.995 /0.270
0.561 00.24
MPRNet
12.67 /3.77
0.993 0.602
0 0.530
14.13/42.40
0.946/ 0.490
0.556 0.513
66.60
0.360
0.546/ 0.446/ 0.432
DHI]PEG
9.71 20.72
1,138/ 0.774
0.634/0 00.53
11,96 27.29
1.057/ 0.660
0.632 0.555
16.89 37.29
0.967 / 0.547
0.619/ / 0.491
JPEG
9.72
1.139/ /
0.631
11.99/
1.054/ / --
0.629/ /
16.97 --
0.963
0.616
Unsharp
30.91 45
0.652 388
0,484 0.403
42,38 / 83.78
0.530/ 0.232
0.419 0.304
60.27
122.51
0.439 0.0097
0.376 0.248
Median Blur
34.92/34.98
0 0.574
0.462 / 0.468
/47.8/
0.915 0..47
0.434 0.424
18.09 67.41
0.856 0.282
0.404 / 0.322
Real-ESRGAN
21.48 21.48/26.51 26.51
0.689 / 0.621
0.564 0.476
/3.15/31.73
0.665 0,566
0.548 / 0.461
26.67 41.73
0.627 0.467
0,509 / 0.380
Color Quantization
15.46
1.016/ -..
0.586
21.08
0.897 /
0,568
36,81
0.726
0,499/
DISCO
8.72 40.31
1.176/0.514
0.479
8.30/ 52.05
0,438
0.612/ 0.453
8,29
0.406
0.613/ 0.49
DiffPure
17.92/ 17.92/15.99
0.780 0 869
0,501 / 0.502
/
0.800 0,766
0.515 / 0.492
17.57 / 32.05
0.797 / 0.593
0,521 /.4467
FCN
15.38 /47.23
0.973 0.471
0.566 0..44
20.74/ 67.76
0.885 0.318
0.529 0.248
30.85 / 100.34
0.771 / 00.194
0.463 0.12
Random Noise
14.72/26.84
0,907 0.688
0.576 0.578
14.58 42.29
0.921 0.490
0.572 0.517
17.
22.84
0.869 00.272
0.566 0.382
Crup
11.51 / 18.44
1.045 0.788
0.435
13.47 / 18.26
0.982 0.791
0.529 0.403
16.93 19.89
0.899 0.762
0.468 / 0.330
Rotate
9.20/ 10.65
1.153/ 1,066
0.533 0.533/0.544 0.543
9.98 15.27
L.100/ 0.886
0.520 0.477
11.21 23
1.022
/ 0.683
0.485 0.355
Flip
6.38 47.67
1,318 0.508
0.557 0.480
7 67.31
1,255 00.47
0.553 0.395
9.81 / 99.551
1.166 0.122
0.520/ 0.300
Table 12, Per-metric comparison of purification defenses in adaptive use case (SROCCater//OOOCCCCoo Evaluated metrics are averaged across all images and attacks on KonIQ and KADID dataset.
Defense
Linearity
KonCept
PAQ2PIQ
MANIQA
Meta-IQA
SPAQ
TOPIQ(NR)
IQA++IQA+
W/o Defense
0.526 0.436
0.477
0.405
0.449/ 0.349
0.497 0.465
0.617 0.446
0.355 / 0.251
-0.133 / 0.070
0.494 / /0.44
0.653/ 0.464
Crop
0.611 0.501
0.236 0.178
0.404/ 0.376
0.522
0.441
0.458 0.386
0.173 0.144
/.,11/
0.592/ 0,518
Real-ESRGAN
0.613/ 0.461
0.708/ / 0.544
0,510/ 0.414
0.786/0.61
0.278 0.303
0.438 0.333
0.295 0.265
0.576 / 0.506
0.681 0.495
Unsharp
0.631 / 0.274
0.706/ / 0.501
0,510/ 0.261
0.549 0.783
0.549
0.624 / 0.247
0.558 / 0.298
0.230/-00 00.007
0.717 / 00.43
0.693/0.388
DISCO
0.662/ / 0.554
0,519 / 0.486
0,555 / 0.423
0.630/0.555
0.643 0l.53
0.571 / 00407 - / -
0.108 / -0.075
0.719 / 00.555
0.653/ 0.653/0.530 0,530
Resize
0.676/0 0,372
0.481 / 0.328
0.510/
0,565 0.400
0.575 0.331
0.209 /-0 109
0.738 / 0.563
0.555 0,390
Bilinear Upscule
0.677 0.522
0.433 / 0.289
0.540/ 0.414
0.481 0.40
0.429 0,280
0.569 0.304
0.193 -0.044
0.644 0.561
0.607 0.417
DiffPure
0.680/ 0.643
0.515/ 0.499
0.564/ 0.483
0.558 0,584
0.424 0.404
0.577 0.467
0.061 0.149
0.611 /
0.685 0,576
FCN
0.700/ 0.306
0.624/ 0.376
0.546/ 0.207
0.684/ 0.450
0.595 / 167
0.515 2.515/02222
0.190 1000000.14 141
0.675 / 0,396
0.693 0.340
Gaussian Blur
0.706 / 0.439
0.534/ / 0,379
0,593/ 0.415
0,586 0.508
0.485 / 0.228
0.582 / 0.297
0.040 / -0.085
0.663 / 0.499
0.695 0,466
Rotate
0.713 / 0.484
0.674/ / 0.546
0.557 / 0.420
0.731 / 0.710
0.506/00228
0.582 / 0.406
0.274 0.228
0.734 / 0,594
0.700 0.470
Median Blur
0.722 0,535
0.548 / 0.480
0.549/0.425
0.598 0.570
0.460/0.302
0.572 / 0.365
0.174/ -0.036
0.668 / 0.580
0.652/ 0.421
Flip
0.743 0.424
0.678 0.546
0
0.743 0.607
0.550/0.312
0,570 0.330
0.220 -0.086
0.731 / 0.564
0.776
Random Noise
0.745/ 0.580
0.683/ 0.592
0.572
0.429
0.732/0 654
0.611/0.444
0.574 0.432
0.238 0.172
0.707 0.600
0.712 0.531
DifUPEG
0.748/ 0.664
0.673
/ 0.608
0,586 / 0.467
0.747
0.712
0.583 / 0.490
0.593 / 0.473
0.307 0.186
0.738 / 0.649
0.734 0.608
MPRNet
0.755/ 0.619
0.665 / 0.600
0,589/ / 0.656
0.653 / 0.621
0,574 0074//0.55 0.455
0.569 / 0.394
0.157 0.089
0.751 / 0.667
0.712 0.525
23

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
(0) 36
-Droon
Dscre: %
Rycore 49
Defurse
Aip
ReRANN
We Dafenas
DIEGO
Gaussian Blur
Resizo
2PCD-LPIPS.2
APGD-58M-4
DITUFEO
MPRNN:
Rotata
APCD-LPIPB4
APOD-S3M-5
DifFure
Median
Unsharp
UPIS
Dsfense
Figure 11. Comparison of Dxcore and Disure (left) and Racore/ Ricore (right) for Purification-hasse and Adversarial Training defenses in adaptive scenario. Results averaged KonIQ-10k, KADID and AGIQA-3K datasets.
Table 13. Per-metric comparison of purification defenses in adaptive case (D Scavee Racore). Evaluated metrics averaged across all images and attacks on KonIQ, KADID and NIPS datasets.
Defense
Lincarity
KonCept
PAQ2PIQ
MANIQA
Mcta-IQA
SPAQ
FPR
TOPIQ(NR)
CLIP-IQA+
W/o Defense
63.66 63.66/0.31
41.80/ 0.47
41.61 /0.49
25.61 1661/0.3
42.62 0.39
60.63 / 0.46
281.18 -0.28
21.57 0.70
21.91 / 0.68
Unsharp
71.57
60.29 0.23
0..26
36.70/0.41
48.48
84.57 0.20
376.46/-0.45
26.08 0.556
24.13/0.60
Resize
66.23 0.225
21.52/ 0.67
26.15/0.64
19.13/0668
45.11
223.38/-0..
2..74/0.64
21.28/0.65
Flip
62.30/0.31
45.41 0.388
42.11/0.44
28.26/0.57
40.12/0.43
66.02 0.36
264.75//0.15
22.44/0.67
22.63/0.65
FCN
61.60/0.29
44.00
43.61/0.41
20.67/0.54
43.05
0.34
256.72/-0.27
23.11/0.63
23.31/0.60
DISCO
56.08 0..40
30.57 0.53
33.90/0.57
2..56/0663
36.75
$7.28 0.43
1.1.53/0.10
20.72/0.72 0..73
Median Blur
49.19/0.36
28.51 00.51
34.85/0.49
2..77
33.52/0.50
0..64/0.42
174.94/-0.12
19.21
22.31 /..63
MPRNet
43.85/0.42
27.66 0.54
32.94/0.52
21.32/0.61
33.54/0.52
41.87 0.449
151.72/-0.04
16.67 0.73
20.77 0.66
Random Noise
42.43/0.46
34.61 /0.52
37.16/0.51
/0.6
30.74
/
165.95 10.10
0.766
18.95 / 0.744
Gaussian Blur
3..2
30.48/0.53
20.24/0.64
/0.53
36.96 0.55
149.81 /0012
14.57 0.84
21.51 / 0.65
Bilinear Upscale
31.03
20.91 0.68
22.89/0.68
17.66
22.14/0.71
135.84/0.03 0..90
0.71
/
Real-ESRGAN
21.71 /0.73
66.37 / 0.08
39.43/0444
36.74/0.32 0.32
19.19
22.89 / 0.75
67.99 /0.20
13.21 / 0.89
17.36 0.72
Rotate
21.39 0.78
23.80 0.64
13.23/0.98
9.10 1.01
/..95
26.92 0.70
12.24 0.98
10.20 1.03
10.42 / 1.02
DifjPEG
20.07 0.74
22.80 / 0.64
2..70
18.04/0.77
17.54 0..79
25.02 / 0.70
84.61 /0.15
10.94 0.95
15.59 / 00.81
DiffPure
19.63 / 0.79
17.53 0.76
/9.73
14.65 / 0.82
14.74 0.90
21.66 0.79
66.03 0.27
13.12 0.87
13.17 0.88
Crop
16.87 / 0.89
31.32/0.46 0.45
20.08 / /0.11
15.68 / 0.77
19.38 0.778
17.24 0.85
8.17 1.14
10.96/ 0.96
24

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
Table 14. Comparison of purification defenses by attack type. Evaluated metrics are averaged across all images, attacks and quality metries for nonadantive use case on KonIQ and KADID datasets.
Defense
Restricted WB
Unrestricted WB
Black-Box SROCC.a.
Racore
SROCCaa
PSNR1 f
Racore
SROCCa. f
PSNR 1
Rucors
PSNR1 1
W/o Defense
0.36+#066
0.387=0.29
42.1225.69
2.06
0.535=0.26
52.31=32.77
1.1900.66
0.590#0.31
38.87=7.00
Unsharp
0.30=0.47
0.329=0.28
30.64:2.20
0.89=0.43
0.525=0.25
26.3947.31
0.87=0.37
0.578+0.27
28.37=3.96
Real-ESRGAN
0.62+0.34
0,474:0.25 23
31.62=1.77
0.70=0.29
0.49420.24
25.64:6.49
0.75=0.28
0.658+0.11
28.31+3.86
FCN
0.62+0.44
0,455+0.21
20,49+0,33
1. 1.07+0.32
0.532=0.25
18.83+1.87
1.21+0.28
0.632+0.22
21.20+0,46
Color Quantization
0.63:0.49
0,500 25
33.62+1.66
1 1,08+0.40
0.53320.25
27.30
1.21
0.632+0.24
32.38+2.45
Bilinear Upscale
0.68#0.33
0.389=0.27
33.69=1.99 94
0.92=0.25
0,493=0024 24
27.30
1.01=0.24
0.555+0.24 26
29. 29.92+4.70
Caussian Blur
0.78+0.24
0.374=0.26
32.70=1.78
0.85#0.22 22
0,443=0.24
26.53=7.37
0.87=0.27
0.522+0.20 2.2
29.00=4..75
DiffPure
0.81+00.25
0.514+0.19
29.46+1.32
0.78+0.21
0. 0.424=0.22
24.39+5.55
0,78 24
0.529*00.2 22
26. 26.26*+4.20 20
Random Noise
0.82+0.27
0.533:1.224
26.00:0.69
0.83 24
0.52322.23
22.44:14.03
0.9920.32
0.633+0.19 19
25. 25.74
Crop
0.8310.30
0.476:1.22
11.83 13
1 1.033
0.54020.24
11.44:0.50
1.1920.34
0.58810,23
11.04
Median Blur
0.84#0.28
0.385#0.25
31.70=1.92 92
0.99#0.31
0.455=0.23
26.07=7.12
1.03=0.28
0.503+0.24
27. 27.83+5.42
JPEG
0.91=0..2
0.612+0.20
33.23+1.72
1.14+0.37
0.572+0.23
26.99+7.34
1.23+0.34
0.655+0.23
30.02+3.37
DiMIPEG
0.91=0.42
0.614+0.20
33.29#1.72
0.14
0.575=0.23
27. 36
1.22=0.33
0.660#0.22
30.03+3.39
MPRNet
0.9410.28
0,557:0.16
32.3621.61
0.9810.26
0.511=0.20
26.14:77.20
0.9910.26
0.62910.16 18
27. 27.9314.92
Resize
0.9420.35
0,545:0.21
32.65=1.78
1.08=0.33
0.555=0.22
26.55=7.39
1.23+0.27
0.639#0.21
28. 28.9624.73
Rotate
1.8=00.00
0.486:1.20
11.41=0.38
1.11
0.513=0.23
11.09=0.70
1.1800.21
0.558+0.24
10.87=1.26
Flip
1.11+0.28
0.564#0.23
10.85=0.21
1.29+0.27
0.553+0.27
10.53+0.41
1.43+0.25
0.655#0.21
10.21+0.52
DISCO
1.20+0.22
0.59410.22
29.62:1.229
1.0710.27
0.54420.22
24.2115.35
1.20s0.25
0.65110.20
26.6413.57
Table 15. Wilcoxon tests in nonadaptive use case of purification defenses on NIPS dataset for SSTM scores.
Defease
Bilincar Upaaie
Unabarr
Coup
JP5G
Quettiociun
DillPur
Flip
FCN
Rial- ESRDAN
DISCO
DimPEr
7.125%
7.41%
104L99%
190,092
63.25%
13.35%
7.64%
02:1156
55 51%5%
109.805
100,15%
0/00%
17.05%
7.38%
5133
1,59X
Bilncar puots
52.71%
<4 1616
12.04%
150 05%
74.21%
53.38%
43 43.995
62.1150
12.05%
105006L
100055
0.00%
$1.3%
1.415
0005
4.8%
Unduep
39-9190
10.54%
16.24%
120.09
100105
54.9%
12.96%
19.33%
45.58%
50.8%
06.85%
100/55
24.23%
1.45%
5.15%
6,00%
Resize
19:6675
0.007
35104%
35,55%
85.8%
63.53%
35.055
21.45%
49.29%
53.96%
18.955
0.074
45.30%
0.57%
000%
3,992
Rouse
00004
0.007
11075
0.00%
00025
0000%5
0
0.056
60 3.5
0.004
01875
0/004
000%
0,00%
Cro
0/065
0.00%
010755
0.002
M6.125
010075
0.0K65
0000
0/455
10000%
0/00%
010775
0/00%
0005
0,00%
Median tiee
0.065
0.00%
21 945
0.007
10010055
100,000
0.0K6E
B.0%
7.24%
0.465
105000
100055
0/00%
0.0075
0.005
0005
0.00%
0.0652
6.21%
96.185
7.125
1001055
100,005
62.65%
7.60%
52 1955
10000F
100055
0.00%
11.45%
0.655
199%
1.713
Gasedan Blue
18,005
0.00%
$7.89%
K.33%
100.055
100.007
65.815
27.64%
55.84%
$3.33%
100,00
100
0.00%
45.01%
0.57%
0.05
4
Color Quantization
0005
0.001
21.685
0.002
10010555
100.0072
14.25%
0
16 165255
100
0.00%
010775
0.00%
000%
0000
Difthare
0.5756
0.00%
235355
0.007
1001055
100.0072
17.60%
0.8%5
17.00%
100
0/00%
1.71%
0/00%
000%
1.1%%
Rondem Novez
0.00%
01005
0.002
98.455
0.00%
0.006%
0.0%
0.00
0.0055
10000%
0.00%
0.00%5
0.005
00%
000
Pli
0.005
0.00%
0.075
0.00%
0.00%
0.05:
0.00%
0.005
8.0%
0.00%
0.605
0L00%
0,00%
0.00%
0.005
00%
600
MPRNNi
$2.7150
15.34%
42.245
41.31%
1000005
100.00%
63.53%
54. 185
45875
61.545
52 16.9
100,00%
100
$4.15%
285%
5413
FCN
E.145
4.58%
20 29.05%
4.847
100.005
100.00
51.29%
1.43%
5.995
18.
42 45%
106,00
100
0.85%
2.38
142%
CEST
Real-ERGGAN
72.08%
9.97%
47.85%
51.0%%
100.005
100.003
74.93%
61.54%
90.11
64.77
$1. 2015
106,00%
100
1.99%
53.85%
114%
4.27%
DISCI
31.5%
m0%
67 07.055
61.4%%
100 000
100.000
81 355
72.45 5%
22.955 965
96.73%
106,006
10000%
45.65%
67.34%
47%7%%
18.52%
Wie Delemo:
90.85%
SK00%
106,00%
$1.45%
100
009
100,00%
97.44%
90.0%%
91.15%
95.44%
92 885
106,000
10000%
85.19%
81.35%
$6.0%%
63.1%
Table 16. Wilcoxon tests in adaptive use case of purification-bast based and Adversarial Training defenses on AGIQA datasct and L Linearity. Koncept IQA metries for Recore values.
Billnar
Raal-
Cansria
Randies
Win
APCDD-
APDE
ABGID-
APGD-
APCD-
APGD-
PCN
11.427%
22 30.50%
15,6797
39.5%
15.15%
23.22%
4.77%
16.37%
14.3%%
47.32%
$7.30%
6,25%
10-29
20.55%
2180
4.37%
+1,07%
DUKPE
12:425
23JAFI
Media tike
5755
13.625
50.4555
4135%
$1.905
16.7%
455395
85 151955
$.355
11.99
479955
1175
14
15 637g
45795
11.25%
1547%
$1.395
4747%%
31175
17 2391%
10465
18.9%%
01,775
43795
13400
41477%
453005
11.00%
68
11.8%
5.303
9-905
47.4%
0.17%
15.1%
11905
41 11:455
11.
17.0%
40005
13.03
11.40%
11.77
Dillbar
75.1790
72.925
13
06.775
7017%
04355
03.3%
13/07
51.2%
22524
31.50%
12528
11.8%
Cre
12.02%
471025
37.307
13.67%
43.1%%
17.8%
7235%
50.15%
12.0%
11.05
68.72%
Baliis
$0.2040
01.45
39
21.355
41.1%%
02.95
0657%
19.88%
11.05
$3.00
62:000
81.2%
200%
Guaaca Blu
$3.2591
25.015
06.15%
91.8%
11.25
4315%
354255
45.17%
5333%
41.3%
1300
43555
15.1V
Rarbe
17.055
17.
56174
40075
15.07%
1597%
15.1%%
16675
635
Rrams
56.54%
05.55
4155
00055
$1.05
63.955
Racdon Nore
4335%
5111%
56.7%
11.7H
11405
41.5%
20176
13997
0 00000
(1TL
43.7795
16.7%%
35.75
03.00
36075
1875
2353
1657%
1477
6778
1148
11.28
4
1508
$3.50
5055
8
$1.50
34.11%
41.87%
47.5%
4006
233
1147%
1395%
411%
14.8%
91.7%
3548%
80
81.8%
5.55%
APID-SSIMS
10.50%
41.67%
11992
12440
16.7%
14.5%
00425
$1.50
1258
453055
25

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
Table 17. Wilcoxon tests in adaptive use case of purification- and Adversarial Training defenses on NIPS dataset and Linearity,
Koncept IQA metrics for PSN R values.
TTN
MPPMN
DSSCO
Upecae
Fig
DUTPEIG
X3RGAN
L2785-2
12055-4
L21P5-4
533662
52335-4
$584-
$7
$1.55
194
375
0.975
41.7%5
189
184 (8:
922
6.65%
0.8%%
4.1%%
316%
6.295
3.8
3.395
37525
330%
120 575
1065
22077
2555
037-
0065
12 10165
10 575
120.00 23.955
0.005
4.17%
2165
23655
4177
218%
2.555
108.855
380.855
22.205
731355
10165
100
33.255
130085
14.0%%
20.355
15807
233875
1607
2/0%%
0.0%%
0.975
0.64
616-%
316%
20055
D/055
1/0%
5/0%
5005
0.00%
0.63
0.035
0.75
0.63
Da0S
5005
6465
0.80
000%
0007
5465
0275
D206
53.355
3.0%
120
133,05
51.0%
91.7%
109
51525
123,055
030%
4.17%
2165
6.2%
1185
05332
$3.074
500%
67.27
130
4.5%5
0.975
635%
55.35
130.005
0
11T%
5305
0.75
DAMYS
500%
3.0%%
500%
0.60%
0.65%
0.005
0.883
010%
0.805
0075
036%
43335%
120.375
0955
0.97
0.864
1.2%
123.955
0 0.585
118%
2807
037
5/0%
370%
6:67%
2.8%
0995
0.8
0LATS
190
015
0 0.004
3163
266
1000
218
097
Aanken Nobe
100.005
101075
191975
10197
11.8%
130
120 055
190375
130.75
331.35
17975
01:22
100 MES
159 165
108
100 88
130.955
0 0.00
33425
62555
33 136
400
2100
11327
4.1%
41A
208
05533
01925
17365
$33055
61554
$1555
61678
190355
F135%
8.58%
10000
11.775
51350
635
18 14.05
13550
01855
$1555
61546
11:52
43355
33976
19335
13255
51.0%
11225
5:.00
5350
115
2000%
07.033
$94933
13.55
67.63%
136.9%
41.17%
1000555
0,975
91.67%
01.65
61.05%
11.55
35485
33333
23.87%
Table 18. Comparison of defenses by defense type. Evaluated metrics are averaged across all images, attacks and quality metries for nonadaptiveaadupivv use cases on KonIQ and KADID datasets.
Defense
D Paoore (P) 4
Dacore +
R 1
Racove t
S SROCCadu 1
SROCC 1
PSN R 1
Filtering
21.13 27.17
20.39
22.34
0.63/ / 0.49
0.72/ 0.68
0.499 / 00545
0.631 / 0.628
19.53 20. 14
Compression
21.86 / 15.60
18.20 / 11.29
0.65 / 0.75
0.81 0.99
0.561 / 0.635
0.687 / 0.697
19.96 / 20.46
Spatial Transforms
21.20 / 29.95
20.27 26.53
0.64/ 0.46
0.69 0.62
0.578 0.508
0.684/ / 0.604
19.62 16.57
Denoising
17.26 19.90
26.05 /5.995
0.80/ 0.71
0.59 / 0.60
0.533/ / 0.569
0,664 / 0.672
19.66 / 20.09
With Randonness
14.93 16.71
19.17
0.83 0.84
0,77 0.69
0.523 0.528
0,634 / 0.596
18.81 14.81
Adv. Defenses
8.15 / 86.86
23.14/22.35
1.11/ / 0.50
0.63 0.69
0.474 / 0.538
0,583 / 0.626
19.09 / 19.16
Adv. Training
2/22.41
//2.41
0.66
/0.68
0.552
0.6667
Table 19, Comparison of purification defenses by dataset. Evaluated metrics averaged across all images, attacks and quality metries for nonadaptive/adaptive use cases.
KoaIQAIK
KADIDUK
AGIQQA-3K
NIPS
Decor
Rocer 1
SHOCCaae
Drcore
Racore
SROCCate
Dicore 4
SHOCC
D
Racore
Wio Defense
51,321 -
0.57
0
45,901
0,74/
0.487/
35.89/
0.57/
0,586 /
47.73 /
0
Unshurp
19.09/92.16
70.21
0.767 2267/0.766
41.96/ 84.67
0.62
0,462
/0423
38.49 102.72
0.55 /0.16
0.625 /0.396
45.11 /85.18
0.50/0.28
Color Quamization
24.43/
0.83
0,760/
24,47/
0
0.475/
24.67/
0.86
0.565/ /
25.36/ /
0.85
Bilinenr Upscale
19.66455.22
082/0055
0.679 /0.587
18.42/ 36.11
0.84 0.61
0,512/ (1.420
12.59 0399/432 /4.36
L02/0.50
0,542 /0.42
2033 / 26.88
0.68/ 0.668
RCN
73.93/3.55
0.84/0.31
0
21.73/
0.465 0.33/
29/ 73.13
0.541 0..2/..2
18.38 /5.78
0.91 /0.45
Gaussian Blur
16.50
0.00
0.607/ 0.615
17.70/ 40.12
0.53
0,512 0.661
16.45 50.24
0.90/0.42
0.568 /0.
/5.83/36.36
0.82/
Median Blur
15.17151.43
/0.41
0.668 00.778
15,001 68.65
0.90/ 0.45
0.440 / 00.402
15.98 /57.85
0.95 /0.36
0,571 /0.512
11.82 / 444.05
0.98/ 0.66
Real-E
23.26:33.41
0.59/
0.682
21.18/ 33.15
0.73
0.484/ 0.385
18.49 30.448
0.61/0.57
/0457
25.38 /35.13
0.61/
JPEG
15:34/
1.02
0.267/
12.45
1.08 /
0.530/
12.97
1.08 --
0.593 /
10:35
L.0
DENIPFG
13.27/28.33
1.02/0.65
0.770/0.765
12.43/ 28.46
1.09/0.6
0:532 00.554
12:96/ 34.54
1 088 70.57
0.593 /0.579
10.33 / 22,32
0.73
Resize
12.35/66.419
1.01/0.35
0.536/ 1.07
13:03
L00/0.31
0.561
744.73
0.55
MPKNet
11.35 46.26
/..4
0.697/ 0.699
15.64/ 42.56
0.91 /0.51
0,569 0.499
14.08 55..9
1.00/ 00.41
0.504 0.5001
12. 44.16
0.98/ 0.98/0.33. 0.52
Crop
14.38/ 19.30
0.93/ 18:72
0,740
00.575
13.56/ 18.20
1.02. 0.77
0.452 06622/0.33 0.310
17:05 / 21.35
0..6/
0.576 /0.409
15.06 / 14.44
0.97/ 00.90
Random Noise
16.4:44..75
/0.46
0.727 0.781
14.73/ 46.72
0,473/
52.67
16.26 /66.3/
0.84/
Flip
64
L/6/00.31
0.762 /0.774
7.41 67
1.28 10.38
0.477 / 00451
9.46 81.05
10.2/
0.55 /
7.64/
/.55
Rotale
10.29:17.72
1.0% 0.84
0.696 00744
9.97 15.34
1,13/0.0 0.91
0:452/ 0.647
12.08
/ 22.70
L.05 0.77
0,5g9 /0.560
/ 13.97
L.10 0.94
DISCO
7.77
1 00.8
0.720 0.735
9 52.58
L12
0,522/ 0.459
66.65
L.05
0.543 00.542
/41.30
L13 0.61
DifPeree
17:53/ 23.15
0.76 006/00:3 00.73
/.550/0.555
19.20/ 22.54
0.77 /.72
0,522 / (0.477
17.57 / 21.91
0.83 0..76
0.463 0.446
12:35 / 20.82
0.89 00.8
26

---

Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metries
Table 20. Comparison of SROCC and PLCC scores averaged across KonlQ, KADID and AGIQA-3K datasets for purification-bas based and adversarial training defenses.
Defense
Common
Non-adaptive case
Adaptive case
SROCC
PLCCetear t
PLCCado t SROCCado t
SROCC
PLCCadv 1
W/o Defense
0.617=0.01
0.648+0.02
0.484+0.13
0.46410.12
0.402+1.00
0.432#0.08
Unsharp
0.604+0.02
0.631
0.452+0.14
0.43340.14
0.34540.12
0.366+0.12
Color Quantization
0.59440.02
0.616#0.02
0.57420.09
0.54240.09
FCN
0.580=0.02
0.591=0.01
0.522+0.10
0.498+0.10 10
0.28240.13
0.29910.12
Bilinear Upscale
0.577=0.02
0.614+0.03
0.499#0.12
0.468+0.11
0.34740.09
0.376+0.09
Gaussian Blur
0.543+0.03
0.572+0.03
0.450=0.13
0.426#0.12
0.360+0.11
0.390+0.10
Median Blur
0.546#0.02
0.579#0.02
0.458#0.11
0.430+0. 11
0.41240.11
0.444
JPEG
0.630#0.02
0.655+0.02
0.637+0.05
0.610#0.04
DiffJPEG
0.632+0.02
0.658+0.02
0.639+0.04
0.613#0.04
0.548+0.07
0.584+0.06
MPRNet
0.560=0.03
0.595+0.04
0.570=0.06
0.533+0.05
0.483+0.08
0.513+0.08
Crop
0.589=0.01
0.624+0.02
0.553#0.08
0.51840.07
0.381
0.389+0.09
Random Noise
0.566#0.03
0.593+0.04
0.573+0.05
0.547
0.501
0.534
Resize
0.606#0.02
0.640+0.03
0.588#0.07
0.561
0.335+0.10
0.370 I0
Real-ESRGAN
0.570=0.05
0.585+0.04
0.537=0.08
0.523+0.09
0.435+0.08
0.467
Flip
0.598=0.01
0.631+0.02
0.597#0.05
0.56440.05
0.403+0.12
0.423
Rotate
0.566#0.01
0.595+0.02
0.530#0.07
0..01
0,45940.10
0.488#0.09
DISCO
0.595#0.02
0.62640.03
0.61740.05
0.58940.03
0.466+0.05
0.494+0.05
DiffPure
0.512+0.04
0.547+0.04
0.531
0.497+0.06
0.472+0.04
0.511
APGD-LPPIP--2
0.642+0.00
0.510=0.09
0.449#0.12
APGD-LPIPS-4
0.669+0.00
0.485+0..4 14
0.475+0.16
APGD-LPIPS-8
0.663#0.00
0.461
0.420#0.12
APGD-SSIM-2
0.620#0.00
0.586+0.02
0.504#0.07
APGD-SSIM-4
0.670=0.00
0.445+0.14
0.428+0.17
APGD-SSIM-8
0.675+0.00
0.50440.10
0.50940.12
Table 21. Comparison guarantees computational complexity of certified defenses. For classification- methods (C) we measured certified radius and number of abstains, for regression- based methods (R) certified relative delta.
Cert.R 1 for C Cert.RD 1 for
Defense
Abst. 4.%
Time. sec
Random. Smoothing (RS) (C)
0.20 / 0.16410.04
4.68.12.85 10.61
36.70+18.66
Denoised RS (C)
0.19+0.03 / 0..00...
330+t3.42/7.43-443
44.43++2.99
Diffusion DRS (C)
0.18+0.02 / 0.17=0.03
10.55
8.33
71.85118.94
DensePure (C)
0.17+00.22 0.16+0.02
7.91 /..219699....8..
116.37-19.04
Median Smoothing (MS) (R)
1.47.10.42 2.25
26.08=117.05
Denoised MS (R)
1.41+:0.4 / 1.8840.49
29.05+
27


Reviews:
[{"review_id": "lHpWpLPhhZ", "reviewer": "Reviewer_H2DF", "summary": "Image Quality Assessment (IQA) mostly uses DNNs to calculate the score, leaving space for attackers to perturb the image to manipulate the score for commercial advantage in ranking. Compared to (Antsiferova et al., 2024) that benchmark attacks to IQA, this paper benchmarks defenses in this task. It considers 17 purification (pre-processing) defenses, 2 adversarial training (with variations), and 6 certified defenses. The evaluation is extensively performed on 4 image datasets, 9 IQA models, and 14 attacks (with adaptive variations). Authors publish the benchmark for the community to assess upcoming defenses/attacks.\n\n## update after rebuttal\nThanks the authors for providing the rebuttal in a limited time. I think it is good to dedicate a paragraph to organize existing insights as promised, but overall, more insights from this work are encouraged. I did not question the novelty against RobustBench, but more discussions on the findings on this benchmark and RobustBench should be presented. I decided to keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Claims are clear.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No theoretical claims made.", "experimental_designs_or_analyses": "Yes, see Other Strengths And Weaknesses.", "relation_to_broader_scientific_literature": "Authors publish the benchmark for the community to assess upcoming IQA defenses/attacks.", "essential_references_not_discussed": "Not found by me.", "other_strengths_and_weaknesses": "Strengths\n1. Attacks to IQA models are well motivated, contributing to the value and timeliness of a defense benchmark. To achieve a high ranking in a search engine, an image releaser has the motivation to perturb the image to bypass the DNN-based quality ranking system, so that a specific image receives unmatched attention.\n2. I am impressed by the comprehensiveness of experiments. For each of the assesed defense, all common attacks are tested. Notably, adaptive attacks that backpropagate attack gradients through the defense are also considered. The authors also study extensively on how the attack/defense hyperparameters affect the results.\n3. Various metrics are adopted. Robustness scores reflect how well a defense can restore the IQA scores of adversarial images to match their original values. Quality scores measure the perceptual similarity between purified images and their original images. Performance scores assess an IQA metric’s performance in the presence of adversarial defense.\n\nWeaknesses\n1. The takeaways for IQA designers are not clear enough. The benchmark is meant to provide guidance for defenders on how to secure their system. However, only limited discussions are put in Section 5. As a researcher not working in this field, I struggle to learn useful insights from a lot of numbers presented.\n2. I am not sure how this work links to previous benchmarks on securing image classifiers, e.g., RobustBench. Both dealing with images, a lot of attacks/defenses are using the same technique. Are there similar conclusions about a defense method? What is new in IQA? There seems to be limited sentences for that, but I think a concentrated paragraph would be helpful. This gives hints on whether it is necessary to develop benchmarks for future new DNN-based image tasks.\n3. I believe the manuscript would benefit from presentation improvement in terms of a clear prioritization.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "As a late reviewer, I do not expect interactions with the authors. Feel free to deprioritize my opinions if rebuttal is important in AC's judgement.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YBuro5vdai", "reviewer": "Reviewer_BpWG", "summary": "The paper proposes a benchmark for defending neural-network based image quality assesment (IQA) against adversarial attacks. The paper makes an extensive study with numerous datasets, IQA models and adversarial attacks of different types and discusses the evaluation results.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and detailed feedback. We truly value your questions and your time reviewing our paper. We address your concerns below:\n1. Our contributions include a novel methodology for comparing defenses for the IQA tasks, addressing a critical gap in the field, an extensive subjective study with 60,000+ responses, and an in-depth analysis of the results. Novelty of methodology includes: sampling strategy for dataset, wide range of attacks of various types (including WB and BB, restricted and unrestricted, dadptive and non-adaptive attacks), comparing different types of defenses with each other, aligning attacks parameters by attack strenth, varying defense parameters, new metrics for evaluation and subjective comparison.\nNotably, our work directly aligns with the main topics from the call for papers for ICML 2025 — specifically, Evaluation (Methodology) and Trustworthy Machine Learning (Robustness, Safety). By targeting an underexplored area with clear real-world impact, our benchmark lays the groundwork for future contributions. To our knowledge, no prior work has systematically examined defenses in this context, underscoring the novelty and importance of our approach.\n2. The manipulation of IQA metrics is a significant concern that extends beyond image processing competitions to numerous critical applications in the ML community. IQA metrics are foundational in detecting copyrighted content, ranking results in internet search engines (e.g., Bing, as noted in Section 1), assessing medical imaging quality for diagnostics [1,2], enhancing face recognition systems[3], and nearly all preprocessing techniques for images and videos. \nThe significance of attacking IQA metrics extends beyond manipulating benchmarks and is already recognized by research and industry areas. Existing papers address issues with IQA robustness because IQA is foundational in detecting errors in medical imaging, ranking the results in search engines, etc. Exploring and enhancing the robustness of IQA modes is an active area of research[4,5,6]. Moreover, after the paper[8] about the adversarial vulnerability of VMAF metric and implementation of this attack in Google's libaom video codec, VMAF developers from Netflix had to release a more robust version VMAF NEG[7], highlighting the importance of robust IQA metrics for the industry. We want to establish a standard to evaluate emerging methods and boost the development in this area by publishing our benchmark.\n\nIf you have no further concerns we would be sincerely grateful if you could consider raising the rating of our submission.\n\n[1] Yuan S. et al. “A Deep-Learning-Based Label-free No-Reference Image Quality\nAssessment Metric: Application in Sodium MRI Denoising”\n\n[2] Dong X., Fu L., Liu Q. “No-reference image quality assessment for confocal endoscopy images with perceptual local descriptor,” Journal of Biomedical Optics, 2022\n\n[3] Terhorst P. et al. “SER-FIQ: Unsupervised estimation of face image quality based on stochastic embedding robustness,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020\n\n[4] Ghazanfari S. et al. “R-LPIPS: An adversarially robust perceptual similarity metric”\n\n[5] Kettunen M. et al. “E-lpips: robust perceptual image similarity via random transformation ensembles”\n\n[6] Chistyakova A. et al. “Increasing the Robustness of Image Quality Assessment Models Through Adversarial Training”\n\n[7] Netflix blogpost: https://netflixtechblog.com/toward-a-better-quality-metric-for-the-video-community-7ed94e752a30\n\n[8] Zvezdakova A. et al. “Hacking VMAF with Video Color and Contrast Distortion”", "claims_and_evidence": "I think, the claim of proposing a comprehensive benchmark for evaluating defences against adversarial attacks on image quality assesment is supported in the paper. Up to my knowledge, there were no such benchmarks up to now.", "methods_and_evaluation_criteria": "I think, the datasets, IQA models, adversarial attacks and evaluation metrics make sense for the problem.", "theoretical_claims": "I have not seen any theoretical claims or proofs that need to be checked.", "experimental_designs_or_analyses": "The validity of experimental designs appears to be adequate, but it was not carefully checked.", "relation_to_broader_scientific_literature": "The paper relies on existing adversarial attacks and previously proposed IQA methods.", "essential_references_not_discussed": "I am not aware of any essential references that were not discussed.", "other_strengths_and_weaknesses": "Strengths\n\n1. An extensive defense evaluation including adaptive and non-adaptive adversarial attacks. \n2. A large-scale subjective study with many participants to provide additional evaluation. \n3. Since the proposed benchmark is claimed to be open for submissions, it may facilitate the development of the field. \n\nWeaknesses\n\n1. I appreciate the effort made in this paper to provide a benchmark for the IQA defences. The main concern is whether the methodological novelty of this paper is sufficient for the main track of the conference where this paper was submitted. The paper proposes some novel metrics for the considered task (e. g. (8), (9)) but in general the novelty seems to be rather limited. I think the proposed benchmark can be valuable for the community. But in my opinion after some adaption and formulating a clear position on the current state of the IQA defences based on the performed studies, the paper would much better fit into the Positon Paper Track of the conference. \n\n2. Apart from the novelty concerns formulated above, I am not fully convinced regarding the significance of the IQA metric defences in general. The paper makes an example based on the image processing and compression competitions where the competitors might exploit the metrics to land higher on the leaderboard. I am not sure whether it is a significant concern for the ML community which requires developing specific defenses for image quality assessment. Thus, the significance of the proposed benchmark is also questionable", "other_comments_or_suggestions": "$-$", "questions_for_authors": "No further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FOsfTnTrIt", "reviewer": "Reviewer_ZZxf", "summary": "The manuscript presents a comprehensive benchmark on defenses against adversarial attacks targeting neural network-based Image Quality Assessment (IQA) metrics. It evaluates 30 defense strategies across three categories (purification, adversarial training, and certified methods) against 14 adversarial attacks. The study covers adaptive and non-adaptive attacks on nine no-reference IQA metrics, with extensive experimentation on multiple datasets. The paper introduces a benchmark, a novel dataset of adversarial images, and an online leaderboard, aiming to guide future research in robust IQA metric development.\n\n## update after rebuttal\nThe positive judgment, also shared by the other reviewers, on this manuscript is confirmed having seen the responses to my comments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the valuable comments and questions. We appreciate the recognition of our study and analysis and will address the questions below:\n1. We employed three metrics for attack strength estimation depending on the attack type. For $L_{\\infty}$, we chose $\\frac{2}{255}$, $\\frac{4}{255}$, and $\\frac{8}{255}$ as the most common thresholds in similar studies [1,2] and aligned them with potential real-world cases where imperceptibility is critical.\nFor perceptual metrics-based attacks, we used SSIM and PSNR to quantify attack strength. To ensure alignment across different attacks, we computed average SSIM and PSNR values on a subset of 1000 images from KonIQ-10k corresponding to the selected $L_{\\infty}$ thresholds. For each selected $L_{\\infty}$ threshold, we calculate corresponding PSNR and SSIM values. For example, the average SSIM value for $L_{\\infty}=\\frac{2}{255}$ is approximately 0.9.\n\n[1] Croce, F., et al. \"RobustBench: a standardized adversarial robustness benchmark,\" in NeurIPS 2020, arxiv: abs/2010.09670.\n\n[2] Dong Y. et al. \"Benchmarking adversarial robustness on image classification,\" in proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, С. 321-331\n\n2. Adversarial training requires no additional cost computation during inference, so this type is suitable for real-time quality measurement scenarios, e.g., image quality screening in CCTV, or for frame-by-frame video quality assessment. Purification methods have a wide range of computational overhead from 6 ms (Real-ESRGAN) to $\\sim$140 ms (DISCO), with the slowest being diffusion-based methods ($\\sim$700 ms for DiffPure). However, they can still be applied to image hostings to measure the quality of new images. Certified methods are much slower (3 to 40 seconds per image). They are best suited for security-sensitive scenarios where robust guarantees outweigh computational costs ー for example, in benchmarks to prevent cheating.\n3. Combining purification and adversarial training could enhance robustness by leveraging their complementary strengths. Purification handles unseen perturbations, while adversarial training improves resilience to known attack patterns. This hybrid strategy may offer stronger robustness against adaptive attacks while minimizing computational costs. However, AT and purification methods decrease the correlation of IQA metrics with subjective quality, so a combined defense may yield a lower correlation.\nIn this work, we focus on existing methods to establish a baseline before exploring more complex, combined approaches in future work.\n4. We did not analyze the transferability of adversarial attacks across IQA metrics as the paper focuses on evaluating defense mechanisms. Some studies focused on attack methods did analyze transferability[1]. This is recognized as an important direction for future work on adversarial attacks to understand cross-model vulnerabilities.\n\n[1] A. Ghildyal, F. Liu, \"Attacking Perceptual Similarity Metrics,\" in Transactions on Machine Learning Research, 2023\n\n5. To ensure diversity and reliability, we conducted a large-scale crowd-sourced study using the Subjectify.us platform, which enabled us to gather opinions from a broad and diverse participant pool. This platform helps reduce sampling bias and expands the range of participants. There were also participants’ answers quality control measures implemented via verification questions. Furthermore, the platform ensures that each assessor can only participate once and that the total number of answers for each pair is at least 10. This process minimizes the impact of unreliable inputs and ensures the robustness of our subjective evaluations.", "claims_and_evidence": "- They argue that defenses such as compression-based methods are particularly effective against adversarial noise, and the experimental results (e.g., DiffJPEG outperforming other purification methods) substantiate this.\n- The manuscript asserts that transformer-based IQA models are naturally more robust than CNN-based ones; the reported robustness scores and attack performance trends align with this claim.\n- The claim that randomized smoothing methods provide theoretical guarantees is valid but could be expanded with more discussion on its practical applicability to IQA.", "methods_and_evaluation_criteria": "- The methodology is thorough, covering multiple attack strengths, defense configurations, and datasets.\n- The evaluation is rigorous, using a mix of objective robustness metrics (e.g., R_{score} and D_{score}) and perceptual quality measures (PSNR, SSIM, crowd-sourced MOS study).\n- Adaptive and non-adaptive attack scenarios are considered, adding realism to the evaluation.\n- The computational complexity analysis of defenses is a strength but could be extended with a discussion on real-time applicability in practical IQA settings.", "theoretical_claims": "- The authors correctly differentiate empirical and certified defense methods. However, the manuscript could benefit from a more detailed theoretical discussion on why certified methods are less practical for IQA, given their computational overhead.\n- The claim that task-specific adaptations are needed for diffusion-based defenses is reasonable and supported by empirical evidence.\n- Some theoretical claims regarding robustness improvements from adversarial training could be better justified.", "experimental_designs_or_analyses": "- The experimental design is robust, with a diverse dataset selection and comprehensive attack-defense comparisons.\n- The use of adaptive attacks strengthens the validity of the results.\n- The subjective evaluation study (60,000+ responses) adds significant value but lacks a detailed breakdown of the participant demographics and potential biases.\n- A minor concern is that some defenses (e.g., adversarial training) were only evaluated on select IQA metrics, which could impact generalizability.", "relation_to_broader_scientific_literature": "- The study fills an important gap in adversarial robustness research for IQA.\n- The comparison with adversarial robustness benchmarks in classification tasks is insightful.\n- It would be beneficial to discuss potential implications for other domains using perceptual metrics, such as medical imaging or deepfake detection.", "essential_references_not_discussed": "While the manuscript is well-referenced, some recent works on adversarial robustness in vision models beyond IQA (e.g., adversarial defenses for GAN-based image generation) could provide additional context.", "other_strengths_and_weaknesses": "Strenghts\n- First systematic study of defenses for adversarial IQA attacks.\n- Extensive experimentation across diverse datasets and metrics.\n- Well-defined benchmark with clear evaluation criteria.\n- Subjective quality assessment adds real-world validity.\n- Open-source dataset and leaderboard promote reproducibility.\n\nWeaknesses\n- Computational complexity of defenses is high, limiting real-world deployment discussion.\n- Some defenses were only tested on select IQA metrics.\n- More details needed on subjective study methodology.", "other_comments_or_suggestions": "- The manuscript could discuss whether findings extend to full-reference or reduced-reference IQA models.\n- The impact statement could be expanded to address ethical considerations of adversarial robustness in applications like image forensics or content moderation.", "questions_for_authors": "1. How were the attack strength levels chosen, and do they align with real-world adversarial scenarios?\n2. How do the authors envision practical deployment of the best-performing defenses given their computational costs?\n3. Would the inclusion of hybrid defense strategies (e.g., combining purification and adversarial training) improve robustness further?\n4. Have the authors considered analyzing transferability of adversarial attacks across different IQA metrics?\n5. What steps were taken to ensure diversity and reliability in the subjective study participants?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q9uASsC4dX", "reviewer": "Reviewer_Dhjc", "summary": "This paper addresses the vulnerability of neural-network-based Image Quality Assessment (IQA) metrics to adversarial attacks. The authors present the first comprehensive benchmark for evaluating defense mechanisms against such attacks. The study systematically evaluates 30 defense strategies—including purification, training-based, and certified methods—against 14 adversarial attacks in both adaptive and non-adaptive settings across 9 no-reference IQA metrics. The authors also conduct an extensive subjective study with over 60,000 responses to evaluate the perceptual quality of defended images. The benchmark aims to guide the development of more robust IQA defense methods and is open to submissions.\n\n## update after rebuttal\nThanks for the authors' reply. They addressed my questions. After referring to the opinions of other reviewers, I decide to keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions and thoughtful feedback. We will use suggestions to enhance the revised version of the paper. Your questions are answered below:\n1. The discrepancy between subjective and objective metrics is not uncommon in IQA tasks. While Real-ESRGAN may not achieve the best scores on metrics like PSNR or SSIM, its superior performance in subjective study indicates that it preserves natural image characteristics. In practical applications, these findings imply that defense methods should be evaluated using both objective and subjective measures. Ultimately, this balance is crucial for developing defenses that improve numerical performance and align with how humans perceive images.\n2. Transformer-based IQA metrics likely benefit from their global attention mechanisms, which enable them to capture long-range dependencies and contextual relationships. Also, pretraining on large datasets can contribute to their generalization capabilities and robustness. While transformer-based architectures offer robustness advantages, they also have increased computational costs. Future IQA research may explore hybrid approaches that combine the strengths of transformers and CNNs, rather than solely prioritizing one architecture based on robustness alone.\n3. Certified methods provide strong security guarantees but are extremely slow because of their design. To optimize them, one could leverage more efficient certification techniques balancing the trade-off between guarantee and number of samples, or incorporate lightweight architectural modifications. Additionally, hybrid approaches that combine certified defenses with empirical ones may offer a balance between robustness and efficiency. Our benchmark provides a foundation for assessing such trade-offs and guiding future improvements.\n4. According to Figure 3, the performance on the AI-generated AGIQA dataset is very similar to the performance on the natural KONIQ dataset for most of the defenses, so we disagree that there is a difference in the results. We suggest general ML practices, covering all possible cases during training, including both types of content, to help defenses to generalize better. However, our results show no significant difference in defense efficiency between natural and AI-generated content.\n\nWe address your concerns below:\n1. Statistical tests cannot prove two samples are identical; a p-value above the alpha level means we cannot reject the null hypothesis. In our case, the p-values combined with the small variance of means reported in Table 5, indicate a high level of consistency in our sampling methodology. Figure 6 supports this, showing that score distributions across samples are nearly identical, with closely aligned means. Together, these results demonstrate that any observed differences are statistically negligible and do not compromise the validity of our sampling approach.\n2. Our study compares two existing solutions for adversarial training of IQA metrics: in one of them, MOS values for adversarial images are penalized, and in another, original labels are used for training on clean data. We highlighted that in general, adversarial training methods that use ground-truth labels for adversarial images may yield decreased correlations, which is not a limitation of our procedure, but encourages future defenses to consider this issue\n3. Computation time grows dramatically with the number of source images, IQA models and attack methods. Thus, using only 10 source images, we need to apply each certified defense method 1260 times. Overall, we spent about 3000 GPU-hours on certified defenses and ~25000 GPU-hours on all calculations. \n4. There is a distinct difference in inference time across defense types. For adversarial training, there is no inference overhead since the fine-tuning is done during training. For purification defenses, most methods add only about 0.05–0.2 s per image. This overhead is generally manageable for real-time cases, e.g. quality screening for CCTV, video streaming. Certified defenses offer strong theoretical guarantees but are significantly slower than other methods. This trade-off makes them most suitable for highly sensitive applications where robustness is a necessity, e.g. in benchmarks. \n5. Our reported computation times in Tables 2, 3 enable developers to choose a suitable defense method based on their deployment constraints. In conclusion, for real-time practical deployment adversarial training and fast purification methods are the best fit.\n6. Practical applications usually seek to inflate the quality of content. For example, cheating in benchmarks that influence project investments, increasing bitrate after transcoding in target network, inflating results (e.g., Google’s libaom encoder with its --tune-vmaf option). Decreasing scores don't lead to these problems because most real-world systems are aiming to higher quality. Given these scenarios, we prioritized score increases in our approach.", "claims_and_evidence": "The authors make several claims that are generally well-supported by their evidence:\n\n1. The claim that neural-network-based IQA metrics are vulnerable to adversarial attacks is established through references to prior work and demonstrated in their experiments. The evidence provided is convincing.\n2. The claim that this is the first comprehensive benchmark for IQA defense evaluation seems accurate based on the literature review provided on page 2, where they note that existing comparisons focus primarily on object classification rather than IQA metrics.\n3. The effectiveness of various defense methods is substantiated through extensive experiments across multiple attack types, datasets, and defense strategies. The authors provide robust statistical analysis to support their conclusions.\n4. The authors' observation that compression-based defenses (DiffJPEG, JPEG) are particularly effective is supported by both objective metrics and subjective evaluation.\n5. The claim that different defense methods work better for different IQA model architectures (page 7-8) is well-supported by the analyses in Figure 4.\nA minor limitation is that while the authors emphasize the importance of their subjective study for evaluating perceptual quality, they don't fully integrate these findings into their main recommendations. Figure 5 (page 13) shows that Real-ESRGAN performs best in subjective evaluation, yet this finding isn't prominently emphasized in the conclusion.", "methods_and_evaluation_criteria": "The methodology is generally sound and appropriate for the problem:\n1. The authors' approach of evaluating across multiple datasets (Page 4, section 3.4) is commendable, as it ensures results are not dataset-specific.\n2. The attack parameter selection method (Page 14-15, section A.3.4) is thoughtfully designed, ensuring fair comparison across attack types by aligning them to target \"weak\", \"medium\", and \"strong\" perturbation levels.\n3. The evaluation metrics (Page 4-5, section 3.5) are well-chosen to capture different aspects of defense effectiveness (robustness, perceptual quality preservation, correlation with human judgment).\n4. The subjective study design (Page 12, section A.2) using the Bradley-Terry model for pairwise comparisons is appropriate for assessing perceptual quality.\n\nHowever, I found some methodological concerns:\n1. Page 13, Table 5 shows validation of the sampling methodology, but p-values of 0.1449 and 0.1958 suggest the differences between samples might be approaching significance (if using α=0.05). This deserves more discussion.\n2. In the evaluation of adversarial training (Page 15-16, section A.3.5), the authors acknowledge that using ground-truth labels from clean images is inaccurate for adversarial examples, yet they don't fully address how this limitation affects their conclusions about adversarial training methods.", "theoretical_claims": "The paper is primarily empirical rather than theoretical. The authors reference theoretical guarantees for certified defense methods (page 6, page 7) but do not develop new theoretical results of their own.\n\nThe authors correctly note on page 7 that \"Despite the questionable applicability of randomized smoothing-based defenses to IQA, they remain effective, as the certified radii are sufficiently high and the number of abstentions is relatively low.\" This is an important observation about the practical utility of theoretical guarantees in this context.", "experimental_designs_or_analyses": "The experimental design is comprehensive and thorough:\n1. The authors evaluate defenses across multiple dimensions: effectiveness against different attacks (Table 14), performance on different IQA models (Tables 12-13), and behavior on different datasets (Table 19).\n2. The statistical significance testing using Wilcoxon Signed Rank Test with Bonferroni correction (Page 18-19, section A.4) strengthens the validity of their comparisons.\n4. The analysis of defense performance across different attack strengths (Table 11) provides valuable insights about defense robustness.\n\nA few concerns about the experimental design:\n1. In page 5, section 3.6, the authors mention using 40 NVIDIA Tesla A100 GPUs, but don't specify how computational constraints influenced their experimental decisions, such as limiting certified defense evaluations to only 10 images.\n2. The authors don't fully explore potential trade-offs between defense effectiveness and inference time for real-time applications. While they report computation times (Tables 2 and 3), the implications for practical deployment aren't thoroughly discussed.", "relation_to_broader_scientific_literature": "The authors position their work well within the broader literature:\n1. They acknowledge prior work on adversarial attacks against IQA metrics (page 1, paragraph 2) while noting the gap in defense mechanism research.\n2. They correctly observe that most defense methods have been developed for object classification rather than IQA tasks (page 2, paragraph 1).\n3. The authors appropriately reference relevant work on adversarial training, purification methods, and certified defenses (page 2).\n\nOne area where the connection to broader literature could be strengthened is in relating their findings to the general principles of adversarial robustness. The paper focuses on the specific domain of IQA metrics, but could better articulate how their findings might generalize to other regression-based neural network tasks.", "essential_references_not_discussed": "no, the citation is appropriate", "other_strengths_and_weaknesses": "Strengths:\n1. The paper addresses a significant practical problem, as vulnerabilities in IQA metrics could affect search engine rankings, benchmarking, and content quality assessment.\n2. The benchmark methodology is thoroughly documented and reproducible, with implementation details and code availability mentioned.\n3. The large-scale subjective study (60,000+ responses) adds significant value by assessing perceptual quality, which cannot be fully captured by objective metrics.\n4. The paper provides nuanced insights, such as the observation that transformer-based IQA metrics have greater intrinsic robustness (page 7, paragraph 2).\n\nWeaknesses:\n1. The paper occasionally lacks clarity in explaining the practical implications of its findings. For example, the discussion of certified methods (page 6-7) focuses on technical metrics without clearly articulating the trade-offs involved in deploying such methods.\n2. In paragraph 2 of page 3, the authors state they increase IQA scores during attacks \"to reflect real-life applications,\" but don't adequately explain why this is more realistic than decreasing scores.\n3. The limitations section in the appendix (page 12) feels somewhat disconnected from the main paper. Some of these limitations, particularly regarding attack parameter handling, should be integrated into the main discussion.\n4. The paper introduces many evaluation metrics, which occasionally makes it difficult to determine which metrics should be prioritized when comparing defenses.", "other_comments_or_suggestions": "1. Page 2, paragraph 1: The authors state, \"making the development of a universally efficient and robust IQA metric impractical.\" This claim would benefit from more substantiation, as it's a significant assertion.\n2. Page 2, paragraph 1: The sentence \"To address this, we present a systematic comparison...\" seems to imply that their work addresses the impracticality of creating universally robust metrics, but instead it focuses on enhancing existing models through defense mechanisms. This connection should be clarified.\n3. Figure 1 provides a good overview, but the explanation of how the components interact could be improved.\n4. Page 20, lines 1230-1231: The sentence \"In summary, results do not change much across strength\" contradicts some of the data in Table 11, where performance differences between weak and strong attacks are substantial for some methods.\n5. The paper would benefit from a clearer discussion of the practical deployment considerations for these defense methods in real-world applications.", "questions_for_authors": "1. In your subjective study, Real-ESRGAN emerged as the top-performing method for perceptual quality (page 8), yet it doesn't perform as well on objective metrics. How do you reconcile this discrepancy, and what implications does this have for developing or selecting defense methods in real-world applications where human perception is the ultimate measure of quality?\n2. Your findings show that transformer-based metrics demonstrate greater intrinsic robustness against adversarial attacks (page 7). Could you elaborate on the architectural features that might contribute to this robustness, and do you believe future IQA metrics should prioritize transformer-based architectures specifically for their robustness properties?\n3. You mention on page 6 that certified methods are highly impractical due to computational overhead. Given the trade-off between security guarantees and computational efficiency, what modifications to certified methods might make them more viable for real-world IQA applications?\n4. Figure 3(b) suggests some defense methods perform differently on AI-generated images versus natural images. As AI-generated content becomes increasingly prevalent, how might defense strategies need to evolve specifically to address the unique characteristics of such content?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "12sYPp2Q4X", "reviewer": "Reviewer_Dmx9", "summary": "This paper presents the first comprehensive benchmark study on defense mechanisms for image quality assessment (IQA) metrics, systematically evaluating the performance of 30 defense strategies against 14 adversarial attacks on 9 IQA models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewers for their thorough evaluation and constructive feedback, and will apply their recommendations to refine the revised paper. We answer your questions below:\n1. For adversarial training, there is no inference overhead since the fine-tuning is done during the training phase. In contrast, purification and certified defenses require additional computation during inference, which is why their training time is explicitly reported.\n2. Yes, we conducted additional experiments to identify the optimal number of classes for discretization. The main challenge is balancing the trade-off between correlations SROCC and certified radius R. As the number of classes increases, SROCC increases, but R decreases. This occurs because a higher number of classes makes it easier to cross class borders during Monte Carlo sampling. The results of these experiments can be found in Table 9, located in Appendix A.3.4.\n3. Our analysis does not indicate significant fluctuations on the AGIQA-3K dataset specifically. The differences in ranking across datasets are minor and observed on all datasets; the closest one to AGIQA-3K is KADID1K. While data bias is important, our results suggest no substantial difference between AGIQA-3K and natural image datasets. Further investigation into potential biases remains a promising direction for future work.\n\nWe address your concerns below:\n1. In classification tasks, diffusion-based defenses only need to generate images that preserve class-relevant features, making perceptual quality less critical. In contrast, IQA tasks require preserving fine image details while removing adversarial noise, which standard diffusion models struggle with. To improve their applicability, future research should focus on task-specific adjustments, such as incorporating perceptual constraints or adapting diffusion models to minimize distortions introduced during defense.\n2. We report time complexity in milliseconds, so DiffPure processes an image in ~0.7 seconds. This makes it applicable in real-world applications.\n3. The mentioned study focuses on the robustness of neural-based image codecs, which makes two significant differences with our work: the area of research in our paper are IQA models, and instead of focusing on the robustness of the models themself, we focus on the effectiveness of defenses. Compared to [1] we use significantly different methodology, including datasets, evaluation metrics and a subjective study. Moreover, we evaluate 30 defense methods of three types (adversarial training, purification, and certified methods), while the authors of [1] used only 7 purification defenses.\n4. Indeed, we considered exactly this kind of attack. The formal definition is in section 3.1, page 2.\n\n[1] Kovalev E, Bychkov G, Abud K, et al. Exploring adversarial robustness of JPEG AI: methodology, comparison and new methods[J]. arXiv preprint arXiv:2411.11795, 2024.\n\nIf you have no further concerns, we would be sincerely grateful if you could consider raising the rating of our submission.", "claims_and_evidence": "All the content submitted has corresponding evidence to support it.", "methods_and_evaluation_criteria": "The methods and evaluation criteria proposed have great novelty and pioneering significance.", "theoretical_claims": "The paper lacks substantial theoretical proof.", "experimental_designs_or_analyses": "The paper provides ample experiments, which can largely substantiate the authors' viewpoints.The author provides abundant experimental results that strongly support their views.", "relation_to_broader_scientific_literature": "The author presents an interesting application prospect in the field of quality evaluation, using a defense model to assess IQA.", "essential_references_not_discussed": "The author has cited the vast majority of relevant literature. The third point of weakness is the need to supplement additional reference paper.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper has a clear structure and rich content, with valuable experiments provided in the appendix.\n2. The author has introduced large-scale subjective studies, and the substantial feedback results further validate the effectiveness of the defense mechanism.\n\nWeaknesses:\n\n1. The paper states that diffusion defense is effective in classification tasks but performs poorly in IQA tasks. However, it seems to lack in-depth analysis, merely mentioning that “future research needs task-specific adjustments.”\n2. In Table 2, the time used by DiffPure is 691.42 seconds. However, it appears to only be theoretically feasible and difficult to apply in real-time.\n3. Can the authors highlight the significant differences between this work and the existing work [1]? A more detailed response from the authors would better help me resolve my confusion and evaluate this paper.\n4. Have existing attack methods considered targeted attacks (e.g., disguising low-quality images as specific high-scoring ones)? Such methods would have a greater impact on ranking.\n\n[1] Kovalev E, Bychkov G, Abud K, et al. Exploring adversarial robustness of JPEG AI: methodology, comparison and new methods[J]. arXiv preprint arXiv:2411.11795, 2024.", "other_comments_or_suggestions": "My suggestions are already included in the weaknesses discussed earlier.", "questions_for_authors": "Q1. In Table 3, why is the training time provided for Cert Defense but not for AT Defense? Q2. When forcibly discretizing regression-based metrics into classification problems, have you considered the potential bias that may arise? This could potentially affect the defense performance. Q3. The defense results show significant fluctuations on the AGIQA-3K dataset, but the authors seem to have not conducted an in-depth analysis of the impact of data bias.  Q4.If the issues are clarified, I will consider increasing the score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review."}}]

Author rebuttal:

Meta-review: This paper presents a systematic benchmark dedicated to defending neural-network–based Image Quality Assessment (IQA) metrics from adversarial attacks. The evaluation is very comprehensive, studying the effectiveness of 30 defense strategies against 14 attack types on 9 no‐reference IQA metrics. An online leaderboard will also be provided to track the latest progress.

Overall, all reviewers agree that this work is comprehensive and well motivated, the experimental design is thorough, and the analysis yields interesting insights. Meanwhile, some concerns are raised, including 1) the practical guidance for IQA designers is not clear enough; 2) more discussions are needed regarding its relationships to RobustBench; and 3) more in-depth analysis is needed, like computational overhead. 

The authors’ rebuttal has satisfactorily addressed most of these issues, leading all reviewers to support the acceptance of this submission.
Decision: Accept (poster)