Adversarial Example Quality Assessment: A Large-scale Dataset and Strong Baseline

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 09:56:22

Abstract: Adversarial examples (AEs), which are maliciously hand-crafted by adding perturbations to benign images, reveal the vulnerability of deep neural networks (DNNs) and have been used as a benchmark for evaluating model robustness. With great efforts have been devoted to generating AEs with stronger attack ability, the visual quality of AEs is generally neglected in previous studies. The lack of a good quality measure of AEs makes it very hard to compare the relative merits of attack techniques and is hindering technological advancement. How to evaluate the visual quality of AEs remains an understudied and unsolved problem. 
In this work, we make the first attempt to fill the gap by presenting an image quality assessment method specifically designed for AEs. Towards this goal, we first construct a new database, called AdvDB, developed on diverse adversarial examples with elaborated annotations. We also propose a detection-based structural similarity index (AdvDSS) for adversarial example perceptual quality assessment. Specifically, the visual saliency for capturing the near-threshold adversarial distortions is first detected via human visual system (HVS) techniques and then the structural similarity is extracted to predict the quality score. Moreover, we further propose AEQA for overall adversarial example quality assessment by integrating the perceptual quality and attack intensity of AEs. Extensive experiments validate that the proposed AdvDSS achieves state-of-the-art performance which is more consistent with human opinions.

Main paper: 
 related work 2.1 adversarial attack techniques: 
 There have been extensive explorations for the generation of adversarial attacks in the literature, which can be categorized into gradient-based methods [10, 13, 14, 18, 23, 30] , optimization-based methods [4, 29] , and learning-based methods [1, 33] . The gradientbased methods originated from [14] where Goodfellow et al. proved that the existence of adversarial examples is the result of the linearity of DNN models and proposed Fast Gradient Sign Method (FGSM) to generate perturbations by the model gradient change.Following works enhance the attack ability by breaking the onestep gradient into iterative generation [18, 23, 30] , local enhancement [13, 37] , and data augmentation [10, 34] Image quality assessment (IQA) has been an active research topic in the last decades [2, 3, 15, 19, 22, 22, 25, 28, 39] . IQA can be divided into full-reference (FR)-IQA and non-reference (NR)-IQA, with FR-IQA has a reference image and the goal is to quantify the visual differences between reference and test images, and NR-IQA does not include reference images and aims to produce subjective human appreciation based on the quality of image content. Given the purpose of quantifying the adversarial distortions over a reference image in our work, we focus on FR-IQA here. Existing FR-IQA methods are either general-purpose methods [9, 11, 31, 39, 40] or distortion-specific methods [7, 16, 20, 41] . In the general-purpose methods, the assessment is conducted by extracting natural statistical characteristics of images, including traditional methods PSNR, SSIM [31] , MAD [11] , FSIM [39] , and deep learning based LPIPS [40] , PieAPP [25] , and DISTS [9] . These methods can evaluate the general quality of images but can not explicitly quantify certain distortions. In contrast, the distortion-specific methods evaluate the image quality by using the features of known distortion type, such as FRFSIM [16] and FADE [7] for defogging, SRIF [41] for image super-resolution, and HDR-VDP-2 [20] for tone-mapping. Despite remarkable progress have been made in FR-IQA, adversarial distortion is seldom explored in the literature.The perceptual evaluation of adversarial examples remains an open problem in the field. To our best knowledge, there have not yet developed any metrics specifically designed for adversarial examples.There is a prior work [12] that conducts an analysis of adversarial example perceptual quality, where the authors built a small-sized database and investigated the existing IQA methods in evaluating adversarial examples. However, the database only contains 13 images which is limited for evaluation, and a metric specifically designed for adversarial examples is still lacking. In this paper, we make the first attempt and focus on distortion-specific FR-IQA for evaluating the perceptual quality of adversarial examples. 
advdb: adversarial example assessment database: 
 In this section, we introduce the details of the proposed database, including the database construction and analysis.Collection. Considering our work focuses on the adversarial examples in the classification task, we choose samples from the classic databases for classification, namely CIFAR-10/100 and ImageNet databases. Specifically, we randomly choose 1, 500 samples from Annotation. We invited 11 professional researchers to perform subjective evaluation by comparing the adversarial examples and their corresponding reference images. The subjective software was conducted under the same environment with a 4K Ultra HD LED Monitor. In addition, the subjects are given instructions before the experiments to circumvent bias in this task. As shown in Figure 1 , a customized interface is designed to render a sequence of images simultaneously. The left image is the original reference image, and images 1-5 are adversarial examples generated by different attack algorithm settings in random order. The subjects are asked to give a score from 1 to 5 for each adversarial example based on the perceptual possibility of this image being attacked. A higher score means that the image is perceptually abnormal, which denotes a higher possibility that the image is perturbed. Finally, the outlier individual score is removed and then the remaining scores are averaged as the MOS of each adversarial example.  
approach: 
 In this section, we introduce our approach, whose main idea is to construct a benchmark image quality assessment metric that is specifically designed for AEs following an overall quality metric for comprehensively evaluating AEs from both perceptual quality and attack intensity. Design. Here our goal is to develop a perceptual quality assessment for adversarial examples. We first revisit the formulation of AEs which is described as:EQUATIONwhere 𝑥, 𝛿, and 𝑥 𝑎𝑑𝑣 denotes clean image, adversarial perturbation, and AE, respectively. B 𝜖 (𝑥) denotes the ℓ 𝑝 -norm ball centered at 𝑥 with radius 𝜖, i.e., B 𝜖 (𝑥) = {𝑥 𝑎𝑑𝑣 : ∥𝑥 𝑎𝑑𝑣 -𝑥 ∥ 𝑝 ≤ 𝜖}. Existing advances in adversarial attacks try to find a proper 𝛿 to make the model loss maximized. Since the AEs are already constrained by 𝐿 𝑝norm, the distortions over benign images are near-threshold, which brings challenges in evaluation. Thus we consider the design of the metric from two aspects: (i) It should reflect the image degradation that is consistent with MOS thus HVS should be incorporated;(ii) It has been pointed out that human visual system tends to locate visible differences in near-threshold distorted images for giving the MOS while attempts to recognize image content when faced very low-quality images. Based on this prior, we employ a detection scheme where the visual saliency is first detected and then the differences are further quantified by conducting structural similarity evaluation. An overall overview of the whole process is shown in Figure 4 .We first conduct visual saliency detection. This phase is composed of two steps, gamma correction and contrast sensitivity function transformation. Gamma Correction. Let 𝑥 denotes a clean image, and 𝑥 𝑎𝑑𝑣 denotes a corresponding adversarial example. We first transform the images from RGB space to YCbCr space for adapting the human perceptual system. According to the HVS theory [11] , human eyes are more sensitive to darker regions in images and less sensitive to brighter regions. Thus, we employ gamma correction (GC) to preserve as many dark details as possible in the images, which is defined as:EQUATIONwhere 𝛾 is empirically set to be 1 2.2 as suggested in previous works. Note that the GC is applied to both 𝑥 and 𝑥 𝑎𝑑𝑣 for better quantification. Contrast Sensitivity Function Transformation. Human eyes are believed to be sensitive to the spatial frequency of images. To account for variations in different spatial frequencies, we adopt the contrast sensitivity function (CSF) proposed by Mannos and Sakrison [19] with adjustments specified by Daly [8] , which is calculated asEQUATIONWhere 𝑓 is the radial spatial frequency in cycles per degree of visual angle (i.e., 𝑐/𝑑𝑒𝑔), 𝜃 is the orientation and varies in [-𝜋, 𝜋], and 𝑓 𝜃 = 𝑓 /[0.15𝑐𝑜𝑠 (4𝜃 ) + 0.85] is an orientation-based modification of 𝑓 that along the diagonal orientations to account for decreased contrast sensitivity. Note that we adopted the adjusted CSF which has a lowpass by setting frequencies below 𝑓 𝑝𝑒𝑎𝑘 to 0.981. Following previous works [8] , the 𝑓 and 𝜃 can be computed from discrete Fourier EQUATIONEQUATIONwhere 𝜌𝜗 tan( 𝜋 180 ) is the display visual resolution in units of pixels per degree of visual angle [32] . We apply CSF to filter both the corrected benign image and adversarial example, which can be described as:EQUATIONwhere F [•] and F -1 [•] are the DFT and reverse DFT respectively. Note that for now, 𝑥 ′ and 𝑥 ′ 𝑎𝑑𝑣 are linearly proportional to both perceived luminance and perceived contrast. The visual saliency maps are calculated as x = 𝑥 -𝑥 ′ , and x𝑎𝑑𝑣 = 𝑥 𝑎𝑑𝑣 -𝑥 ′𝑎𝑑𝑣 , respectively. 4.1.2 Structural Similarity Quantification. After the visual saliency map is obtained, inspired by the data analysis stated in Sec. 3.2 where the structural information is shown to be able to reveal the adversarial distortion, we use the structural similarity index to quantify the perceived distortion. Specifically, it combines the luminance, contrast, and structure differences between adversarial examples and benign images and is defined as:𝐴𝑑𝑣𝐷𝑆𝑆 = [𝑙 ( x, x𝑎𝑑𝑣 )] 𝜆 1 • [𝑐 ( x, x𝑎𝑑𝑣 )] 𝜆 2 • [𝑠 ( x, x𝑎𝑑𝑣 )] 𝜆 3 , (7)where 𝑙, 𝑐, 𝑠 denote the luminance, contrast, and structure comparison respectively, which can be computed from the intensity distribution as described in [11, 31] . 𝜆 1 , 𝜆 2 , 𝜆 3 represents the weights for the three terms, where we set 𝜆 1 = 1, 𝜆 2 = 0.48, and 𝜆 3 = 0.48. Please refer to Sec. 5.3.2 for the parameter adjustments.Here we take a step further to develop an overall quality assessment for adversarial examples. As the two crucial factors for adversarial examples are the invisibility and lethality, we consider the perceptual quality and attack intensity as two major terms to evaluate the overall adversarial example quality. That is, a good adversarial example should have good visual quality and strong attack intensity. As we have AdvDSS as the perceptual quality index, we need to quantify the attack intensity of adversarial examples. Inspired by existing attacks [17, 36] Attack Intensity Quantification. To quantify the distribution divergence of adversarial examples and benign images for evaluating the attack intensity of adversarial examples, we use the Jensen-Shannon (JS) divergence to measure the distribution divergence. Specifically, we first feed 𝑥 and 𝑥 𝑎𝑑𝑣 into the backbone model 𝑓 and obtain the logits distribution 𝑝 𝑥 = 𝑓 (𝑥) and 𝑝 𝑥 𝑎𝑑 𝑣 , respectively. The attack intensity of 𝑥 𝑎𝑑𝑣 is then computed as:EQUATIONwhere the JS(•) denotes the JS divergence computation. Note that the larger JS divergence value between adversarial example and benign image denotes a stronger attack intensity since the adversarial example can easily trigger different predictions from that of benign image. An illustration is shown in Figure 5 .Combining the perceptual quality and attack intensity, the AEQA is defined as:EQUATIONwhere 𝐴𝐸𝑄𝐴 ∈ [0, ∞] denotes the overall quality of an adversarial example. A higher AEQA value indicates a better quality of adversarial example. 𝛼 is the weight for adjusting the impact of attack intensity and image quality. To achieve the balance between these two factors, we adopt a dynamic scheme to determine 𝛼 as follows:EQUATION)where 𝛽 1 and 𝛽 2 are free parameters and we empirically set 𝛽 1 = 0.8 and 𝛽 2 = 0.1. By this dynamic weight determine strategy, the AEQA pays more attention on the image perceptual quality when the value of 𝐷 𝐽 𝑆 is large, and gives more weight on the attack intensity when the value of 𝐷 𝐽 𝑆 is small. 
experiments 5.1 experimental settings: 
 Compared Baselines. We compare the proposed AdvDSS with extensive state-of-the-art IQA metrics on AdvDB, including FR-IQA metrics PSNR, SSIM [31] , CW_SSIM [26] , FSIM [39] , LPIPS [40] , and LPIPS-VGG [40] , and NR-IQA metrics NIQE [22] , BRISQUE [21] , MUSIQ [15] , and PI [3] .Evaluation Criteria. We adopted four metrics to comprehensively evaluate our approach: SROCC, KROCC, PLCC, and RMSE. SROCC and KROCC are employed to measure the prediction monotonicity of IQA methods, while PLCC and RMSE are used to measure the prediction accuracy. Note that higher SROCC, PLCC, KROCC values and lower RMSE value denote better performance.Implementation Details. Our method is implemented with Python on an NVIDIA RTX 3060Ti. In order to measure the attack intensity of adversarial examples, we use the backbone model i.e., Inception v3 network pre-trained on ImageNet, which produces the adversarial examples for AdvDB to generate the logits distribution of adversarial examples and clean images. Note that the AEQA cannot assist in the evaluation of a single adversarial example whose source model is unknown, but in the evaluation of an adversarial attack technique.Comparisons with the Existing IQA Metrics. We first compare our AdvDSS with the existing IQA metrics in evaluating the perceptual quality of adversarial examples. The tested SROCC, PLCC, KROCC, and RMSE results are summarized in Table 3 , where we can have the following observations: 1) The NR-IQA methods including MUSIQ, PI and BRISQUE generally do not perform well for evaluating the perceptual quality of adversarial examples, as the SROCC and PLCC values of these methods for Deepfool attack are less than 0.4. There is no surprise because adversarial perturbations are always near-threshold, it is hard to directly perceive the distortions without reference images; 2) Compared to the NR-IQA methods, the FR-IQA methods perform much better since they use references. Among them, SSIM outperforms the other methods and was second only to our method, suggesting that the structural information may reveal useful perceptual attributes of adversarial perturbations; 3) The SROCC, PLCC, and KROCC values of our method are larger than all the compared methods, and for RMSE are smaller than that of the others. This denotes that, for adversarial examples, our method is better correlated with subjective evaluation scores than the existing IQA methods. The benefits come from both the detection-based scheme and structural similarity evaluation in our AdvDSS design. We also visualize the plot distribution of objective scores versus MOS obtained on AdvDB in Figure 6 . We can see that the points obtained by our proposed AdvDSS are more tightly distributed on the fitted curve than other methods, which also indicates that our method has a better consistency with the subjective evaluation scores, and is more suitable for evaluating the perceptual quality of AEs.Studies. Now we take a closer look at the design of our AdvDSS. We examine the effect of each component in AdvDSS by conducting ablation studies. Specifically, we testify AdvDSS w/o detection scheme, w/o CSF, and w/o GC. The results are reported in Table 4 . As we can see, without the detection scheme, the performance drops in all metrics since the adversarial distortions are nearthreshold and hard to capture. Employing the detection scheme can improve the evaluation performance, and AdvDSS achieves the best when equipped with both GC and CSF operations.Adjustments. In our AdvDSS, the weights 𝜆 1 , 𝜆 2 , and 𝜆 3 denote the impact of luminance, contrast, and structure in the [18] 0.8570 0.9761 0.7673 FGSM [14] 0.7442 0.9386 0.6422 MIM [10] 0.7346 0.9510 0.5892 as the value of 𝜆 2 increases. The peak value appears when 𝜆 2 = 0.48. We adopt this setting throughout the whole experiments. Examples using AEQA. We also conduct an interesting experiment in which we try to generate AEs by directly optimizing the AEQA. Specifically, we set the AEQA as the objective and use L-BFGS algorithm to directly generate an adversarial example from a clean image with high AEQA score. We use the Inc V3 pre-trained on ImageNet as the target model and also the backbone to compute JS-Distance. We set the iteration step as 20 during optimization. The attacks are performed on different datasets and the results are reported in Table 7 . It can be observed that the AEs exhibit a high attack success rate on all the datasets. Due to the page limitation, more details and qualitative results are referred to our supplementary material. 
conclusion and discussion: 
 In this paper, we have constructed the first and largest adversarial example database, namely AdvDB, for assessing the perceptual quality of adversarial examples. Based on AdvDB, we have proposed a detection-based quality assessment approach, AdvDSS, which employs a visual salience detection scheme to first obtain the salience map from the HVS perspective and then quantify the distortions by using structural similarity between adversarial examples and benign images. Also, we take a step further to propose AEQA to conduct an overall quality assessment of adversarial examples by integrating the perceptual quality and attack intensity.The experiments demonstrate that our method can achieve superior performance than the existing IQA methods.Discussion. In this work, the AdvDSS is performed in a traditional HVS manner and did not employ deep models for two reasons: 1)The adversarial examples have the property of transferability that can potentially disturb other deep models; 2) The proposed method is a simple yet powerful benchmark method that can be easy to apply, which we believe can be set as a baseline for future research.Limitations. The proposed database only contains gradient-based attacks since they are the most effective attack strategies for now.The MOS annotations can also be manually decomposed into diverse attributes such as contrast, color, and saturation for deep analysis.  
: 
 discovery has attracted growing attention and spawned considerable research for adversarial attacks. In recent years, numerous adversarial attack methods have been proposed, including gradientbased [5, 10, 13, 14, 18, 23, 30] , optimization-based [4, 29] , and learning-based attacks [1, 24, 33] .As a general rule, adversarial attacks generate adversarial examples to fool the target model by pursuing two goals: (1) Invisibility: Perceptually, the adversarial examples should be similar-looking to the benign images; (2) Lethality: In the target models, the distribution of generated adversarial examples should differ from that of benign images and trigger wrong prediction behaviors of DNN models. However, while numerous efforts are devoted to developing adversarial examples with stronger attack ability, the perceptual quality of adversarial examples is generally neglected. Almost all the existing methods adopted 𝐿 𝑝 -norms (e.g., 𝐿 1 , 𝐿 2 , and 𝐿 ∞ ) as the constraint to keep the generated adversarial examples similarlooking to the original image; however, the norm-based constraint does not correlate with human perceptual quality since it computes the pixel-wise error and does not include the properties of human visual system (HVS). We present a diagnosis report of the perceptual quality of AEs in Table 1 , where we can see that the 𝐿 𝑝 -norm cannot well-assist the perceptual quality of AEs, and different attack strategies can produce AEs with different perceptual qualities. The lack of good quality measurement of adversarial examples makes it very hard to compare the relative merits of adversarial attack techniques.On the other hand, image quality assessment (IQA) has been an active research topic in the last decades [7, 15, 16, 20, 25, 31, 39, 41] . An intuitive way is to directly take advantage of existing IQA metrics; however, most of the existing IQA metrics are designed to capture certain distortions, such as blur and color fading for defogging [7, 16] , edge sharpness for super-resolution [41] , content compensation for tone-mapping [20] . Since adversarial perturbations have different properties from those applications, using these IQA metrics can lead to unsatisfied results (see the fifth row of Table 1 ). Developing a new reliable metric specifically designed for Adversarial examples remains an unsolved and challenging problem.In this paper, we make the first attempt to fill a gap in the literature by presenting an image quality assessment metric specifically designed for evaluating the perceptual quality of adversarial examples. To achieve this goal, we face two challenges: First, a suitable adversarial example quality assessment metric should reflect the degradation of the image that is consistent with human subjective evaluation, e.g., the mean opinion score (MOS). However, there is no such off-the-peg dataset for evaluation; Second, the perceptual properties of adversarial examples are unknown. Since the existing attacks have already used 𝐿 𝑝 -norm constraint, the adversarial perturbations are near-threshold (i.e., just-visible to human eyes), which is hard to capture and quantified. 2) The adversarial distortion is near-threshold that is just-visible to human eyes; 3) Typical IQA metric can hardly capture the distortion in AEs, e.g., the AE produced by PGD has a better visual quality than BIM but achieves a lower PSNR value.To address these challenges, we first construct a new annotated adversarial example quality assessment database, named "AdvDB", focusing on adversarial examples with diverse perturbations, which is the first and largest database for assessing adversarial examples. Specifically, we collected 18, 346 adversarial examples produced by 5 dominant adversarial attack algorithms with different parameter settings, and each image is annotated with two attributes by 11 professional subjects. Based on the AdvDB, we conducted analysis and found that the structural information can reveal the perceptual quality of adversarial examples, thus we further proposed a visual Detection-based Structural Similarity index (AdvDSS) for assessing the perceptual quality of adversarial examples. To capture the near-threshold adversarial distortions from the perceptual perspective, AdvDSS employs a detection scheme that incorporates HVS technology to first compute the visual saliency error map and then conduct structural similarity comparison to predict the perceptual scores. Moreover, by integrating the perceptual quality and attack strength of adversarial examples, we take a further step and present an adversarial example quality assessment (AEQA) for comprehensively evaluate the quality of adversarial examples.In a nutshell, our contributions can be summarized as follows:• We build so far, the largest annotated adversarial example quality assessment database named "AdvDB". Specifically, we collected  
 
 Appendix:  


Decision: A rebuttal letter is submitted, and the reviewers acknowledged the clarification and update for the paper. All reviewers merge to the rating of borderline accept by considering the work novelty and thorough experiment design and evaluation. However, more experimental details will be added and further clarification of novelty will be updated in the camera-ready version. A decision of accept is suggested.

Reviews: review: paper_summary: paper_summary: This paper presents a large-scale adversarial dataset, AdvDB, to serve as a foundation for image quality assessment for adversarial examples (AEs), a detection-based structural similarity index, AdvDSS for AR perceptual quality assessment, and a combinational index, AEQA, which measures the quality of AEs from the perspectives of perception quality and attack intensity. The authors conduct experiments with 5 attack methods, 11 baseline IQA metrics, and 4 evaluation criteria. The experiment results show that AdvDSS outperforms the baseline metrics among all adversarial attacks and evaluation criteria.
strength_weakness: strength_weakness: + Timely and interesting problem.
+ New dataset and perceptual quality assessment index for AEs, which may benefit the community.
+ Large-scaled experiments with various baseline metrics, evaluation criteria and attack methods.
limitations: limitations: - Novelty requires further clarification.
- Lack of justification for some technical details.
- The experiments for AEQA seem incomplete.

### Detailed comments:

1. Some related works have studied the adversarial examples with visual quality[1-3], although few works systematically investigate the evaluation of AEs from a perceptual quality perspective. The authors propose AdvDss, a new index to measure the corresponding perceptual quality; however, to me, it is more like a combination of existing techniques. The authors may further highlight the novelty of such an index design.

2. The authors select 5 methods for adversarial example generation; nevertheless, there are some concerns regarding the selection criteria. In particular, it is not clear why selecting these 5 methods for generation; are they representative and/or commonly used? Besides, some of these methods do not seem to be state-of-the-art AE generation approaches; it is vital to justify the rationale behind such selection. Also, the authors mentioned that they only took the gradient-based attacks for the experiment; however, I would recommend including optimization and learning-based methods to deliver a more generalized conclusion.

3. Some formulas need further explanation. The authors define many nuanced parameters to facilitate the design of the similarity index (i.e., equations 7, 9, 10). There is a lack of details about the design scheme of these parameters, which may confuse readers who are not familiar with this field.

4. The authors propose two evaluation indexes, AdvDSS and AEQA, to facilitate the quality measurement of AEs from a perceptual quality aspect and an integrated aspect, respectively. Regardless of the experiments for AdvDSS, the paper does not provide a comprehensive evaluation of AEQA. The authors may discuss more about the effectiveness of AEQA among different attack methods (like Table 6) and justify the trade-off between the perceptual quality assessment and the attack intensity assessment, namely, AdvDSS and Djs.

5. The manuscript is overall written in a motivating and systematic manner.


[1] "Improving the invisibility of adversarial examples with perceptually adaptive perturbation." Information Sciences 635 (2023).

[2] "Chaotic Variational Auto encoder-based Adversarial Machine Learning." (2023).

[3] "Improving perceptual quality of adversarial images using perceptual distance minimization and normalized variance weighting." In The AAAI-22 Workshop on Adversarial Machine Learning and Beyond. (2021)
paper_summary: paper_summary: The paper proposes a novel benchmark adversarial example quality assessment metric for comprehensively evaluating the quality of adversarial examples.
strength_weakness: strength_weakness: The paper demonstrates the SOTA performance of the proposed adversarial example quality assessment metric through large-scale experiments. The paper also generates better adversarial examples based on this metric.
limitations: limitations: The experiment only includes gradient based attacks, and these attacks are relatively old. I hope the experiment can supplement some other attacks, such as optimization based or frequency domain based, etc. I am confused that CIFAR-10/100 was also used for dataset construction, and the number of pixels in these datasets is very small. Participants may have difficulty distinguishing the adversarial perturbations, especially when the metrics have five levels. This paper focuses on the issue of single media and does not align well with the theme of multimedia.
paper_summary: paper_summary: This paper introduces a dataset named AdvDB, which comprises 18,346 adversarial examples generated using five different attack methods, complete with annotations. Building on this dataset, the authors develop AdvDSS, a system designed to capture the characteristics of adversarial perturbations. Additionally, they propose integrating AdvDSS with the Jensen-Shannon divergence to create AEQA, a new quality assessment metric designed specifically for evaluating adversarial examples.
strength_weakness: strength_weakness: (1) This paper focuses on the image quality assessment of adversarial examples, a topic that has not yet been extensively explored within the community.

(2) The proposed AdvDSS system demonstrates performance that is slightly superior to existing full-reference image quality assessment (FR-IQA) metrics, such as LPIPS.

(3) In my opinion, the most significant contribution of this work is the annotation of the MOS for adversarial examples through subjective experiments. It experimentally demonstrates that existing FR-IQA metrics correlate well with the MOS. Therefore, it is reasonable for researchers in the attack community to use existing FR-IQA metrics.
limitations: limitations: Some main questions about this work:

Q1: About the dataset. As the paper illustrated, there are totally #total=1500(samples)*5(attack methods)*5(parameters choices) adversarially perturbed images in therotical. However, only 18,346 images successfully attacked the backbone, so the attack success rate is only about 18,346/#total < 50%. Even without considering cases where $\epsilon < 0.01$, the attack rate is about 60%. This seems unusual, as all the attack methods used in this paper are generally known to achieve over 90% attack sucess rate when $\epsilon \geq 0.03$.

Q2: About the meaning of AdvDSS. As the paper shows in Fig.3 and Tab.3, exsting FR-IQA metric, such as LPIPS and LINIQA, align well the human perceptual system when assessing the quality of adversarial examples. The proposed AdvDSS only marginally improves the SROCC (from Tab. 3, it increases SROCC from LINIQE-0.9415 to 0.9423), KROCC, PLCC and the RMSE. I believe the improvement is minimal. In other words, existing models could suffice for evaluating adversarial samples.

Q3: The motivation behind AEQA is unclear. According to the authors, AEQA is not applicable if the source model is unknown, suggesting that AdvDSS might be more broadly applicable. It raises the question: In what scenarios is AEQA more useful than AdvDSS? This needs to be addressed to understand the distinct advantages of AEQA over AdvDSS.

Q4: The results shown in Fig.7 are puzzling. The AEQA scores for clean images (epsilon=0) are lower than those for images with small adversarial perturbations (epsilon=0.03), suggesting that the quality of clean images is perceived to be worse than that of slightly perturbed adversarial examples. This seems counterintuitive and potentially incorrect. Even if I misunderstood and the Fig.7 does not include epsilon=0, the conclusion that the AEQA score increases and then decreases with larger epsilon values still seems strange to me.

Other minor questions that do not afftect my rating:

M1: In calculating RMSE in Tab.3, how do the authors map the scores from existing FR-IQA metrics to the MOS range? For instance, SSIM is scaled between [0,1], but the MOS collected in this work ranges from [1,5]. I think the RMSE is related with the mapping.

M2: Some typos in the paper. (1) Tab.2, all methods are incorrectly marked the same in the column $L_p$ norm. (2) Line 344(right), I think there are only 25 attack strategies, not 75. (3) Tab.6, in the caption, $L_2 -> L_\infty$. (4) Fig.3(b), the score range should be left-closed and right-open, e.g. $BIM[1,2), BIM[2,3)$.