Attacking Perceptual Similarity Metrics
Authors: Abhijay Ghildyal
Feng Liu
Venue: Accepted by TMLR ()

Abstract: Perceptual similarity metrics have progressively become more correlated with human judgments on perceptual similarity; however, despite recent advances, the addition of an imperceptible distortion can still compromise these metrics. In our study, we systematically examine the robustness of these metrics to imperceptible adversarial perturbations. Following the two-alternative forced-choice experimental design with two distorted images and one reference image, we perturb the distorted image closer to the reference via an adversarial attack until the metric flips its judgment. We first show that all metrics in our study are susceptible to perturbations generated via common adversarial attacks such as FGSM, PGD, and the One-pixel attack. Next, we attack the widely adopted LPIPS metric using spatial-transformation-based adversarial perturbations (stAdv) in a white-box setting to craft adversarial examples that can effectively transfer to other similarity metrics in a black-box setting. We also combine the spatial attack stAdv with PGD ($\ell_\infty$-bounded) attack to increase transferability and use these adversarial examples to benchmark the robustness of both traditional and recently developed metrics. Our benchmark provides a good starting point for discussion and further research on the robustness of metrics to imperceptible adversarial perturbations.

Published in Transactions on Machine Learning Research (05/2023)
Attacking Perceptual Similarity Metrics
Abhijay Ghildyal
ablijay &pzz. edu
Department of Computer Science Portland State University
Feng Liu
fliu@pde. edu
Department Computer Science Portland State University
Reviewed on OpenReview: htips: // openreview. net/ met/ forum? id- r9VGSpbbRO
Abstract
Perceptual similarity metrics have progressively become more correlated with human judg- ments on perceptual similarity; however, despite recent. advances, the addition of an im- perceptible distortion can still compromise these metrics. In our study, we systematically examine the robustness of these metrics to imperceptible adversarial perturbations. Fol- lowing the two-alternative forced-choice experimental design with two distorted images and one reference image, we perturb the distorted image closer to the reference via an adversar- ial attack until the metric flips its judgment. We first show that all metrics in our study are susceptible to perturbations ions generated via common adversarial attacks such as FGSM, PGD, and the One-pixel attack. Next, we attack the widely adopted LPIPS metric us- adversarial perturbations (stAdv) in white-box setting to craft. adversarial examples that can effectively transfer to other similarity metries in attack
ing ppatial-transformatoonnbo
black-box setting. We also combine the spatial attack stAdv with PGD to increase transferability and use these adversarial examples to benchmark the robustness of both traditional and recently developed metrics. Our benchmark provides a good start- ing point for discussion and further research on the robustness of metrics to imperceptible adversarial perturbations. Code available at https://tinyurl.com attackingssm.
1 Introduction
Comparison of images using a similarity measure crucial for defining the quality an image for many applications in image and video processing. Recently, perceptual similarity metrics have become vital for
optimizing and evaluating deep neural networks used low-level computer vision tasks (Dosovitskiy & Brox
2016 Zhu et al.] 2016 Johnson
al.] 2016 Ledig et al. 2016 Sajjadi et al.] 2017: Kettunen et al.) 2019a
Karras et al. 2020). Learned percept image patch similarity (LPIPS) metric by Zhang et al. (2018b) is one such widely adopted perceptual similarity metric. Apart from these image enhancement and generation tasks, similarity metrics are also used in
Zhang et al. 2020 Son et al. 2020 Niklaus & Liu 2020
optimizing, constraining, and evaluating adversarial attacks (Szegedy et al. 2014 Goodfellow et al.) 2015 Carlini & Wagner, 2017 Kurakin et al., 2017 Hosseini & Poovendran 2018 Dong et al.) 2018 Shamsabadi et al. 2020 Laidlaw & Feizi, 2019). limitation in early adversarial robustness studies has been the use of Lp norms as distance metric to judge the imperceptibility of synthesized adversarial perturbations. These attack methods optimized for stronger adversarial perturbations while keeping the perturbations within imperceptibility levels via an lp norm. However, as we now know, fp distance metrics are not good proxy to human perception, and several learned perceptual similarity metrics have been developed to correlate better with human judgment. More recently, Laidlaw et al. (2020) proposed neural perceptual threat models (NPTM) and subsequently defense method that could generalize well against unforescen adversarial attacks, in which, instead of an bp norm, the severity, or perceptibility of the adversarial perturbations, is bounded LPIPS, learned perceptual similarity metric. Hence, they employed LPIPS in their optimization to

---

Published in Transactions on Machine Learning Research (05/2023)
f
adv
?
Humon SSIM
0.059
0.037
(a > ((mash))
0.595
0.697
0.144
LPIPS
0.045
Figure 1: In is more similar to Iref than Io according to all perceptual similarity metrics and humans. We attack In by adding imperceptible adversarial perturbations (6) such that the metric (f) flips its earlier assigned rank, ie., in the above sample, lo becomes more similar to Leef.
generate adversarial examples. However, it remains unanswered whether LPIPS itself is robust towards imperceptible adversarial perturbations. The question then arises, "How robust are perceptual similarity metrics against imperceplible adversarial perturbations?" We posit that more accurate and robust perceptual similarity metrics can lead to stronger defenses against adversarial threats. In a recent study, Mahloujitar jet al. (2020) showed that a better perception model to test. the imperceptibility of adversarial perturbations can lead to stronger robustness guarantees for image classifiers.
We begin by examining whether it is possible to find imperceptible adversarial perturbations that can overturn uarceptual similarily judgments. It is well known that machine learning models are easy to fool with adversarial perturbations imperceptible to the human eye (Szegedy et al. 2014). Interestingly, similar imperceptible perturbations can bring about a sizcable change in the measured distance of a distortod image from its reference. As shown in Figure [ we examine this change in measured distances using a two-alternative forced choice (2AFC) test example, where the participants were asked, "which of the two distorted images (lo and 41) is more similar to the reference image )". Then, we apply an imperceptible perturbation to the distorted image that has the lower perceptual distance (i.c., more similar to Iref) to see if the similarity judgment for the sample overturns. In such a scenario, human opinion remains the same, while perceptual similarity metrics often overturn their judgment. Perceptual similarity metries measure the similarity between two images and are widely used in many real-world applications. Having a robust metric is sometimes critical. Copyright protection is one critical use case where automatic image similarity assossment plays an important role. A malicious user can upload copyright-protect imperceptible perturbations, making the images less detectable the internet. Intcrestingly, recent work began to investigate the perceptual robustness of image quality assessment (IQA) methods via adversarial perturbations Zhang et al. (2022) and Lu et al.] (2022). However, these studies focus on no-reference image quality assossment methods. The robustness of perceptual similarity metrics, often used as full-reference image quality assessment methods, has been less studied.
images with
There are two popular approaches to examining the robustness of perceptual similarity metrics: (1) addition of small amounts of hand-crafted distortions such as translation, rotation, dilation, JPEG compression, and Gaussian blur, and (2) analysis of more advanced adversarial perturbations. For the former, seminal
contributions have been made (Ma et al. 2018 Ding et al. 2020 Bhardwaj et al. 2020 Gu et al., 2020).
However, in contrast to previous work, we focus on performing the latter as it has not received considerable attention. In our work, we demonstrate that threats to similarity metrics can be casily created using common gradient-based iterative white-box attacks, such as fast gradient. sign method (FGSM) (Goodfellow ct al.) 2015) and projected gradient descent (PGD) (Madry et al.) 2018). These attacks do not dcform the structure but rather manipulate pixel values in the image. In recent research, questions regarding the robustness of perceptual similarity metrics towards geometric distortions are of central interest (as discussed above). Hence, we also use the spatial adversarial attack st.Adv (Xiao et al.) 2018), which geometrically deforms the image. It utilizes optical flow for crafting perturbations in the spatial domain. We use this attack to generate adversarial samples for comparing the robustness of various metrics.
We also examine whether perceptual motrics can be attacked in black box scttings. To this end, we first use the One-pixel attack (Su et al. 2019) that uses differential evolution (Storn & Price 1997] optimize a single-pixel perturbation on the adversarial image. While compared to white box attacks such as FGSM and
2

---

Published in Transactions on Machine Learning Research (05/2023)
PGD, this One-pixel attack does not need the model parameters of similarity metric, it needs to access its output. Therefore, we furthermore explore transferable attacks (Liu et al., 2017 Xie et al. 2018 2019) which requires no information about the model. Specifically, we generate adversarial examples using the parameters of a source model and use them to attack a target model. In our study, we use (PIPS(AlexNet) as the source model and attack it via stAdv. We exlend the successfully attacked examples onto a larget perceptual similarity metric. It is a black-box setting as does not require access to the target perceptual metric's 's parameters. In our work, we combine stAdv (spatial attack) with PGD (foe-bounded attack) that strengthens the severity of the adversarial examples.
The main contribution of this paper is the first systematical investigation on whether existing perceptual similarity metrics are susceptible to state-of-the-ar adversarial attacks. Our study includes a set of carefully selected attacking methods and a wide variety of perceptual similarity metrics. Our study shows that all these similarity metrics, including both traditional quality metrics and recent deep learning-basod metrics, can be successfully attacked by both white-box and black-box attacks.
2 Related Work
2011] were designed to ap- features of local regions in the images. Whereas, recent metrics (Zhang et al.
Earlier metrics such as SSIM (Wang al., 2004) and FSIMe (Zhang al.,
proximate the human visual systems' ability to perceive and distinguish images, specifically using statistical
2018b Prashnani et al. 2018
Ma et al. 2018 Kettunen et al. 2019b Ding et al.
2020 Bhardwaj et al. 2020 Ghildyal & Liu 2022) are deep
neural network based approaches that learn from human judgments on perceptual similarity. LPIPS (Zhang et al.
2018b is one such widely used metric. It leverages the activations of a feature extraction network at
each convolutional layer to compute differences between two images which are then passod on to lincar layers to finally predict the perceplual similarity score. Prashnani et. al. (2018) developed the Perceptual Image Error Metric (PieAPP) that uses weight-shared feature extractor on each of the input images, followed by two fully-connected networks that use the difference of those features to generate patch-wise errors and corresponding weights. The weighted average of the errors is the final score. Liu et al. (0022) used the Swin Transformer (Liu et al.] 2021) for multi-scale feature extraction in their metric, Swin-IQA. Its final score is the average across all cross-attention operations on the difference between the features. Swin-IQA performs better than the CNN-based metrics in accurately ranking, according to human opinion, the distorted images synthesized by methods from the Challenge on Learned Image Compression (CLIC) 2022).
In recent years, apart from making the perceptual similarity metrics correlate well with human opinion, there has been growing interest in examining their robusiness towards geometric distortions. [Wann & Simoncelli (2005) noted that geometric distortions cause consistent phase changes in the local wavelet coefficients while structural content stays intact. Accordingly, they developed complex wavelet SSIM (CW-SSIM) that used phase correlations instead of spatial correlations, making it less sensitive to geometric distortions. [a et al.) (2018) benchmarked the sensitivity of various metrics against misalignment, scaling artifacts, blurring, and JPEG compression. They then trained a CNN with augmented images to create the geometric transformation invariant metric (GTI-CNN). In a similar study, Ding et al. (2020) suggested computing global measures instead of wisel-wise differences and then blurred the feature embeddings by replacing the max pooling layers with l2-pooling layers. It made their metric, deep image structure and texture similarity (DISTS), robust to local and global distortions. Ding et al.] (2021) extend DISTS making it robust for perceptual optimization of image super-resolution methods. They separate texture from the structure in the extracted multi-scale feature maps via a dispersion index. Then, to compute feature differences for the final similarity score, they modify SSIM by adaptively weighting its structure and texture measurements using the dispersion index. Bhardwaj et al. (2020) developed the perceptual information metric (PIM). PIM has a pyramid architecture with convolutional layers that generate multi-scale representations, which get processed by dense layers to predict mean vectors for each spatial location and scale. The final score is estimated using symmetrized KL divergence using Monte Carlo sampling. PIM is well correlated with human opinions and is robust against small image shifts, even though it is just trained on consecutive frames of a video, without any human judgments on perceptual similarity. [Zoolbe et al./ (2020) usod Watson's 's perceptual model (Watson) 1993) and replaced discrete cosine transform with discrete fourier transform (DFT) develop a perceptual similarity loss function robust against small shifts. Kettunen et al. (201bb) compute the average I.PIPS
3

---

Published in Transactions on Machine Learning Research (05/2023)
score over an ensemble of randomly transformed images. Their self-ensembling metric E-LPIPS is robust to the Expectations over Transformations attacks {Athalye et al.) 2018 Carlini & Wagner 2017). Our attack approach is similar to an attack investigated by Kettunen et al. (2019b), where the adversarial images look similar but have a large LPIPS distance (smaller distance means more similarity). However, they only investigate the LPIPS metric. Ghildyal & Liu (2022) develop a shift-tolerant perceptual metric that is robust to imperceptible misalignments between the reference and the distorted image. For it, they test various neural network elements and modify the architecture of the LPIPS metric rather than training it augmented data to handle the misalignment, making it more consistent with human perception. So far, the majority of prior research has focused on geometric distortions, while no study has systematically investigated the robustness of various similarity metrics to more advanced adversarial perturbations that are more perceptually indistinguishable. We seek to address this critical open question, whether perceplual similority metrics are robust against imperceptible adversarial perturbations. In our paper, we show that the metrics often overturn their similarity judgment after the addition of adversarial perturbations, unlike humans, to whom the perturbations are unnoticeable.
There exists a considerable body of literature ct al.
adversarial attacks (Szegedy et al.,
2014
Goodfellow
2015 Liu et al.] 2017 Papernot et al., 2016
Carlini & Wagner
2017
Xie et al.)
2018
Hosscini &
Poovendran 2018 Madry et al. 2018 Xiao et al., 2018 Brendel et al.
2018 Song et al.
2018:
Zhang et al.
2018a;
Engstrom et al.
2019;
laidlaw & Feizi 2019; Su et al.
2019; Wong et al. 2019 Bhattad et al.
2019 Xie et al.) 2019; Zeng et al. 2019 Dolatabadi et al. 2020 Tramer et al. 2020 Laidlaw et al.) 2020
Croce et al., 2020 Wu & Zhu 2020), but none of the previous investigations have ever considered attacking
perceptual similarity metrics, except for E-LPIPS (Kettunen et al.) 2019b) which only studies the LPPPS metric. This paper builds upon this line of rescarch and carcfully selects a set of represcntative attacking algorithrs to igvestigate the adversarial robustness of similarity metrics. We briefly describe these methods and how we employ them to attack similarity metrics in Section 3 In parallel, Lu et al. (2022) developed an adversarial attack for neural image assessment (NIMA) (Talebi & Milanfar 2018) to prevent misuse of high-quality images on the internet. NIMA is NR-IQA, while we systematically investigate several FR-IQA methods against various attacks.
Recent work underlines the importance of perceptual distance a bound for adversarial attacks (Laidlaw et al. 2020 Wang et al. 2021 Zhang et al. 2022). Laidlaw et al. (2020) developed a neural perceptual threat model (NPTM) that employs the perceptual similarity metric I.PPS(AAexNet)) as a bound for gen- erating adversarial examples and provided evidence that 4p-bounded and spatial attacks are near subsets of the NPTM. Similarly, Zhhang et al. (022) developed a perceptual threat model to attack no-reference IQA methods by constraining the perturbations via full-reference TQA, i.e., perceptual similarity metrics such as SSTM, LPIPS, and DISTS. They posit that the metries are "approximations to human perception of just-notieeable differences" (Zhang et al., 2022). therefore, can keep perturbations imperceptible. Moreover, Laidlaw et al. (2020) found LPIPS to correlate well with human opinion when evaluating adversarial exam- ples. However, it has not yet been established whether LPPPS and other perceptual similarity metrics are adversarially robust. We investigate this in our work, and the findings in our study indicate that all metrics, including LPIPS, are not robust, to various kinds of adversarial perturbations.
3 Method
Dataset. Our study uses the Berkeley- perceptual patch similarity (BAPPS) dataset, originally used to train a perceptual similarity metric (Zhang et al., 2018b). Each sample in this dataset contains a set of 3 images: distorted (lo and 11) and 1 reference (Tref). For perceptual similarity assessment, the scores were generated using a two-alternative forced choice (2AFC) test where the participants were asked, "which of two distortions is more similar to reference" (Zhang et al. 2018b). For the validation set, 5 responses per sample were collected. The final human judgment is the average of the responses. The types of distortions in this dataset are traditional, CNN-based, and distortions by real algorithus such as super resolution, frame interpolation, deblurring, and colorization. Human opinions could be divided, i.e., all responses in a sample may not have voted for the same distorted image. In our study, to ensure that the two distorted images in the sample have enough disparity between them, we only select those samples where humans unanimously voted for one of the distorted images. In total, there are 12,227 such samples.
4

---

Published in Transactions on Machine Learning Research (05/2023)
It is non-trivial to compare metrics based norm- comd- constraint simply because change of 10% in metric A's score is not equal to 10% change in metrie B's score. But how does one calculate the fooling rate that measures the susceptibility of similarity metric? A straightforward method is to compare all metrics against human perceptual judgment. The 2AFC test gathers human judgment on which of the two distorled images is more similar to the reference. Using this knowledge, we can benchmark various metrics and test whether their accuracy drops or, i.c. if they flip their judgment when attacked. To make a fair challenge, we only use samples where human opinion completely prefers one distorted image over the other.
Attack Models. As observed in Figure | the addition of adversarial perturbations can lead to rank flip, We make use of existing attack methods such as FGSM (Goodfellow et al. 2015 , PGD (Madry et al. 2018) One-pixel attack (Su et al.) 2019), and spatial attack stAdv (Xiao et al., samples. These attack methods were originally devised to fool image classification models, therefore, we introduce minor modifications the distorted images, Iu or I, that more similar to Iref to attack. The distorted image being attacked is T Iprey, and the other image is Tother; accordingly, for the sample in Figure [ /1 is Iprey and To is Tother- Consider s; as the similarity score between 1, and [ Before the attack, the original rank is Sother > Sprey: but after the attack Iprey turns into Jadv: and when the rank flips, Sadv Sother In image classification, a misclassification is used to measure the attack's success, while for perceptual similarity metrics, an attack is successful when the rank flips.
2018) to generate such adversarial
their procodures to attack perceptual similarity metrics. We select one of
Fast Gradient Sign Method. FGSM a popular white-box attack introduced by Good- fellow et al. 2015). This attack method projects the imput image onto the boundary of an sized fx ball, and therefore, restricts the perturbations to the locality of I. We fol- low this mothod to generate imperceptible per- turbations by constraining e to be small for our experiments. This attack starts by first com- puting the gradient with respect to the loss function of the image classifier being attacked. The signed value of this gradient. multiplied by e generates the perturbation, and thus, Jade == I c. (V I, target)), where 0 are the model parameters. We adopt this method attack perceptual similarity met- rics. We formulate a new loss function for an untargeted attack as:
1
E (Q. bro)))
ladv
+
LPIPS: 0.0445
E = 0.024
LPIPS: 0.1149
Figure 2: FGSM attack on LPIPS(Alex). In this white-bot attack, we use the LPIPS network parameters to compute the signed gradient. With increase in c, the severity of the at increases. In this example, the adversarial perturba- tions are hardly visible. The RMSE between the prey image I and the adversarial image Iadv is 3.53.
J(0, preyy, Tother, Tref) (
Sother
(1)
Sother + Sprey
We maximize this loss, i.e,, move in the opposite direction of the optimization by adding the perturbation to the image. The human score of all the samples in our selected dataset is either 0 or 1, unanimous vote. Hence, we can easily employ the loss function in Equation [ because if the metric predicts the rank correctly then (oother/(Sother + Sprey)) would be 1. Afterwards, if the attack is successful then (Sother/ (Soterrr + Sudv)) becomes less than 0.5, causing the rank to flip. Algorithm 3 (refer Appendix
B provides the details for the FGSM attack. First, Iprey is selected based on the original rank. The model parameters remain constant, and we compute the gradients with respect to the input image Iprey- To increase perturbations in normalized images, we increase the in steps of 0.0001 starting from 0.0001. When e is large enough, the rank flips. It would mcan that the attack was successful (sce example in Figure 2). If the final value of e is small then the perturbation imperceptible, making it hard to discern any dillerence between the original image and its adversarial sample.
Projected Gradient Descent. PGD attack by Madry et al. 20018) takes a similar approach to FGSM, but instead of a single large step like in FGSM, PGD takes multiple small steps for generating perturbation
smaller 8; means 1 is more similar Tref
5

---

Published in Transactions on Machine Learning Research (05/2023)
4
lavv
+
LPIP5:0.045
LPIP5:0.1465
Figure 3: PGD attack on (PIPS(Alex) In this white-box attack, we use the LPIPS network parameters to compute the signed gradient. With increase in the number of attack itcrations, the severity of the attack increases. In this example, perturbation ladn are not visible. The RMSE between the prey image I and the adversarial image Lodu is 2.10.
8. Hence, the projection of I stays either inside or on the boundary of the e-ball. This multistep attack is defined as:
odu (o sign( V rea Lothers Fres))
(2)
where J is the loss defined in Equation The perturbation each pixel is bounded to prede- fined range using the projection constraint Pa. We implement P. using clip operation on the final perturbation 5 (Line 14 Algorithm []. As shown in Algorithm [ the signed gradient is multiplied with step size a, and this adversarial perturbation is added Lade- The final perturbation 8 is the dif- ference between Iedv and Iprey, and in our method, 6 is bounded by fxx norm. Hence, the PGD attack is an lo attack.
Algorithm 1: PGD attack on Similarity Metrics
Input: lu. J1, Iref, metric J, (perturbation limit 0.03
}, mar iterations (30), a (step size 0.001)
Output: attack_success success True on rank flip 80 f(Tnef Jo):
[Trref: [1); rank int(so > 81); // If To is more similar to Tref then rank is 0 else if rank = then Iprey = I1; fother = so;
else Iprey = lo: Bother 81;
= zeros like([ppre)) // perturbation
while max iterations do
Ladv clip(1prey + 6, min 1, max 10
Sade = f(Treft lade)
11
if Sadu > Sather then return True // Attack successful
One-pixel Attack. The previous two approaches are white-box allakss. We now use a black-boo attack, the One-pixel attack by (Su et al. (2019
12
J = ((Nather/ (Bother + Sadw)) -1) //
Loss
signed grad = sign Vigg(Vinnv' (Vina Ied = Tndto + a = signed grad
18
14
that perturbs only a single pixel using differential evolution (Storn& & Price. 1997p.
15 ô = clip( ('ado Ipreys min -E, maz +e)
18
The objective of the One-pixel attack is defined as:
17 return Palse // Attack unsuccessful
maximize e(Uprez))
f(Iprey (ref)
(3)
subject to [le(Tprey)llo
where f is the similarity metric, and the vector e(Tprey) is the additive adversarial perturbation, and d is for the One- attack. This algorithm aims to find mutation to one particular pixel such that similarity metric f, such LPIPS, will consider Iprey is less similar to Iref than it is originally, and thus, the rank is flipped. Note, for LPIPS, a larger score indicates the two images being less similar. Please refer to Su et al. (2099) for more details of this attack algorithm. For attack example, please refer to Figure in Appendix C
Spatial Attack Adv.. The goal of the stAdy attack to deform the image geometrically by displacing pixels (Xiao et al. 2018). generates adversarial perturbations in the spatial domain rather than directly manipulating pixel intensity values. This attack synthesizes the spatially distorted adversarial image (Jadv) via optimizing flow vector and backward warping with the input image (Eprey) using differentiable bilinear interpolation (Jaderberg el. al.] 2015). For each sample, we start with flow initialized with zeros and then optimize it using L-BFGS (Liu & Nocedal 1989) for the following loss.
L ahrank + fL
6

---

Published in Transactions on Machine Learning Research (05/2023)
Flow
lav
LPIP5:0.0445
LPIPS: 0.1465
Backward warping via Bilinear Interpolation
Figure 4: Spatial attack stv on LPII'S(AleXNed We attack LPIPS (PIeX(()))Nee) to create adversarial images. This attack optimizes flow vector to create perturbations in the spatial domain. In this example, flow distorts the structure of the horse to generate the adversarial image. The RMSE between the prey image 7 and the adversarial image Ladv is 2.50.
pizels neighbors(p)
flow
Σ
Σ
V(up uny (Up U¹)
(5)
P
where (u, is the displacement vector at pixel location p and its neighbors q.
Sother
Crank
(6)
Sother + Sadv
where a is 50 and 3 is 0.05.
As we minimize Crank, the pertur- bations in Iado will increase, and
A Algorithm 2: stAdy attack on LPIPS
Input: To, J1, Tref, LPIPS J, max _iterrrttrtions (250)
Output: atfack_ atfack_success success True rank flip
thus rank will flip. Simultaneously, also minimize Lilow which de-
Function stAdv stAdv_sttack sttack(flow,
Iprey, fnef- bother):
lade saap(flow,Iprey // Backwarp via bilinear interpolation
fines the amount of perturbations generated by flow to distort the image. It enforces the pert urba- tions to be constrained to make as
fudo f( Tado)
Crank: cale loss(Lref IpreysAader Sother:
L= Lrank Lperturb
gradient VVooo
if Badu > Bother then return 0, gradient, flow Attack successful
little change to the attacked im-
8 else return L, gradient, flow Attack unsuccessful
age Lprey as possible. Xiao et al.
1 f(lvef, lo); =1 =1
(2018) performed user study to
2 rank = int(ee >> =1) // In is more similar to Iref then rank is 0 else
test the perceptual quality of the images having perturbations gen- erated by the stAdy attack and found them to be indistinguishable by humans. By visual inspection, we found the adversarial perturba- tions on the images imperceptible in our studies well.
a if rank = then Iprey = I1; 6other #0; else Iprey la;
// Initialize flow vector with zeros flow = seros like(2 Iprey height + Lprey width) converge, grad, flow L-BFGS(func-stAdd
attack, args=(flow, f, Ipreys Tref Sather), iterations=mar, _isemmatttterross) // Optimize flow vector if converge then attack_ True else attack _dtack_success False attack. success stcces_succes
4 Experiments and Results
We experiment with wide variety of similarity metrics including both traditional ones, such L2, SSIM (Wang al.) 2004), MS-SSIM (Wang al. 2003) CW-SSIM (Wang & Simoncelli 2005) and FSIMc (Zhang et al. 2011). and the recent deep lcarning based ones, such as WaDIQaM- el al. 2018). GTI-CNN (a et. al. 2018). LPIPS (Zhang el. al. 2018b), E-LPIPS (Kettunen et. al. 2019b), (Coole et al.) 2020), PIM (Bhardwaj et al. 2020), A-DISTS DDng ct al. 2021). ST-L.PIPS (Ghildyal & Liu 2022), and Swin-IQA (Liu al.) (2022). We adopt the BAPPS val- idation dataset (Zhang et al. 2018b) for our experiments. Following Zhang et al. 2018b) we scale the image
[Bosso
DISTS Ding et al. 2020), Watson- DFT
7

---

Published in Transactions on Machine Learning Research (05/2023)
patches from size 256 256 to 64 x 64. As mentioned in Section B we believe that the predicted rank by a metric will be easy to flip on samples close to the decision boundary; therefore, we take a subset of the samples in the dataset which have a clear winner, i.c., all human responses indicated that one was distinctly better than the other. Now, in our dataset, we have 12,227 samples. We report the accuracy of metrics on the subset of selected samples and compare it with their Two-alternative forced choice (2AFC) scores on the complete BAPPS validation dataset. As shown in Table [
Table 1: Accuracy on the subset selected for our experiments correlates with the 2AFC score computed on the complete BAPPS validation dataset.
2AFC (%) on complete BAPPS (36344 samples)
Accuracy (%) ou subsct BAPPS (12227 samples)
Network
1.2
63.2
79.7
SSIM (Wang et al. 2004 WaDIQuM-FR [Bosee
63.1
80.8
66.5
83.3
LPIPS(Alex)
Zhang et aL
20188
69.8
92.4
LPIPS(YGG) iZhang ef AL.
20186
68.1
89.8
DISTS DDin el al. 2020
68.9
91.3
all these metrics consistently correlated better with the human opinions on the subsct of BAPPS than on the full dataset, which is expected as we removed the ambiguous cases.
In this section, we first show that similarity metrics susceptible to both white-box and black-box attacks. Based on this premise, we hypothesize that these similarity metrics are vulnerable to transferable attacks. To prove this, we attack the widely adopted LPIPS using the spatial attack stAdv to create adversarial examples and use them to benchmark the adversarial robustness of these similarity metrics. Furthermore, we add a few iterations of the PGD attack, hence combining our spatial attack with -b-uuude perturbations, to enhance transfcrability to other perceptual similarity metrics.
4.1 Adversarial Attack on Perceptual Similarity Metrics
Through the following study, we lest our hypothesis that similarity metrics are susceptible to adversarial attacks. We first determine whether it is possible to create imperceptible adversarial perturbations that can overturn the perceptual similarity judgment, i.e. flip the rank of the images in the sample. We try to achieve this by simply attacking with widely used white-box attacks like FGSM, and PGD, and a black-box attack like the One-pixel attack. As reported in Table 2 all these attacks can successfully flip the rank assigned by both traditional metrics such as L2, and SSTM (Wang et al. 2004), and learned metrics such as WaDIQaM-FR (Bosse et al. 2018), LPIPS (Zhang et al. 2018b), and DISTS (Ding et al., 2020 in a significant amount of samples.
For the PGD attack, the maximum foom perturbation cannot be more than 0.03 as the step size is 0.001, and the maximum attack iterations is 30. We chose 30 after visually inspecting for the imperceptibilitty of perturbations on the generated adversarial samples. With the same threshold, the FGSM attack would not be as successful as PGD, which we show Appendix E. Therefore, to report the results of the FGSM attack, based on empirical evaluation, we select the maximum e as 0.05. We present the results separately for samples where the originally predicted rank by the metrie matches the rank provided by humans. Now, focusing only on the samples where the metric matches with the ranking by humans, we found L2 and DISTS to be the most robust against FGSM and PGD with only about 30% of the samples flipped, while LPIPS and WadIQaM-FR were the least robust, with about 80% of the samples flipped. The same conclusion can also be reached by observing 6 (or perturbations) required to attack them. Next, despite being a black-box attack, the One-pixel attack can also successful flip ranks. IPIPS(AAexMN) One-pixel attack with 82% of the samples flipped, and this lack of adversarial robustness is consistent across all three attacks. SSIM and WadIQaM-FR are more robust to this attack, with only 18% and 31% samples flipped. It is interesting to note that similar results are achievable by using just the score of the adversarial image, i.e., Sade as loss for optimization.
has the least robustness to the
Not surprisingly, it is easier to flip rank for the samples where the metric does not match with human opinion. As reported in Table g a much higher number of thase samples flip where the rank by metric and humans did not match. These samples have a lower e, which means that lesser perturbations were required to flip the rank. We attribute the easy rank flippiig for these samples to the fact that the distorted images in each sample, i.c., lother and Ipreys are much closer to the decision boundary for the rank flip.
2 All e (or pertarbation) in this paper were computed from normalized images the range [-1,1]-
8

---

Published in Transactions on Machine Learning Research (05/2023)
Table 2: FGSM, PGD, and One-pixel attack results. Larger allows more perturbations, and lower RMSE relates to higher imperceptibility.
Same Rank by Human Metric
FGSM (t < 0.05)
PGD
Oue-pixel
Network
Total Samples
#Samples Flipped
Mean
RMSE
#Samples Flipped
% pixels with e
RMSE
# #Samples
>0.001
>0.01
>0.03
Plipped
9750
3759 /399
0.023
2.9
1.7
2318/219
81.4
56.1
0.0
1.9
1.0
4225/13%
L2
2477
1550/63%
0.017
2.2
1.6
1202/49%
82.0
42.7
0.0
1.5
1412/57%
SSIM
9883
6922/70%
0.018
2.5
1.7
6297/54%
94.6
53.6
0.0
1.8
1.0
1787/18%
Wang et sL. WadIQaM-FR
2004
2314
2013/86%
0.011
1.6
1.3
1843/79%
87.3
32.0
0.0
1.3
0.8
1005/ 13%
T0191
8841
0.008
1.0
1.0
T0176/ 100%
69.2
4.3
0.0
0.7
0.3
3130/31%
Bosse et al.
2018
x
2036
2012/100%
0.001
0.6
0.3
1005
41.2
0.1
0.0
0.5
0.1
1598/79%
Alex
T1303
7247/64%
000018
2.4
1.7
8806/78%
86.8
28.7
0.0
1.3
0.6
9255/82%
Zhaug et al.
20186
924
912/99%
0.004
0.9
0.7
917
59.5
3.2
0.0
0.8
0.3
921
/PIPS/VGG)
T0976
8434/77%
0.012
1.7 1.5
/888
81.6
15.6
0.0
1.0
0.5
7212/60%
Zhang et sl.
20186
1251
1244/100%
0.003
0.8
0.5
1246/10022
52.3
0.0
0.7
1219/98%
DISTS
11158
3043/27%
0.025
3.3
1.8
2306/21%
97.0
75.4
0.0
2.6
1.3
7416/67%
Ding et al.
2020
1009
795/74%
0.016
2.2
1.7
723/08%
91.9
50.0
0.0
2.0
1.3
1033/97%
To test this, we calculate the absolute difference between Sother and Sprey, i.e,, the perceptual distances of Tather and prey from Iref. As reported in Table 3 the similarity difference for these samples is much lesser than samples where the rank predicted by metric is the same as the rank assigned by humans. This result indicates that the samples where rank predicted by met-
Comparing samples where the rank by metric was the same as assigned by humans versus samples where it was not.
Table 3:
Same Rank by Similarity Diff.
Network
Human & Metric abs(so 81)
L2
0.036
ric is not the samne the rank assigned by humans lie closer to the decision boundary, causing them to flip easier
0.025
SSIM
0.114
Wang
2004
0.054
Imperceptibility. We discuss the imperoeptibility of the ad- versarial perturbations by comparing the root mean square er-
Wad|Qu
0.231
Bossc ct al.
2018
x
0,064
ror (RMSF between the original and the perturbed image.
LPIPS A
lex
0,169
Zhang et ml.
2018b
0,024
As expected, the PGD attack stronger than FGSM as it is
LPIPS
VG))
0.174
capable of flipping significant number of samples with lesser
Zhang et
2018b
0.037
DISTS
0.103
adversarial perturbations. In Appendix E. we experiment with increasing step size a for the PGD attack, which further in- creases its severity.
Ding et al.
2020
0,022
Table 4: Comparing PSNR of adversarial images gen- erated via FGSM vs. PGD. The e for the adversarial images generated via FGSM is < 0.05. A higher mean PSNR of the PGD examples shows that the adversarial perturbations are less perceptible.
As reported in Table 2 for the PGD allack, a good portion of the adversarial image (Jade) has 0.01, while for FGSM, the amount of pixel per- turbation all over the image is constant value which moreover is higher for a successful attack. Thus, on average, the lode generated via PGD has lower RMSE and a higher PSNR (see Table with the original image Iprey: compared to the Lodo gen-
Same Rank by Human & Metric
FGSM PSNR
PGD
Network
PSNR
P
L2
40.81
6.49
44.15
5.49
crated via FGSM. We also perform a visual sanity check and find the perturbations satisfactorily im- perceptible. Only a single pixel is perturbed for
43.75
7.00
46.08
5.70
SSIM
12.51
6.55
11.60
5.31
Wang
2004
46.39
6.09
47.19
WadIQaM-
60.8T
5.00
52.15
3.47
lado generated via the One-pixel attack, which we consider suitably imperceptible.
Bosse al.
2018
53.92
3.25
54.35
2.73
LPIPS/AN
Alex
42.80
6.70
16.82
1.09
2 Zhang et al.
2018b
49.98
4.19
50.80
3.14
LPIPS/VG. VGG)
45.96
6.38
48.68
3.72
4.2 Transferable Adversarial Attack
Zhang et al.
2018b
50.56
3.27
51.09
2.16
DISTS
39.50
6.22
41.19
5.75
In real-world scenario, the attacker may not have access to the metric's architecture, ayper-parameters, data, or outputs. In such scenario, practical solution for the attacker is to transfer adversarial examples crafted on a source metric to a target perceptual similarity metric. Previous studics have suggested reliable approaches for creating such black-bor transfcrable
Ding al
2020
43.64
6.95
44.41
6.39
Thhoogghou this paper, RMSE was calculated on images with pixel values ranging [0.255].
9

---

Published in Transactions on Machine Learning Research (05/2023)
Table 5: Transferable adversarial attacks on perceptual similarity metrics. The adversarial examples were generated by attacking LPIPS((leXNet) via stAdv. In total, there are 2726 samples. Next, we attacked LPIPS(AlexNet) using PGD(10). Then, we combined StAdv+PGD(10) by perturbing the stAdv generated images with (1D( Accurate samples are the ones for which the predicted rank by metric is equal to the rank assigned by humans. The transferability increases when the attacks are combined.
# Accurate Samples Plipped
# Accurate Samples
Network
PGD(10)
(2D()0)
stAdv
stAdv + PSD(5)
stAdv + PGD(10)
stAdv + PG)(
stAdv +
(2D(20)
L2
2099/77% 77%%
101/5%
174/8%
77/4%
134/6%
189/9%
257/12% 200/10%
SSIM Wang et al. 2004
2093/77%
237 11%
442/21%
78/4%
180/9%
339/16%
370/18%
540/26%
MS-SSTM Wang of all 2003
2022 774%
158/8%
256/ 13%
76/4%
162/8%
224/11%
234/ 12%
333/10%
CWSSIM Wang & Sumoneelli
2006
1883/697
101/5%
172/9%
42/2%
60/3%
128/7%
139/7%
193/10%
FSIMe (Zhang et al. 2011
2025/743
222/11%
325/ 16%
202/10%
233/12%
302/15%
310/15%
393 393/19%
WaDIQuNITI Booest et
2018
2083/76%
95/5%
186/9%
59/3%
85/4%
146/7%
156/7%
238 238
GTI-CNN (Ma et al 2018
1916 1916/71% 9
148 148/23%
480 /25%
194/25%
488 488/25%
501/26%
510/26%
5 541/28%
LPIPS iZhang al.
2018b
2503/92%
298/12%
656/ 6566/26%
114/5%
221/9%
519/21%
555/22%
886/3 (5%%
LVCC)(VCC) (Zhang ob al.
201186
2317 2317/85%
435/19%
814/ 414/35%
131/6%
285 288/12%
643/28%
685/30%
992 /43%
E-LPIPS (Kettonen of al. 20196
2442/ 990%
503/21%
643/ 20%
/17/21%
552/ 23%
641/26%
656/27%
817/337
DISTS [Ding cl al: (80M0
2113 11%
311/13%
376/219
146/6%
257
510/21%
516/23%
801 /33%
Watson-DFT Czolbe et al.
2020
2179/80%
387/18%
614/28%
216/10%
324/15%
532/24%
562/26%
750/34%
PIM-1 Bhardwa
20120)
2468/91% %
696/28%
814 414/33%
756/31%
772/31%
626/33%
852/35%
958/39 39%
PIM-5 Biardwaj et a..
2020
2457 /90%
751/31%
844/34%
765/31%
791/32%
864/35%
893/36%
963/ /39%
A-DISTS Ding et al. 2021
2346/869
339/14%
661 /28%
164/7%
276/12%
561/24%
590/25%
850/369
ST-LPIPS Xlex)
Ghildyal & Liu
2022
2470/919 %
104/4%
198/8%
96/4%
123/5%
205/8%
212/0%
310/ 310/13%
ST-LPIPS/VGG) (Ghldyal & LTu
2022
2193/91%
210/8%
453/18%
103/1%
153/6%
321/13%
300/ 14%
376/ 23%
SwinlQA Liu 2022
2310/85%%
249/11%
357/15%
262/11%
279/12%
312/15%
375/16%
482/21% 21%
adversarial examples for image classifiers (Tramèr et al. 2017 Zhou et al. 2018 Inkawhich et al. 2019 Huang
ct al. 2019 Li al., 2020 Hong et al.] (021). This paper focuses on perceptual similarity metrics and how they perform against such transferable adversarial examples. Specifically, we transfer the stAdv allack on LPIPS(AlexNet)) to other metrics. We chose LPIPS(AlexNet)) as it is widely adopted in many computer vision, graphics, and image video processing applications. Furthermore, we combine the stAdy attack with PGD to increase the transferability of the adversarial examples to other metries. In this study, we only consider samples for which the metrics and the human opinions agree on their rankings.
stAdv. As shown in Figure | stAdy has the capability of attacking high-level image features. As a white- box attack on LPIPS(AlexNet), out of the 11,303 accurate samples from total 12,227 samples, stAdv was able to flip judgment on 4658 samples with a mean RMSE. of 2.37 with standard deviation 1.42. Because we need high imperceptibility, we remove samples with RMSE > 3 and are left with 3327 samples. We then perform a visual sanity check and remove some more with ambiguity, keeping only strictly imperceptible samples. In the end, we have 2726 samples, with a mean RMSE of 1.58 with standard deviation 0.63, which we transfer to other metrics as a black-box attack. As reported in Table 5 all metrics are prone to the attack. WaDIQuM-FR (Bosse el al. 2018 is most robust, while PIM (Bhardwaj et al, 2020 that was found robust to small imperceptible shifts is highly susceptible to this attack, although PTM is 15% more accurate than WaDIQaM-FR. DISTS, ST-LPIPS, and Swin-IQA have similar high accuracy as PIM but better robustness. Finally, we saw that, on average, learned metrics are more correlated with human opinions, but traditional metrics exhibit more robustness to the imperceptible transferable Adv adversarial perturbations.
PGD(10). We now attack the original 2726 selected samples with the PGD attack. As shown in Section 4.1 perturbations generated via PGD have low perceptibility; hence, we create adversarial samples using PGD. In stAd,, we stopped the attack when the rank predicted by (Alex(e)))Net) flipped. While in PGD, for comparison's 's sake, we fix the number of attack itcrations to 10 for cach sample to guarantee the transfcrability of perturbations. We call this transferable attack PGD(10), and the mean RMSE of the adversarial images generated is 1.28 with standard deviation of 0.11. The metrics SSLM and WaDIQaM-FR, are most robust to the transferable PGD(10) attack, as reported in Table 5
Combining stAdy and PGD( The allacks stAdv and PGD are orthogonal approaches as PGD boundddd attack) manipulates the intensity of individual pixels while stAdy (spatial attack) manipulates the location of the pixels. We now combine the two by attacking the samples generated via stAdv with PGD(10). The mean RMSE of the generated adversarial images is 2.19 with a standard deviation of 0.41, just. 0.61 higher than images generated via stAdy. As reported in Table 5. the increase in severity of the
10

---

Published in Transactions on Machine Learning Research (05/2023)
adversarial perturbations in stAdv+PGD(10)) leads to increased transferability. This result also is consistent with previous findings by Engstrom et al. (009)) where they combined PGD on top of their spatial attack and found that it leads to an additive increment in the misclassification rate.
Summary. this paper, we successfully demonstrate that a wide variety of perceptual similarity metrics are susceptible to adversar- ial attacks. We show that adversarial pertur- bations crafted for LPIPS(AlexNer) generated via st.Adv, can be transferred to other met- rics. Furthermore, combining stAdv (spatial attack) with PGD -boundddd attack) in- creases their transferability. We showcase a few examples in Figure @ and Figure 7 In ad- dition, the severity of the allack increases with the increasing number of PGD iterations (see Table Our investigations also show that although more accurate, learned metrics may not be more robust than traditional ones (see Figure Further tests carried out on two ad- ditional datasets and higher resolution images, in Appendix D corroborate with our previous results. We demonstrate the reverse of our attack in Ap- pendix i.c., we attack the less similar of the two distorted images to make it more similar to the reference image. In summary, our findings point towards the need to develop robust perceptual similarity metrics.
Learned
Samples
Traditional metrics
Accurate Samples Flipped
2500
2000
* Samples
1500
1000
22%
500
8%
166%
5%
13%
-1 12%
0
PGD(10)
StAdv
StAdV+PGD(10)
Figure Comparing traditional metrics (L.2, SSIM, MS-SSIM, CW-SSIM, and FSIMc) versus learned met- rics (WaDIQaM-FR, GTI-CNN, LPIPS, DISTS, E-LPIPS, Watson-DFT, PIM, A-DISTS, ST-LPIPS, and Swin-IQA).
5 Broader Impacts Statement
Perceptual similarity metrics have wide variety of applications. Hence, there are benefits to studying the robustness of these metries, and this work presents an opportunity to further improve the alignment of these metrics with human perception. At the same time, it important to consider the negative outcomes of our work. Exposing the vulnerability of these metries provides more details to malicious actors who would want to misuse this information to attack applications that make use of these similarity metrics in their pipeline, such as evading copyright. detection. Perceptual similarity metrics synthesize malware images that could go undetected online. Therefore, suggest further research on this topic to include appropriate defenses or more discussion ways for mitigating such vulnerabilitics. To aid further research on this topic, we shall make our code and data publicly available.
also be misused to
6 Conclusion
In this paper, we studied the robustness of various traditional and learned perceptual similarity metrics to imperceptible perturbations. We devised a methodology to craft such perturbations via adversarial attacks. Our findings suggest that, when comparing two images with respect a reference, the addition of imper- ceptible distortions can overturn a metric's similarity judgment. The results of our study indicate that even learned perceptual metrics that match with human similarity judgments are susceptible to such impercep- tible adversarial perturbations. We crafted adversarial examples using the spatial attack, stAdy, that were transferable to other metrics. We show that when combined with the PGD attack, the transfcrabilitty of the adversarial examples can be further increased. Perceptual similarity metrics are designed to simulate the human visual system, and for this reason, these metrics are increasingly used in the assessment of image and vidco quality in real-world scenarios. Since invisible distortions can nogatively impact the performance of similarity metrics, future studies for the design and development of newer metries should also focus on validating robustness.
11

---

Published in Transactions on Machine Learning Research (05/2023)
Sef
latter
Sprey
Jdd PGD(10)
1att StAdv
1 +
12
0.0091
0,0127
0.0128
0.0128
0.0128
SSIM
0.8754
0.8823
0.8721
0.8770
0.8635
PSIMc
WaDIQuM-FR
1.2747
1.3567
1.3730
1,3622
1.3572
GTI
135.61
255.97
220.48
217.10
217.65
DISTS 1
0.0996
0.0729
0.0952
0.0873
0.1152
LPIPS(Squeezz))
0.0736
0,0393
0.0421
0,0490
0.0517
LPIPS(VGG)
0.0916
0.0669
0.0802
0.0783
0.1011
E-LPIPS
0.0057
0.0041
0.0069
0.0068
0.0075
Watson-DFT
908.63
922.66
1112.21
1071.77
1136.02
1.2937
PIM-5
6.2894
5.0282
11.3717
12.0675
12.2006
lre
Latter
lorcy
1a PGD(10)
Jady stAdv
Jadr PGA(1+
L2 4
0.0361
0,0050
0.0057
0.0056
0.0063
SSIM 1
0.3163
0,5807
0.5528
0.5646
0.5357
PSIMc
0.98102
0,98274
0.98079
0.98016
0.97770
WaDIQuM-FRA
1.3614
1.2760
1,2575
1.2983
1,2943
CTI-CNN
133.18
59.11
77.51
78.95
85.07
DISTS 1
0.2772
0.2324
0,2739
0,2678
0.3021
0.0986
0,0761
0.1231
0.1058
0.1762
LPIPS(VOO)
0,2451
0.3269
E-LPIPS
0.0115
0.0103
0.0169
0.0170
0.0178
Watson-DFT
2433.66
1314.98
1415.91
1392.29
1410.53
PIM-1
2.9635
2.5469
3.2072
3.2161
3,5531
PIM-5
35.6628
37.6837
39.1791
Latw
Sprey
Law PGD(10)
1as stAdv
1a StAdv=PGD(10)
L2
0.0010
0.0010
0.0012
0.0012
0.0015
SSIM
0.9739
0.9779
0.9730
0.9743
0.9681
PSIMc
0.99992
0.99985
0.99983
0.99983
0.99980
WaDIQuM-FR
1.1214
1.1190
1.1177
1.1184
GTI-CNN &
47.72
11.53
79.21
85.79
84.42
DISTS 1
0.1180
0.0065
0.0200
0.0129
0.0283
(PIUe(Qqe
0.0023
0.0013
0.0025
0.0017
0.0033
LPIPS(VGG)
0.0069
0.0103
E.PPPSS
0.0139
0.0002
0.0045
0.0047
0.0052
Walson-DFT
924.09
541.48
783.71
693.21
861.64
PIM
0.7539
0.0110
1.0787
1.1750
1.1291
7.0737
11.2964
11.7169
Figure Transferable attack on perceptual similarity metries. In example (Top), the RMSE between
Iprey and Iade images (left to right) is 1.26, 2.89, and 2.47. In example 2 (Mid.), the RMSE between I Iprey and Tado images (left to right) is 1.29, 1.02, and 1.91. In example 3 (Bot.), RMSE between Iprey and I Tado images (left to right) is 1.43, 1.2, and 2.15. Please refer to Figure 7 in Appendix A for more examples. Text in red indicates that the rank has flipped.
12

---

Published in Transactions on Machine Learning Research (05/2023)
References
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Cireunventing defenses to adversarial examples. In International Conference Machine Learning, vol- ume 80, pp. 274-283, 2018.
Sanguie Bhardwaj, lan Fischer, Johannes Ballé, and Troy Chinen. An unsupervised information-theoreee percoptual quality metric. In Advances in Neural Information Processing Systems 93, 2020.
Anand Bhattad, Min Jin Chong, Kaizhao Liang, Bo Li, and DA Forsyth. Unrestricted adversarial examples via semantic manipulation. In International Conference on Learning Representations, 2019.
Sebastian Bosse, Dominique Maniry, Klaus-Robert Müller, Thomas Wiegand, and Wojeiech Samek. Deep neural networks for no-reference and full-reference image quality assessment. IEEE Transactions on Image Processing, 27(1):206 219, 2018.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. In International Conference Learning Representations, 2018.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Securily and Privacy, pp- 39-5 57, 2017.
CLIC. Workshop and challenge on learned image compression, 2022. URL httt:://ww..ooppresstt cc/
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Pratock Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. orXiv/2010.00
2020.
Steffen Czolbe, Oswin Krause, Ingemar Cox, and Christian Tgel. loss function for generative neural networks based watson's perceptual model. In Advances in Neural Information Processing Systems, pp. 2051 2061, 2020.
Keyan Ding, Kede Ma, Shiqi Wang, and Eero P. Simoncelli. Image quality assessment: Unilying structure and texture similarity. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2020.
Keyan Ding, Yi Liu, Xueyi Zou, Shiqi Wang, and Kede Ma. Locally adaptive structure and texture similarity for image quality assessment. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 2483 2491, 2021.
Hadi Mohaghegh Dolatabadi, Sarah Erfani, and Christopher Leckie. Advflow: Inconspicuous black-box adversarial attacks using normalizing flows. In Advances in Neural Information Processing Systems, 2020.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 9185-9193, 2018.
Alexcy Dosovitskiy and Thomas Brox Generating images with perceptual similarity metrics based on deep networks. In Advances in Neural Information Processing Systems, pp. 658 666, 2016.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring the landscape of spatial robustness. In International Conference on Machine Learning, pp. 1802-1811, 2019.
Abhijay Ghildyal and Feng Liu. Shift-tolerant perceptual similarity metric. In European Conference on Computer Vision, 2022.
Ian Goodfellow, Jonathon Shlens, and Christian Szegody. Explaining and harnessing adversarial examples.
Tn International Conference on Learning Representations, 2015.
13

---

Published in Transactions on Machine Learning Research (05/2023)
Jinjin Gu, Haoming Cai, Haoyu Chen, Xiaoxing Ye, Jimmy S. Ren, and Chao Dong. Pipal: A large-scale image quality assessment dataset for perceptual image restoration. In European Conference Compuler Vision, volume 12356, pp. 633 651, 2020.
Sanghyun Hong, Yigitcan Kaya, Tonut-Vlad Modoranu, and Tudor Dumitras. A panda? no, it's sloth: Slowdown attacks adaptive multi-cxit neural network inference. In International Conference Learn- ing Represeniations, 2021.
Hosscin Hosseini and Radha Poovendran. Semantic adversarial examples. In IEEE Conference on Computer Vision and Pullern Recognition Workshops, pp. 1614-1619, 2018.
Qian Huang, Isay Katsman, Horace He, Zeqi Gu, Serge Belongie, and Ser-Nam Lim. Enhancing adversarial example transferability with an intermediate level attack. In IEEE International Conference on Computer Vision, pp. 4733 4742, 2019.
Nathan Inkawhich, Wei Wen, Hai Helen Li, and Yiran Chen. Feature space perturbations yield more transferable adversarial examples. In IEEE Conference on Compuler Vision and Paltern Recognition, pp. 7066-7074, 2019.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, al. Spatial transformer networks. In Advances in neural information processing systems, pp. 2017-2025, 2015.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In European Conference on Computer Vision, volume 9906, pp. 694-711, 2016.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 8110-8119, 2020.
Markus Kettunen, Erik Härkönen, and Jaakko Lehtinen. Deep convolutional reconstruction for gradient- domain rendering. ACM Transactions on Graphics, 38, 2019a.
Markus Kettunen, Erik Härkönen, and Jaakko Lehtinen. E-lpips: Robust perceptual image similarity via random transformation ensembles. arXis/1906.03997 2019b.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples the physical world. International Conference on Learning Representations Workshop, 2017.
Cassidy Laidlaw and Soheil Feizi. Functional adversarial attacks. In Advances in Neural Information Pro- cessing Systems, 2019.
Cassidy Laidlaw, Sahil Singla, and Soheil Feizi. Perceptual adversarial robustness: Defense against unseen threat models. In International Conference on Learning Representations, 2020.
Christian Ledig, Lucas Theis, Ferene Huszar, Jose Caballero, Andrew Aitken Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using generative adversarial network. arXio/ (xkie/660.000 04802, 2016.
Qizhang Li, Yiwen Guo, and Hao Chen. Yet another intermediate-leve attack. In European Conference on Computer Vision, pp. 241-257, 2020.
Dong C Liu and Jorge Nocedal, On the limited memory bfgs method for large scale optimization. Mothe- matical programming, 45(1):503-585 528, 1989.
Jianzhao Liu, Xin Li, Yanding Peng, Tao Yu, and Zhibo Chen. Swiniqa: Learned swin distance for compressed image quality assessment. In IEEE Conference on Computer Vision and Pattern Recognition. Workshops, pp. 1795-1799, 2022.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attak.s In International Conference on Learning Representations 2017.
14

---

Published in Transactions on Machine Learning Research (05/2023)
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE International Conference Computer Vision, 2021.
Ning Lu, Li Dong, Diqun Yan, and Xianliang Jiang. On attacking deep image quality evaluator via spatial transform. In IEEE International Conference on Systems, Mon, and Cybernetics, pp. 2876 2881, 2022.
Kede Ma, Zhengfang Duanmu, and Zhou Wang. Geometric transformation invariant image quality assess- ment using convolutional neural networks. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6732-6736, 6736, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Saeed Mahloujifar, Chong Xiang, Vikash Sehwag, Sihui Dai, and Prateek Mittal. Robustness from perception. In International Conference on Learning Representations Workshop on Security and Safety in Machine Learning Systems, 2020.
Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 5437 5446, 2020.
Nicolas Papernot, P. Mcdaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami. Practical black-box attacks against. deep learning systems using adversarial examples. arXiv/1602. 02697, 2016.
Ekta Prashnani, Hong Cai, Yasamin Mostofi, and Pradeep Sen. Picapp: Perceptual image-error assessment
through pairwise preference. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1808 1817, 2018.
Mchdi SM Sajjadi, Bernhard Scholkopf, and Michael Hirsch. Enhancenet: Single image super-resohution through automated texture synthesis. In IEEE International Conference Computer Vision, pp. 4491 4500, 2017.
Ali Shahin Shamsabadi, Ricardo Sanchez-Matilla, and Andrea Cavallaro. Colorfool: Semantic adversarial colorization. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1151 1160, 2020.
Sanghyun Son, Jaerin Lee, Seungjun Nah, Radu Timofte, Kyoung Mu Lee, Yihao Liu, Liangbin Xie, Li Siyao, Wenxiu Sun, Yu Qiao, Chao Dong, Woonsung Park, Wonyong Seo, Munchurl Kim, Wenhao Zhang, Pablo Navarrete Michelini, Kazutoshi Akita, and Norimichi Ukita. AIM 2020 challenge on video temporal super-
resolution. In European Conference on Computer Vision Workshops, pp. 23 23-4000-0 40, 2020.
Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. Advances in Newral Information Processing Systems, 31, 2018.
Rainer Storn and Kenneth Price. Differential evolution a simple and efficient heuristic for global optimization over continuous spaces. Journal of global oplimization, 11(4):341 359, 1997.
Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Compulation, 23(5):828 841, 2019.
Christian Szegedy, Wojciech Zaremba, Nyya Sutskever, Joan Bruna, Dumitru Erhan, Tan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations, 2014.
IEEE Transactions on. Image Hossein Talebi and Peyman Milanfar. Nima: Neural image assessment. Processing, 27(8) 4011, 2018.
Florian Tramer, Nicolas Papernot, Ian Goodfellow, Dan Bonch, and Patrick McDaniel. The space of trans- ferable adversarial examples. arXin/1704.000 09453, 2017.
15

---

Published in Transactions on Machine Learning Research (05/2023)
Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adver- sarial example defenses. Advances in Neural Information Processing Systems, 33, 2020.
Yajie Wang, Shangbo Wu, Wenyi Jiang, Shengang Hao, Yu-an Tan, and Quanxin Zhang, Demiguise attack: Crafting invisible semantic adversarial perturbations with perceptual similarity. In International Joint Conference Artificial Intelligence, 2021.
Zhou Wang and Eero P Simoncelli. Translation insensitive image similarity in complex wavelet domain. In IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 2, pp. ii 573, 2005.
Zhou Wang, EcrO Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality as- sessment. In The Terith--eventhh Asilomar Conference on Signals, Systems & Computers, volune 2, pp. 1398 1402. IEEE, 2003.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on Image Processing, 13(4):600-61) 2004.
Andrew B Watson DCT quantization matrices visually optimized for individual images. In Human vision, visual processing, and digital display IV, volume 1913, pp. 202 216. International Society for Optics and Photonics, 1993.
Eric Wong, Frank Schmidt, and Zico Kolter. Wasserstein adversarial examples via projected Sinkhorn iterations. In International Conference on Machine Learning, volume 97, pp. 6808-6817, 2019.
Lei Wu and Zhanxing Zhu. Towards understanding and improving the transferability of adversarial examples in deep neural networks. In Asian Conference on Machine Learning, volume 129 of PMLR, pp. 837-850, 18-20 Nov 2020.
Chaowei Xiao, Jun- Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. In International Conference Learning Representations, 2018.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. In International Conference on Learning Representations, 2018.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan Yuille. Improving transferability of adversarial examples with input diversity. In IEEE Conference on Computer Vision and Pallern Recognition, 2019.
Xiaohui Zeng, Chenxi Liu, Yu-Siang Wang, Weichao Qiu, Lingxi Xie, Yu-Wing Tai, Chi-Keung Tang, and Alan L Yuille. Adversarial atlacks beyond the image space. In JEEE Conference on Computer Vision and Pattern Recognition, 2019.
Huan Zhang, Hongge Chen, Zhao Song, Duane Boning, Inderjit S Dhillon, and Cho-Jui Hsieh. The limitations of adversarial training and the blind-spot attack. In International Conference on Learning Representations, 2018a.
Kai Zhang, Shuhang Gu, and Radu Timofte. NTIRE 2020 challenge perceptual extreme super-resolution: Methods and results. In IEEE Conference on Computer Vision and Pattern Recognation Workshops, pp. 492-493, 2020.
Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. FSIM: feature similarity index for image quality assessment. IEEE Transactions Image Processing, 20(8):2378 2386, 2011.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 586-595, 2018b.
Weixia Zhang, Dingquan Li, Xiongkuo Min, Guangtao Zhai, Guodong Guo, Xiaokang Yang, and Kode Ma. Perceptual attacks of no-reference image quality models with human-in- in-tho-lopp. In Advances in Neural Information Processing Systems, 2022.
16

---

Published in Transactions on Machine Learning Research (05/2023)
Wen Zhou, Xin Hou, Yongjun Chen, Mengyun Tang, Xiangqi Huang, Xiang Gan, and Yong Yang. Trans- ferable adversarial perturbations. In European Conference on Compuler Vision, pp. 452-467, 2018.
Jun Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros. Generative visual manipulation the natural image manifold. Tn European Conference on Computer Vision, volume 9909, pp. 597-613, 2016.
17

---

Published in Transactions on Machine Learning Research (05/2023)
A
Transferable Attack on Perceptual Similarity Metrics
leaf
lother
prry
L.v PGD(10)
Las StAdv
Lay StAdv+PCD(10) PCD(10)
12
0.0079
0.0047
0,0048
0.0047
0,0048
SSIM 1
0.7717
0,8554
0.8501
0,8526
0.8436
FSIMc
0.99940
0,99937
0,99926
0,99922
0.99903
WaDIQuM-FR
1.3602
1.2796
1,2999
1.2962
1.3113
CTI-CNN &
95.99
81.11
139.53
133.61
165.44
DISTS 1
0.1303
0.1030
0.1111
0.1070
0.1139
LPIPS(Squ
0.1149
0,0794
0.0880
0,0855
0.0940
LPIPS(VCG)
0.1188
0.1346
0.1244
0.1409
E-LPIPS
0.0115
0.0077
0.0124
0.0131
0.0140
Walson-DFT DFT
1501.56
1025.
1278.13
1305.37
1422.43
PIM 4
2.1654
1.0225
2.8536
3,2559
3.1688
PIM-5
21.3579
10.0332
26.9732
29.6540
29.2036
leef
lother
lerey
Tedy PGD(10)
Tat stAdv
Tet www.sttttttptttt
ashool
withfin ashool
wrrfien school
achool
school
wittion
0.0121
0.0133
0.0133
0.0133
SSIM
0.9068
0.9112
0,9006
0.9103
0.8958
FSIMc
0,99392
0.99181
0,99185
0,99187
0,99183
WaDIQaM-FR
1.1942
1 1,2634
1,2653
1. 1,2699
1.2813
GTI-CNN &
66.66
28.88
62.75
61.31
DISTS
0.1341
0.1034
0.1121
0.1056
0.1132
(Squere))
0.0264
0.0371
0.0395
0.0375
0.0411
LPIPS(VGG) 4
0.0545
0.0462
0.0520
0.0472
0.0571
E-LPIPS 1
0.0039
0,0033
0,0055
0.0054
0.0063
Watson-DFT
1097.13
901.26
1147.84
1078.05
1157.19
PIM-1 4
0,2170
0.2429
1,0924
1.2546
1,2119
PIM
3.4366
2.9138
12.0777
13.0601
13.2696
leer
lather
Ieray
Low PCD(10)
Lasy StAdv
Lavv StAdv+PGD(10)
12 J
0.0368
0.0336
0.0339
0.0338
0.0342
SSIM
0.6526
0.6797
0.6693
0.6778
0.6667
FSIMc
0,96532
0,97225
0.97213
0.97228
0.97208
WaDIQuM-FR
1.1196
1.0508
1.0513
1.0550
1,0580
GTI-CNN &
473.92
325.45
379.77
382,28
386,48
DISTS 1
0.2266
0.2105
0,2278
0.2132
0,2297
LPips(Sq
0.0694
0.0467
0,0516
0.0545
LPIPS 1
0.2057
0.1614
0.1823
0.1640
0.1897
E-LPIPS 1
0.0138
0.0119
0.0131
0.0139
0.0132
Watson-DFT
2314.26
2072.06 06
2137.63
2006.34
2305.92
1.6024
0.8015
1.4746
1.5086
1.5491
PIM-5
14.5537
10.2235
14.8395
15.0432
15.2823
Figure 7: Transferable attack on perceptual similarity metrics. In example 1 (Top), the RMSE between
Ipeey and Iado images (left to right) is 1.35, 1.43, and 2.25. In example 2 (Mid.), the RMSE between I Iprey and Tado images (left to right) is 1.25, 0.95, and 1.77. In example 3 (Bot.), RMSE between Iprey and Tado images (left to right) is 1.37, 0.99, and 2.0. Text in red indicates that the rank has flipped.
18

---

Published in Transactions on Machine Learning Research (05/2023)
B FGSM Attack on Similarity Metrics
We explain the FGSM attack on perceptual similarity metrics in Algorithm 3
Algorithm 3: FGSM attack Similarity Metrics
Input: I1. 12, Trefs metric J, mas_ (0.05) Output: Least value which led rank flip so = f(drer,To); 81 f(dref, I1);
rank int(an > 81) // Io more similar to leef then rank 0 if rank = then Iprey = I1; Sother = Sui
else Iprey = Iui Sother = 81;
Sprcy = f(Lret,
J ((oother/ (Rother Sprey]) -1) // Loss signed grad sign( V Vinnay J)
0.0001
while S max do
Tndo Eprcy + signed grad
Tndv dipp addu min -1, max 1) // range [-1,1]
Sade f(Iref, Jadu)
if Sade Bother then
return True // Attack successful
18
c 0.0001
return // Largest value of
C One- Pixel Attack on Similarity Metrics
h
1 pixel
Jadv
LPIPS:0.0445
LPIPS:0.1868
Figure 8: One-pixel attack on LPIPS(Alex). This is a black-box attack as it does not require LPIPS network parameters to generate the adversarial perturbations. The one-pixel perturbation hardly visible. The RMSE between the prey image I and the adversarial image Jado is 1.38.
Results on Additional Datasets and High Resolution Images
Additional datasets. To test the vulnerability of perceptual similarity on higher image resolutions to adversarial attacks, we use the PicAPP test dataset (Prashnani et al., 2018) and the CLIC validation dataset (CLIC 2022). The CLIC dataset contains 5220 triplet samples (reference, distorted image A, and distorted image B), and acts as test datasct for us since none of the metrics have been trained on it. The PieAPP test set consists of 40 reference images with 15 distorted images per reference image. Out of these, only select those triplet samples where the preference for distorted image A over B was 85% and vice versa. Hence, we end up with 1381 samples for our experiment. The original image size of the CLIC samples is 768x768 while for PieAPP is 256x256.
White-box PGD attack. We first test the white- box attack on metrics via PGD. As shown in the Tables 0 7 the whito-box PGD attack casily flipping rankings on both datasets. The samples on the PicAPP dataset are harder to flip than the CLIC dataset. We posit that the reason for this lies in the selection criteria for our samples. Since for the PieAPP dataset, we chose only those samples where human preference
19

---

Published in Transactions on Machine Learning Research (05/2023)
Table 6: Whitebox PGD attack results on the PieAPP dataset.
Same Rank
PGD
Image Resolution
by Human & Metric
Total Samples
#amplees Plipped
% pixels with
RMSE
50.001
>0.01
>0.03
64x64
899
126/14.0%
67.5
49.9
0.0
1.7
0.8
65/13.5%
80.2
48.6
0.9
L2
59/6.1%
87.8
0.9
256x256
X
418
46/11.0%
86.9
62.4
0.0
2.1
1.0
64x64
910
391/43.0%
97.8
67.0
0.0
2.0
0.9
SSIM
471
120/25.5%
91.3
44.1
0.0
1.7
1.0
Wang
2004
900
364 /36.8%
96.7
68.9
0.0
2.1
0.9
256x256
391
185/47.3%
95.0
54.2
1.8
1.0
LPIeS(Alex)
64x64
1016
365
861 661/8877%
347/95.155
90.2
89.9
30.3
31.7
0.0
1.3
1.3
0.6
0.7
Zhang et al.
2018
256x256
X
197
191 191/99.09
/73.%
90.4
84.2
20.3
39.9
0.0
1.1
0.6
1041
125/12.0%
97.8
73.5
0.9
340
70/20.6%
96.4
66.
0.0
2.1
1.0
Ding al.
2020
256x256
1286
47/3.7%
97.4
73.8
0.0
2.3
1.0
25/26.3%
95.4
70.1
1005
823/81.9%
89.4
21.4
0.0
1.2
0.7
ST-LPIPS(Alex)
64x64
376
370/98.4%
87.2
26.3
0.0
1.2
0.6
Ghildyal & Liu 2022
256x256
1239
599/48.3%
93.9
56.1
L8
0.7
142
138/97.2%
90.4
33.9
0.0
144
0.7
Table Whitebox PGD allack results on the CLIC dataset.
Same Rank by Metric
PGD
Network
Image Resolution
Total Samples
#Samples Flipped
% pixels with
IMSE
>0.001
>0.01
>0.03
3167
1152/99.5%%
67.6
26.6
0.0
1.1
0.6
256x256
2053
2027/98.
67.5
19.6
0.0
1.0
3120
2911 2911/93.399
74.8
37.2
0.0
1.4
0.8
512x512
X
2100
1918/91.5 1.3%
74.8
30.7
0.0
1.3
2992
2399//80.29
0.0
768x768
X
2228
1762/79. 1%
80.5
48.0
0.0
1.6
0.8
3307
3307 100.00
84.2
5.7
0.0
0.8
0.1
X
1913
1912/99.999
76.0
6.2
0.0
0.8
0.4
Wang et al.
SSIM
2004
512x512
2020
3200
2005/99.3%
3189/99.75 7%
89.1
85.8
14.6
0.0
0.0
0.9
1.0
0.5
2997
2941/98.. 1%
89.9
18.6
0.0
11
0.6
768x768
2223
2173/97.88
87.8
14.9
0.0
1.0
3820
100.09 0% 3820
54.5
0.0
0.0
0.7
0.0
256x256
X
1400
1400/100 100.0%
39.3
0.0
0.0
0.7
0.0
LPIPS(Alex)
3965
3905 100.0%
64.4
0.3
0.0
0.7
0.1
Zhang
2018b
512x512
1255
1255/100.0%
50.8
0.1
0.0
0.7
0.1
38-19
3839/99.7%
73.6
2.
0.0
0.7
0.2
768x768
1371
1371/100.0%
0.0
3822
3327/87.0%
97.4
56.2
0.0
1.7
256x256
X
1398
1308/93.69 6%
95.9
11.4
0.0
1.5
0.8
Ding al.
2020
512x512
X
4004
1216
2626/65.6%
06./79.6%
98.6
98.2
62.4
72.7
0.0
0.0
1.9
2.1
0.9
0.9
768x768
3952
2356/32.5%
98.6
80.0
0.0
24
0.9
1268
199/39.499
96.9
69.4
0.0
22
3793
3793/ 100.0%
56.1
0.0
0.0
0.7
0.0
256x256
1427
1427/ 100.0%
40.2
0.0
0.7
Ghildyal & Liu
ST-LPIPS(ALL) (elx)
2022
512x512
X
1194
4026/10 100.0%
1194 100.0%
53.5
0.1
0.0
0.0
0.7
0.7
0.1
4021
4009/99.77
81.3
5.2
0.0
0.8
0.3
X 1199
1199 100.0%
72.8
1.8
0.0
0.7
0.2
for distorted image over the other was > 85%, scems that the margin between the classes, namely, *loss similar" and "more similar" to the reference, is larger, than in the CLIC dataset, making it harder to flip the rank.
20

---

Published in Transactions on Machine Learning Research (05/2023)
White-box stAdv attack. We attack the (PIPS(Alex) metric using StAdv the PicAPP dataset. For the images with higher resolution, it was harder to flip rank. However, that could be due to the settings of our setup. In the loss defined in Equation 4 for the stAdv attack, minimizing Liow constrains the amount of flow used to generate the adversarial perturbations while minimizing Lrank encourages more perturbations. Hence, we increase a, i.e., the weight for Lrank: larger amount of perturbations would be generated as the flow generating adversarial perturbations will be less constrained. As shown in Table 5 we observe that increasing a helps flipping rank for more samples, but the RMSE of the Lado with Iyrey is also higher.
Table 8 Whitebox stAdv attack on (AleS(Alex) on the PieAPP dataset.
Image Resolution
# Accurate Samples
from Equation
Accurate Samples Flipped
RMSE (u/a)
50
909/88.5%
4.3/2.0
64x64
1016
200
1000/98.4%%
5.8/3.1
1000
1016/100.00
7.8/4.9
28 1%
0.8/0.3
256xx256
1184
200
158 158/13. 133.%
2.1/1.3
1000
566/47.8%
3.7/1.9
Transferable Adversarial attack. Here we test the transferable PGD(20) attack. In this experiment, we attack the (PIPS(Alex) metric using the PGD. This experiment is performed on the PieAPP dataset because we found harder to flip samples on it. Out of the 1184 accurate samples, the rank flipped for 635 samples with a mean RMSE of 1.92 with a standard deviation of 0.15. We test the transfera of these 635 samples to other perceptual similarity metrics. We found that although the metrics did change their scores due to the adversarial perturbations, worsening their prediction, this datasel. Less than 10% of the samples flipped ranks. However, the transferable allack results in Table g are consistent with the results on the BAPPS dataset in Table of the main paper. The traditional metrics are more robust than the Icarned metrics, while the learned metrics are more accurate. The transformer- based metric swinIQA has high accuracy and robustness. E-LPIPS and ST-LPIPS(VGG) which are more robust variants of I.PIPS(VGG), showcase more robustness, with ST-LPIPS(VGG) also being more accurate. Similarly, PIM-1 and DISTS are also accurate, along with being more robust. Surprisingly, WaDIQ&M-FRR showcases higher accuracy the PieAPP dataset than the BAPPS dataset, along with being robust both datasets.
still harder to flip ranks
Table 9: Transferable PGD(20) attack on perceptual similarity metries.
Network
Accurate Samples
# #AAcurate Samples Flipped via PGD(20)
448/71%
2/0.4%
SSIM (Wang et al. 2004
456 456/72%
17/3.7%
MS-SSEM Wang ot al.
2003
460/72%
11/2.4%
CWSSIM (Wang Simoncell
2005
114/65%
15/3.6%
PSIMe (Zhang et a.. 2011
461/73%
4/0.9%
WaDIQAEITI [Bosse
2018
602/95%
13/2.2%
GTI-CNN (Ma 21 al. 2018
154/71%
2/0.4%
PieAPP Prashmant
476/75%
8/1.7%
LPIPS (Zhang of 201 8b
611 611996%
26/4.3%
LPIPS(VGG) (Zhang et
554/87%
03/ 03/11.4%
E-LPIPS (Kettunen el al
20191
354/87%
8/1.4%
DISTS (Ding el ai. 2020
607 9
24/4.0%
Watson-DFi CZolbe
2020
475 75%
32/6.7%
PIM-1 Bhardwaj et al
2020
558 /88%
22/3.9%
PIM-5 [hardwa al
550/87%
33/6.0%
A-DISTS Ding of Bl 102T
512/81 %
36/7.0%
ST-LPIPS (Ghldyal
2022
614/97%
14/2.3%
ST-LPIPS(VGG Giildval
2022
584/92%
25/1.3%
SwinIQA (Lu et nl.
2022
994%
17/2.8%
21

---

Published in Transactions on Machine Learning Research (05/2023)
E FGSM versus PGD attack
In our experiments in Table 2 the value chosen for the maximum allowable fs perturbation for the PGD attack lower than that for the FGSM attack. However, if the value the same, then PGD would be better at flipping the rankings. As shown in Table TO the PGD attack is more successful than FGSM. In the case of traditional metrics, the results for both attacks are similar. However, for learned perceptual similarity metrics like LPIPS, the number of flips by PGD are greater with a lesser amount of perturbation required.
Table 10: FGSM and PGD attack results when the maximum lo -orm perturbation is the same for both.
Same Rank
FGSM (e < 0.03)
PGD
Network
by Human & Metric
Total Samples
#Samples Flipped
Mean
RMSE
# Samples Flipped
% pixels with
RMSE
"
>0.001
>0.01
50.03
2419/25%
0.014
1.9
2348/288
56.1
1.0
L.2
2477
1220/49%
0.011
1.5
1.0
1202/49%
82.0
42.7
0.0
1.5
1.0
SSIM
9883
5383/54%
0.012
1.7
1.0
5297
/54%
94.6
53.6
0.0
1.8
1.0
Wang et al
2004
2344
1851/79%
0.008
1.3
0.8
1843/79%
87.3
32.0
0.0
1.3
0.8
LPI'S(Ales
11303
5620/50%
0.012
1.7 1.0
8806/ 788%
86.8
28.7
0.0
1.3
0.6
Zhang
897/97%
0.003
0.9
917/999 99%
59.5
3.2
PIPS/VGG
10976
7431/68%
0.008
1.3 U.9
1689//88%
81.6
15.6
0.0
1.0 0.5
Zhang al.
2018
1251
1235/99%
0.002
0.8
1246/100%
52.3
1.6
0.0
0.2
DSTS
11158
1827/16%
0.015
2.1
1.6
2306 21%
97.0
75.1
1.3
Ding ot al.
2020
1069
643/60%
0.011
1.0
1.0
723/68%
91.9
50.0
0.0
2.0
1.3
In the PGD attack, the step size a is often greater when compared to ours, i.e., 0.001, such that it allows the perturbations to go beyond the maximum -orm threshold and then can be projected back to the radius. We Lest with a larger values of a, and as shown in Table [1 the severity of the attack increases as a is increased, however, at the expense of more % pixels with perturbation > 0.01. The other parameters for the attack are kept the same.
Table 11: PGD attack results with increasing step size a.
Same Rank by Human
PGD
Network
Total Samples
#Samples Flipped
% pixels with
RMSE
& Metric
>0.001
>0.01
s0.03
V
9750
2348/24%%
84.4
56.1
0.0
1.9
1.0
0.00100
2477
1202/49%
82.0
42.7
0.0
1.0
V
9750
2419/25%
87.3
63.8
0.0
1.9
1.0
0.00375
2477
1220/49%
88.2
51.0
0.0
1.6
1.0
9750
2419/25%%
87.3
67.9
1.1
0.00600
x
2477
1220/419%
88.2
55.6
0.0
1.8
1.1
0.00100
v
9883
5297/54%
94.6
53.6
0.0
1.0
X
2344
1843/79%
87.3
32.0
0.0
1.3
0.8
SSIM
0.00375
7
9883
2344
5418/55%
1858/79%
99.0
99.2
63.7
40.5
0.0
1.8
0.9
1.0
9883
5118/55%
99.1
70.5
0.0
2.1 1.1
0.00600
2344
1858/79%
46.8
1.0
8806/78%
28.7
0.6
0.00100
924
917 /90%
59.5
3.2
0.0
0.8
0.3
(PIeS(Alex)
Zhang al. 2018b
0.00375
11303
924
920/1009 100%
9926/887
90.1
93.4
45.3
7.5
0.0
0.0
0.8
1.5
0.8
0.3
11303
9994/88%
$8.0
55.2
0.0
1.8 0.8
0. 0.00600
920/
93.4
15.1
0.4
10976
9689/88%
81.6
15.6
0.0
1.0 0.5
0.00100
1251
1246/1009
52.3
1.6
0.7 0.2
LPIPS(VGG)
10976
10322/91%
89.9
29.6
0.7
Zhang et: al
0.00375
X
1251
1248 1040
95.8
3.7
0.0
0.7 0.2
10976
10337 10337/94%
994%
88.4
40.8
0.0
1.4 0.8
0.00000
1251
1218/100%
96.2
7.9
0.0
0.3
22

---

Published in Transactions on Machine Learning Research (05/2023)
F Reversing the PGD Attack
4
Ldvv
+
LPIPS:0.144
APIPS:0.044
Figure 9: PGD attack on LPIPS((Aee)) Alex). We make the less similar of the two distorted images more similar to the reference image. The RMSE between the prey image lo and the adversarial image Lodv is 4.20.
In this experiment, we try the reverse of the white-box PGD attack in Section 3 For this attack, we do the opposite, i.e, we attack the distorted image that is less similar to Leef- Before the attack, the original
rank Sother < Spreys but after the allack Jrey turns into Jadv; and when the rank flips, Sado < 8other. We use the LPIPS network parameters to compute the signed gradient via the loss function in Equation 7 As
shown in Table 12 it is possible to reverse the attack performed in Table 2
(0, Apreys Lothers Lref) = (
2
Sother
8othher + Sprey
Table 12: Reverse PGD attack results. Here attack the less similar distorted image and make it more similar to the reference image. Below are the results of the Whitcbox PGD attack the BAPPS dataset.
Same Rank by Human & Metric
PGD
Network
Total Samples
# Samples Flipped
% pixels with
RMSE
> 0..01
>0.01
50.03
LPIPS (Alex)
11303
6758/39.9% 858/92.9%
87.0
27.4
0.0
1.27
0.59
Zhang et al. 2018b
X
924
60.3
5.6
0.0
0.82
0.34
23


Reviews:
[{"review_id": "Ob5mfcq5b7", "reviewer": "Reviewer_EFA2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Studying the robustness of perceptual metrics is a relevant topic.\n\n- The paper analyzes many existing perceptual metrics, and shows that all are to some extent vulnerable to adversarial perturbations.\n\nWeaknesses\n- In Table 2 it seems that FGSM is often more successful than PGD (according to number of samples flipped). This is quite surprising since PGD should lead to a better optimization of the target loss. One reason might be that FGSM is actually tested with several radii, but this would also indicate that PGD is suboptimal: for example, in Alg. 1, with step size $\\alpha$ of 0.001 and 40 iterations, the maximal $\\ell_\\infty$-norm of the perturbation can be 0.04, which is smaller than both the radius in PGD (0.1) and the maximal radius for FGSM (0.05). For PGD the step size and iterations are typically chosen to allow the algorithm to reach the borders of the $\\ell_\\infty$-ball. Then, it is not clear how effective the attacks used for evaluation are, and if the robustness of the various metrics can be further reduced (this might be important to assess the relative robustness of different metrics).\n\n- As minor point, if I understand it right, in the experiments only the image with initial higher similarity is perturbed: it might be interesting to show also the other way, i.e. increasing the similarity of the other image.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the adversarial robustness of both traditional and learned metrics for perceptual similarity. In particular, given two distorted images whose similarity to a reference image is computed with a certain metric, it shows that it possible to flip the ranking of the two images via a small adversarial perturbation to one of them. In the experiments, this is shown to happen quite consistently across metrics and threat models for generating the attacks. Finally, the paper studies the transferabitlity across metrics of the adversarial perturbations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I think it is necessary to clarify the strengths of the attacks (see above).", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8WrulQXtdD", "reviewer": "Reviewer_hxh8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n(+) Relevance: The study of adversarial examples on perceptual similarities is an important topic. After I had a look at the literature, I was surprised that a systematic study of this topic had not already been discussed.  \n(+) Systematic examination: The paper presents a systematic examination of the robustness of various traditional and learned perceptual similarity metrics to imperceptible adversarial perturbations.  \n(+) Methodology: The authors propose a methodology to craft such perturbations via adversarial attacks and demonstrate that these attacks can be transferable to other metrics.  \n(+) Benchmark: The paper provides a benchmark for the robustness of metrics to imperceptible adversarial perturbations.  \n\n## Weaknesses\n(-) Significance: The result that perceptual similarity metrics are susceptible to adversarial attacks is not really surprising. Since for example the LPIPS leverage “standard” trained neural networks to calculate the perceptual distance between two images, which are well known to be susceptible to adversarial attacks it is not surprising that the metric in itself will be susceptible to adversarial examples.  \n(-) Overclaim: The authors claim that “none of the previous investigations have ever considered attacking perceptual similarity metrics.” However, in E-LPIPS the authors show “that such learned perceptual similarity metrics (LPIPS) are susceptible to adversarial attacks that dramatically contradict human visual similarity judgment.”  \n(-) Novelty: The authors do not introduce any novel methodology, since existing datasets, metrics, and attack techniques are used.  \n(-) The evaluation is sparse: Since the authors aim to systematically examine the robustness on perceptual similarity, more attacks and similarity scores should be taken into account. For example, currently, the AutoAttack is considered the strongest adversarial example technique. Additionally, various perceptual similarity techniques have been proposed such as E-LPIPS, Shift-tolerant perceptual similarity metric, etc. A thorough investigation of these metrics would be beneficial for the readers. Especially, the results for which adversarial attacks do not succeed are of interest for the community. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a systematic examination of the robustness of various traditional and learned perceptual similarity metrics to adversarial perturbations. The authors demonstrate that such perturbations can compromise these metrics, which can overturn a metric's similarity judgment. They also propose a methodology to craft such perturbations via adversarial attacks and show that these attacks can be transferable to other metrics. The paper provides a benchmark for the robustness of metrics to imperceptible adversarial perturbations. The contributions of the paper include identifying vulnerabilities of perceptual similarity metrics and highlighting the need for robustness validation in the design and development of newer metrics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would appreciate it if the authors could address the points in my weakness section. \n\n1. The authors should clarify the significance of their study in light of previous works on the susceptibility of neural networks to adversarial attacks. This clarification should address the extent to which the findings presented in this paper extend and complement existing knowledge in the field.  \n2. The authors should revise the claim that \"none of the previous investigations have ever considered attacking perceptual similarity metrics\" to acknowledge previous work that has investigated the susceptibility of perceptual similarity metrics to adversarial attacks.  \n3. The authors should consider evaluating their methodology on additional datasets, using additional perceptual similarity techniques, and using stronger adversarial attack techniques such as AutoAttack.  \n", "broader_impact_concerns": "Based on the information provided in the paper, it does not appear that the authors have explicitly addressed the limitations of their work or provided an ethical statement. It is important for authors to acknowledge the potential limitations of their study and discuss any ethical considerations related to their research, including the potential impact on society or the use of the findings in a harmful manner. This is especially important in work on adversarial attacks. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Cfpju9IHBF", "reviewer": "Reviewer_oM18", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n1. The topic studied in this paper is interesting. Considering we always use perceptual similarity metrics to help us develop new algorithms in low-level vision tasks, the study on metric robustness may help us reconsider the progress in recent years.\n2. The experimental results are notable and convincing, indicating the vulnerability of current perceptual similarity metrics, including the traditional and the DNN-based ones. \n\n### Weakness\n**Major:**\n1. The motivation in this paper is unclear. Why should we study the robustness of the perceptual metrics? Which problem would a vulnerable perceptual metric cause? The authors are expected to provide a more detailed discussion to convince the readers of this study's values.\n2. Why should we use the two-alternative forced-choice experimental design? Actually, we could use the norm-based constraint as used in attack methods to measure the difference between the original image and the perturbed image. There might be something special in the requirements for perceptual similarity metrics, which is unfamiliar to researchers from other fields (e.g., me from adversarial attacks and defense). The authors are expected to explain this in this paper. \n3. What is the difference between employing LPIPS to optimize adversarial examples and adversarially attacking LPIPS directly? It confuses me for a long while reading the Introduction. The authors are encouraged to explain their difference clearly.\n4. Why do we use different optimization losses between Eqn (1) and Eqn (6)? \n\n**Minor:**\n\n5. In Summary of Sec. 4,  Figures 7 -> Figure 7.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the robustness of perceptual similarity metrics regarding adversarial attacks. Specifically, the authors craft adversarial perturbations using FGSM, PGD, stAdv, and single-pixel attack and find they can easily flip the rank in two-alternative forced-choice experiments across various perceptual similarity metrics, which indicates the vulnerability of the", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Strength and Weakness.", "broader_impact_concerns": "No broader impact statement is present and does not seem necessary.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ttHWnZEDy7", "reviewer": "Reviewer_RTsU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think this paper has the following strengths: \n\n1. It studies an interesting and important problem: the robustness of perceptual similarity metrics to imperceptible adversarial perturbations.\n\n2. The paper is well-written and easy to understand. It provides enough background for understanding the paper. The related work is appropriately discussed. \n\n3. The experiments are extensive and can support the main claims of the paper. It also creates a benchmark that provides a good starting point for discussion and further research on the robustness of perceptual similarity metrics to imperceptible adversarial perturbations. \n\nHowever, I think this paper has the following weaknesses: \n\n1. It just modifies the attack objectives of existing attacks and uses them to attack perceptual similarity metrics. No new attack methods are introduced. Thus, the novelty is limited.\n\n2. In the experiments, the image patches are scaled from size 256$\\times$256 to 64$\\times$64. Since the low image resolution may affect human judgment and perceptual similarity metrics, it would be good to also show results for larger image resolution. \n\n3. It is unclear whether the adversarial robustness of perceptual similarity metrics is a real concern in practice. It should give some real-world scenarios where the vulnerability of perceptual similarity metrics can cause problems.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the robustness of various traditional and learned perceptual similarity metrics to imperceptible perturbations. It devises a methodology to craft perturbations via adversarial attacks such as FGSM, PGD, and the One-pixel attack. It finds that the addition of imperceptible distortions can overturn a metric's similarity judgment when comparing two images with a reference. The results indicate that even learned perceptual metrics that match with human similarity judgments are susceptible to such imperceptible adversarial perturbations. It shows that when combined with the PGD attack, the transferability of the adversarial examples can be further increased. It creates a benchmark that provides a good starting point for discussion and further research on the robustness of perceptual similarity metrics to imperceptible adversarial perturbations. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It should clarify the novelty of the paper more. This would strengthen the work. \n\n2. It should perform experiments on images with high-image resolution. This is critical to securing my recommendation for acceptance.\n\n3. It should clarify the motivation for studying the adversarial robustness of perceptual similarity metrics more. This is critical to securing my recommendation for acceptance.", "broader_impact_concerns": "I don't have any concerns about the ethical implications of the work. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the reliability of both classical and learned perceptual similarity metrics to adversarial manipulations. Since the evaluation of adversarial attacks heavily depends on the defined metrics, this paper aims to revisit the fundamentals of adversarial examples and highlight the weaknesses of existing metrics. All reviewers unanimously recommend acceptance, and I concur.

I am also recommending Feature Certification based on the criterion that "If this paper was submitted to a top-tier conference, it would likely be presented as an oral/spotlight." My rationale is that as the community of adversarial machine learning is gradually shifting the focus from simple Lp norm perturbations to a more realistic perturbation model, such as the studied perceptual similarity metrics, this paper shows these metrics still have their own flaws. I believe this study will play an important role to push the frontier of research in this field.
Decision: Accept (poster)