LipSim: A Provably Robust Perceptual Similarity Metric
Authors: Sara Ghazanfari
Alexandre Araujo
Prashanth Krishnamurthy
Farshad Khorrami
Siddharth Garg
Venue: ICLR 2024 poster (2024)
Keywords: Perceptual similarity metric
certified defense
deep learning
Primary area: metric learning, kernel learning, and sparse coding

Abstract: Recent years have seen growing interest in developing and applying perceptual similarity metrics. Research has shown the superiority of perceptual metrics over pixel-wise metrics in aligning with human perception and serving as a proxy for the human visual system.
On the other hand, as perceptual metrics rely on neural networks, there is a growing concern regarding their resilience, given the established vulnerability of neural networks to adversarial attacks. It is indeed logical to infer that perceptual metrics may inherit both the strengths and shortcomings of neural networks.
In this work, we demonstrate the vulnerability of state-of-the-art perceptual similarity metrics based on an ensemble of ViT-based feature extractors to adversarial attacks. We then propose a framework to train a robust perceptual similarity metric called LipSim (Lipschitz Similarity Metric) with provable guarantees. 
By leveraging 1-Lipschitz neural networks as the backbone, LipSim provides guarded areas around each data point and certificates for all perturbations within an $\ell_2$ ball. Finally, a comprehensive set of experiments shows the performance of LipSim in terms of natural and certified scores and on the image retrieval application.

Published as a conference paper at ICLR 2024
LIPSIM: A PROVABLY ROBUST PERCEPTUAL SIMILARITY METRIC
Sara Ghazanfari", Alexandre Araujo
Prashanth Krishnamurthy, Farshad Khorrami, Siddharth Garg
Department of Electronic and Computer Engineering New York University
#sg7457enyu.edu
ABSTRACT
Recent years have seen growing interest in developing and applying perceptual similarity metries, Research has shown the superiority perceptual metrics over pixel-wise metrics in aligning with human perception and serving as a proxy for the human visual system. On the other hand, as perceptual metries rely on neural networks, there is a growing concern regarding their resilience, given the estab- lished vulnerability of neural networks adversarial attacks. It indeed logical to infer that perceptual metrics may inherit both the strengths and shortcomings of neural networks. In this work, we demonstrate the vulnerability of state-of-the-art perceptual similarity metrics based tors to adversarial attacks. We then propose a framework to train a robust pereep- tual similarity metric called LipSim (Lipschitz Similarity Metric) with provable guarantees. By leveraging 1-Lipschitzz neural networks as the backbone, LipSim provides guarded areas around each data point and certificates for all perturba- tions within an E ball. Finally, a comprehensive set of experiments shows the performance of LipSim in terms of natural and certified scores and the image retrieval application.
ensemble of ViT-based feature extrac-
INTRODUCTION
Comparing data items and having a notion of similarity has long been a fundamental problem in computer science. For many years Ep norms and other mathematically well-defined distance metrics have been used for comparing data items. However, these metrics fail to measure the semantic similarity between more complex data like images and more focused on pixel-wise comparison. To address this problem perceptual distance metrics (Zhang et al. 2011 Fu et al., 2023) have been proposed that employ deep neural networks as backbone to first compute embeddings, then apply traditional distance metrics on the embeddings of the data in the new space.
It is well-established that neural networks susceptible to adversarial attacks (Goodfellow al., 2014). That is, imperceptible variations of natural examples can be crafted to deliberately mislead models. Although perceptual metries provide rich semantic interpretations compared to traditional metrics, they inherit the properties of neural networks and therefore their susceptibility to adversarial attacks (Kettunen et al. 2019 Sjögren et al. 2022, Ghildyal & Liu, 2022). Recent works have tried to address this problem by training robust perceptual metrics [Kettunen et al. 2019 Ghazanfari et al., 2023). However, these works rely on heuristic defenses and do not provide provable guarantees. Recent research has focused on designing and training neural networks with prescribed I. Lipschitz constants (Tsuzuku et al. 20188) Meunier et al. 20022) Wang & Manchester] 2023), aiming to improve and guarantee robustness against adversarial attacks. Promising techniques, like the SDP-based Lipschitz Layer (SLL) (Araujo et al. 2023), have emerged and allow to design of non-trivial yet efficient neural networks with pre-defined Lipschitz constant. Constraining the Lipschitz of neural has been known to induce properties such as stability in training (Miyato et al. 2018), robustness Tsuzuku al. 2018), , and generalization (Bartlett al. 2017).
Recently, the DreamSim metric (Fu et al. 2023) has been established as the ofate-of-the-an per- ceptual similarity metric. This metric consists of concatenation of fine-tuned versions of ViT-

---

Published as a conference paper at ICLR 2024
0.64
0.59
0.50
0.76
0.65
0.64
0.62
0.65
0.73
d( 0,
0.68
0.63
0.54
0.75
0.66
0.64
0.66
0.62
0.75
Demonstrating the effect of an attack on the alignment of DreamSim distance values with the real perceptual distance between images. Instances of original and adversarial reference images from the NIGHT dataset are shown in the first and second rows and the DreamSim distance between them (i.e., d( )) is reported below. To get a sense of how big the distance values are, images that have the same distance from the original images are shown in the third row (i.e.,
Figure I:
d(0, @ = d(O 2)). Obviously, the first and third rows include semantically different images, whereas the images on the first and second rows are perceptually identical.
based embeddings. namely. DINO CCarn et al. 2021 CLIP (Radford et al. 2021). and Open CLIP (Cherti al. 2023). To compute the distance between images, DreamSim measures the cosine similarity distance between these ViT-based embeddings.
In this work, initially demonstrate with a series of experiments that the DreamSim metric is not robust to adversarial examples. Consequently, it could be easy for an attacker to bypass important filtering schemes based on perceptual hash, copy detection, etc. Then, tackle this problem, we propose LipSim, the first perceptual similarity metric with provable guarantees. Building on the DreamSim metric and recent advances 1-Liischitzz neural networks, we propose novel student- teacher approach with Lipschit--constrannce student model. Specifically, train a I-Lipschitz feature extractor (student network) based on the state-of-the-ar SLL architecture. The student net- work is trained to mimic the outputs of the embedding of the DreamSim metric, thus distilling the intricate knowledge captured by DreamSim into the I-Lipschitz student model. After training the 1- Lipschitz feature extractor the ImageNet-1k 1 dataset, we fine-tune it on the NIGHT dataset which a two-alternative forced choice (2AFC) dataset that seeks to encode human perceptions of image similarity (more explanation and some instances of NIGHT dataset are presented in Appendix [B). By combining the capabilities of DreamSim with the provable guarantees of Lipschitz network, our approach paves the way for a certifiably robust perceptual similarity metric. Finally, we demon- strate good natural accuracy and tfte-of--the-rr certified robustness on the NIGHT dataset. Our contributions can be summarized as follows:
We investigate the vulnerabilities of state-of-the-art ViT-based perceptual distance including DINO, CLIP, OpenCLIP, and DreamSim Ensemble. The vulnerabilities are highlighted using APGD (Croce & Hein 2020) the 2AFC score which is index for human alignment and PGD attack against the distance metric and calculating the distance between an original image and its perturbed version.
We propose framework to train the first certifiably robust distance metric, LipSim, which leverages pipeline composed of I-Lipschitz feature extractor, projection to the unit &g ball and cosine distance provide certified bounds for the perturbations applied the reference image.
We show by comprehensive set of experiments that not only LipSim provides certified accuracy for a specified perturbation budget, but also demonstrates good performance in terms of natural 2AFC score and accuracy image retrieval which is serious application for perceptual metrics.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORKS
Similarity Metrics. Low-level metrics including l norms, PSNR as point-wise metries, SSIM (Wang et al. (2044) and FSIM (Zhang et al. 2011) as patch-wise wise metrics fail to capture the high-level structure and the semantic concept of more complicated data points like images. In order to overcome this challenge the perceptual distance metries were proposed. In the context of percep- tual distance metrics, neural networks are used as feature extractors, and the low-level metries are employed in the embeddings of images in the new space. The feature extractors used in recent work include a convolutional neural network as proposed by ZZhang et al. (2018) for the LPIPS metric, or ensemble of ViT-based models (Radford et al.] [ Caron et al. 2021)) proposed by Fu (et al. (2023) for DreamSim. As shown by experiments the perceptual similarity metrics have better alignment with human perception and are considered good proxy for human vision.
Adversarial Attacks & Defenses. Initially demonstrated by SZzeedy et al. 2013), neural net- works are vulnerable to adversarial attacks, i.e., carefully crafted small perturbations that can fool the model into predicting wrong answers. Since then a large body of research has been focused on generating stronger attacks (Goodfellow et al.] 2014, Kurakin et al.] 2018, Carlini & Wagner
2017| Croce& & Hein| 2020, 2021) and providing more robust defenses (Goodfellow et al.) 2014 Madry et al. 2017] Pinot et al. 2019 Araujo et al.) 2020 2021; Meunier et al. 2022). To break this pattern, certified adversarial robustness methods were proposed. By providing mathematical guarantees, the model is theoretically robust against the worst-case attack for perturbations smaller than specific perturbation budget. Certified defense methods fall into two categories. Randomized Smoothing (Cohen et al., 2019 Salman et al. 2019) turns an arbitrary classifier into a smoother classifier, then based on the Neyman-Pearson lemma, the smooth classifier obtains some theoreti- cal robustness against a specific l, norm. Despite the impressive results achieved by randomized
smoothing in terms of natural and certified accuracy (Carlini et al. 2023), the high computational cost of inference and the probabilistic nature of the certificate make it difficult to deploy in real-time applications. Another direction of research has been to leverage the Lipschitz property of neural networks (Hein & Andriushchenkol [017 Tsuzuku et al.] 2018) to better control the stability and robustness the model. Tsuzuku et al. (2018) highlighted the connection between the certified radius of the network with its Lipschitz constant and margin. As calculating the Lipschitz con- stant of a neural network is computationally expensive, a body of work has focused on designing 1-Lipschitz networks by constraining each layer with its spectral norm (Miyato et al.) 2018 Farnia
ctt al. 2018), replacing the normalized weight matrix by an orthogonal ones (Li et al. 2019 Prach
& Lampert 2022) or designing 1-Lipschitz layer from dynamical systems (Meunier al. 2022) or
control theory arguments (Araujo et al. 2023 Wang & Manchester 2023).
Vulnerabilities and Robustness of Perceptual Metries. Investigating the vulnerabilities of per-
ceptual metrics has been overlooked for years since the first perceptual metric was proposed. As shown in (Kettunen et al. 2019 Ghazanfari et al., 2023 Sjögren et al. (022/GGildge & Liu, 2022) perceptual similarity metrics (LPIIPS (Zhang et al. 2018)) are vulnerable to adversarial attacks. (Sjögren et al. 2022) presents a qualitative analysis of deep perceptual similarity metrics resilience to image distortions including color inversion, translation, rotation, and color stain. Finally (Luo) jt al. 2022) proposes a new way to generate attacks to similarity metrics by reducing the similarity between the adversarial example and its original while increasing the similarity between the adver- sarial example and its most dissimilar one in the minibatch. To introduce robust perceptual metrics, (Kettunen al., 2019) proposes e-lpips which uses an ensemble of random transformations of the input image and demonstrates the empirical robustness using qualitative experiments. (Ghildyal & Liu 2022) employs some modules including anti- alii-aiissing filters to provide robustness to the vul- nerability of LPIPS to one-pixel shift. More recently (Ghazanfari et al. (2023) proposes R-LPIPS which is a robust perceptual metric achieved by adversarial training Madry et al. (017)) over LPIPS and evaluates R-LPIPS using extensive qualitative and quantitative experiments on BAPPS (Zhang ctt al., 2018) dataset. Besides the aforementioned methods that show empirical robustness, (Kumar & Goldstein 2021 Shao et al. 2023) propose methods to achieve certified robustness on perceptual metries based on randomized smoothing. For example, Kumar& & Goldstein (2021) proposed cen- ter smoothing which is an approach that provides certified robustness for structure outputs. More precisely, the center of the ball enclosing at least half of the perturbed points in output space is considered the output of the smoothed function and is proved to be robust to input perturbations
3

---

Published as a conference paper at ICLR 2024
bounded by Ez-size budget. The proof requires the distance metric to hold symmetry property and triangle inequality. As perceptual metrics generally don't hold the triangle inequality. the triangle inequality approximation is used which makes the bound excessively loose. In Shao et al.] (2023), the same enclosing ball is employed however, the problem is mapped to a binary classification set- ting leverage the certified bound as in the randomized smoothing paper (by assigning one the points that are in the enclosing ball and zero otherwise). Besides their loose bound, these methods are computationally very expensive due the Monte Carlo sampling for each data point.
BACKGROUND
Lipschitz Networks. After the discovery of the vulnerability of neural networks to adversarial attacks, one major direction of research has focused on improving the robustness of neural networks to small input perturbations by leveraging Lipschitz continuity. This goal can be mathematically achieved by using a Lipschitz function. Let f be Lipschitz function with L Lipschitz constant in terms of f2 norm, then we can bound the output of the function by [I(()) - f(x 6)1|2 ,, ||01||-- To achieve stability using the Lipschitz property, different approaches have been taken. One efficient way is to design network with I-Lipschitz layers which leads to a 1-Lipschitz network (Meunier
it al. 2022 Araujo et al. 2023 Wang & Manchester 2023).
State of the Art Perceptual Similarity Metric. DreamSim is recently proposed perceptual dis- tance metric (Fu et al.
2023) that employs cosine distance on the concatenation of feature vectors generated by an ensemble of ViT-based representation learning methods. More precisely DreamSim is concatenation of embeddings generated by DINO (Caron et al., (021), CLIP (Radford et al.) 202T], and Open CLIIP (Cherti et al. 2023). Let be the feature extractor function, the DreamSim distance metric d(x1,T2)) (2) is defined as:
(.(f(
where S2(x1, 22) is the cosine similarity metric. To fine-tune the DreamSim distance metric, the NIGHT dataset is used which provides two variations, To and 21 for reference image æ, and label y that based on human judgments about which variation is more similar to the reference image x (some instances and more explanation of the NIGHT dataset are deferred to Appendix B). Supplemented with this dataset, the authors of DreamSim turn the setting into a binary classification problem. More concretely, given triplet (I, To, I1), and a feature extractor f, they define the following classifier:
M(x) - {b.
(0, S d(x, 0o d(z,z1) d(z, zo)
(2)
Finally, to better align DreamSim with human judgment, given the triplet (z, zo, 21), they optimize hinge loss based on the difference between the perceptual distances d(z, 21) and d(æ, x1) with margin parameter. Note that the classifier h has a dependency on f, d and each input comes as triplet (I, 21) but to simplify the notation we omit all these dependencies.
LIPSIM: LIPSCHITZ SIMILARITY METRIC
In this section, present the theoretical guarantees of LipSim along with the technical details of LipSim architecture and training.
4.1 A PERCEPTUAL METRIC WITH THEORETICAL GUARANTEES
General Robustness for Perceptual Metric. A perceptual similarity metric can have a lot of impor- tant use cases, e.g., image retrieval, copy detection, etc. In order to make robust perceptual metric we need ensure that when a small perturbation is added to the input image, the output distance should not change lot. In the following, we demonstrate a general robustness property when the feature extractor 1-Lipschitz and the embeddings lie on the unit E2 ball, i.e., [] 1.
Proposition 1. Let f + R² be a 1-Lipschitz feature extractor and [x)]] = 1, let d be a
distance metric defined as in Equation Land let 5 E X and E R$ such that [ l < E. Then, have
[d(x),02) d(zi 8, x2)1 S 1101|2
(3)
4

---

Published as a conference paper at ICLR 2024
Step Lipechite-basee bast Student- Student-Teacher training embbeddings
Step Lipschitz fineturning Night Dataset
d(3.
LLipp. Model
RMSE
label
ImageNet
Night Dataset
Image
Dino, Clip, OpenCiip
Dreamsim
minimize
(1)
Projocted HUpp. Model
Hinge Loss
minimiee
Data Jittering
1.L.p. Model
RMSE
MER
Figure 2: Two-step process for training the LipSim perceptual similarity metric. First (left), a dis- tillation on the ImageNet dataset performed where DreamSim acts the teacher model, and 1-Lipschitz neural network (i.e., the feature extractor) is learned to mimic DreamSim embeddings. To reduce color bias with the Lipschitz network, we use two different dataset augmentation schemes: simple random flip and jittered data augmentation technique that varies the brightness, contrast, hue, and saturation of the image. Second (right), the I-Lipschitz neural network with projection is then fine-tuned on the NIGHT dataset with hinge loss.
The proof is deferred to Appendix A 1-Lipschitz and its output is projected on the unit & ball then the composition of the distance metric and the feature extractor, i.e., d f. is also I-Lipschitz with respect to its first argument. This result provides some general stability results and guarantees that the distance metric cannot change more than the norm of the perturbation.
This proposition implies that when the feature extractor is
Certified Robustness for 2AFC datasets. We aim to go even further and provide certified robust- ness for perceptual similarity metrics with 2AFC datasets, iee. in a classification setting. In the following, show that with the same assumptions as in Proposition [ the classifier h can obtain certified accuracy. First, let us define a soft classifier H X - R2 with respect to some feature extractor as follows:
(xx) = [d(x,, x1), d(x, vo)]
(4)
is clear that hxx) arg max (x) where 11, represent the i-th value of the output H. The classifier is said to be certifiably robust at radius 2 0 at point x if for all ||ó1| S we have:
h(x + 5) = M(x)
(5)
Equivalently, one look at the margin of the soft classifier: MH. Hg(x)) 11, (1- and provide a provable guarantee that:
MH,2+5 > 0
Theorem 1 (Certified Accuracy for Perceptual Distance Metrie). Let H X IR.2 be the soft classifier as defined in Equation 4 Let d E X' and E E Rt such that [|0]|2 S E. Assume that the
feature extractor f X + Rett is I-Lipschitz and that for all x, [l(())) = 1, then we have the
following result:
M 2 (x))))))) [(x1))l2
Mn 2 0
The proof is deferred to Appendix A Based on Theorem [ and assuming Ii # No. the certified radius for the classier h at point x can be computed as follows:
M
R(h,z) [f(zo) - f(x1)ll
Theorem provides the necessary condition for a provable perceptual distance metric without chang- ing the underlying distance the embeddings (i.e., cosine similarity). This result has two key ad- vantages. First, as in Tsuzuku et al.] (2018), computing the certificate at each point only requires efficient computation of the classifier margin H. Leveraging Lipschitz continuity enables efficient certificate computation, unlike the randomized smoothing approach of Kumar & Goldstein) (202T) which requires Monte Carlo sampling for each point. Second, the bound obtained on the margin to guarantee the robustness is in fact tighter than the one provided by Tsuzuku et al. (2018). Re- call Tsuzuku et al. (2018) result states that for a L-Lipschitz classifier H. we have:
Mn 2 2
Mf.t+8
(9)
5

---

Published as a conference paper at ICLR 2024
Given that the Lipschitz constant of H is v2, this lead to the following bound:
MH 2 2s 2 =zlf(z)) - f(11)ll2
(10)
simply based on the triangle inequality and the assumption that Il [2)]l2 = 1.
4.2 LIPSIM ARCHITECTURE & TRAINING
To design reliable feature extractor that can be used with Proposition I and Theorem | we com- bined a 1-Lipschitz neural network architecture with an Euclidean projection. Let f X R such that:
f(x) = 12) a 8 2 o .... (())
(11)
where 1 is the number of layers, (0,11) is projection the unit l ball, i.e., (B2 (z) arg min sEB(0, || 2||B and where the layers & are the SDD-based Lipschitz Layers (SLL) pro- posed by Araujo et al. (2023):
-1 o(W x x+b).
<(x) x 22 didg
W WWTWle) ls
(12)
where W is parameter matrix being either dense or convolution, {q:} forms diagonal scaling matrix and a is the ReLU nonlinear activation.
To apply Theorem 1 for certification, we need [
with 1-LLipschitz. To respect these (z)] 2 in order for the projection x (..().)) to be 1-
conditions, we need || o (لو Lipschitz. The mapping 3 x + + b is I-Lipschitz translation which is set to increase the norm of the embedding (i.e., ideally above one) before the Euclidean projection. However, we not have a guarantee that the norm of the embedding after the 3 projection is above one, and therefore concerning the assumption of Theorem in such cases, we abstain from the prediction. In practice, however, for all the samples of the NIGHT dataset || ((9)) abstain percentage is equal to zero.
(x) ll and thus the
Proposition 2. The neural network f ; x IR describe in Equation 11 is 1-Lipschitz and for all with || (z)]l2 under the assumption that ||3 g(l)) (1)) 2
Proof of Proposition 2 The proof is straightforward application of Theorem 3 of |raauo et al. (202)) and Corollary 2.2.3 of Nesterov et al. (2018).
Two-step Process for Training LipSim. LipSim aims provide good image embeddings that are less sensitive to adversarial perturbations. We train LipSim in two steps, similar to the DreamSim approach. Recall that DreamSim first concatenates the embeddings of three ViT-based models and then fine-tunes the result on the NIGHT dataset. However, to obtain theoretical guarantees, we can- not use the embeddings of three ViT-based models because they are not generated by a 1-Lipschitz feature extractor. To address this issue and avoid self-supervised schemes for training the feature extractor, we leverage a distillation scheme on the ImageNet dataset, where DreamSim acts as the teacher model and student model. This first step is described on the left of Figure In the second step, we fine-tuned the I-Lipschitz neural network with projection the NIGHT dataset using hinge loss increase margins and therefore robustness, as in Araujo et al. 2023). This second step is described on the right of Figure
1-Lipschitz neural network (without the & unit ball projection)
EXPERIMENTS
In this section, present comprehensive set of experiments first highlight the vulnerabilities of DreamSim which is the state-of-the-art perceptual distance metric, and second to demonstrate the certified and empirical robustness of LipSim adversarial attacks.
Recall the Lipschitz the concatenation. Let f and g be I,, and Lipsciip, then the function x H (xx, [(]]] can be upper bounded by / LL, 3 + L3 -ip-cipsc

---

Published as a conference paper at ICLR 2024
100
LipSi DremSii
Base metho brsaric rOaBi
PSNR
SSIM
LPIPS
DiST8
Chp
OpenGip
Dina
DreamSim
UpSin
0.0
0.2
0.4
0.6
0.8
1.0
(b)
Figure Figure 3a compares percentages of alignment of several distance metries with human vision based on the NIGHT dataset. The ViT-based methods outperform the pixel-wise and CNN- based metrics for the original images. However, LipSim with the 1-Lipschitz backbone composed of CNN and Linear layers has decent natural score and outperforms the (Base) Clip. Moreover, the figure shows the performance under attack (f2-APGD with 2.0) for the SOTA metric. While perturbing the reference image, other methods experiencing large decay in their performance but LipSim is showing much stronger robustness. Figure 3b shows the distribution of d(x, + 6) for LipSim and DreamSim. The ó perturbation is optimized for each method separately.
5.1 VULNERABILITIES OF PERCEPTUAL SIMILARITY METRICS
To investigate the vulnerabilities of DreamSim to adversarial attacks, we aim to answer two ques- tions in this section; Can adversarial attacks against SOTA metrics cause: (1) misalignment with human perception? (2) large changes in distance between perturbed and original images?
Q1 Alignment of SOTA Metric with Human Judgments after Attack. In this part we focus on the binary classification setting and the NIGHT dataset with triplet input. The goal is to generate adversarial attacks and evaluate the resilience state-of-the-art distance metrics. For this purpose, use APGD (Croce & Hein 2020), which is one of the most powerful attack algorithms. During optimization, we maximize the cross entsooy loss, the perturbation is crafted only on the reference image and the two distortions stay untouched:
arg max Lyc 6) Len([d(ar d, d( (,)))) 8:11611,55
(13)
Where E {0,1) and ý = [d(x + 5, d(x + 8, mo)] which is considered as the logits generated by the model. The natural and adversarial 2AFC scores of DreamSim are reported Table The natural accuracy drops to half the value for tiny perturbation of size 0.5 and decreases to zero for c = 2.0. order to visualize the effect of the attack on the astuteness of distances provided by DreamSim, original and adversarial images (that generated by &2-APGD and caused misclassi- fication) are shown in Figure [ The distances are reported underneath the images as d( @). To get a sense of the DreamSim distances between the original and perturbed images, the third row is added so that the original images have (approximately) the same distance to the perturbed images and the perceptually different images in the third row (d( 0, @) d( O )).. The takeaway from this experiment is the fact that tiny perturbations can fool the distance metric to produce large values for perceptually identical images.
- -2-SSecialized Attack for Semantic Metric. In this part, perform direct attack against the feature extractor model which is the source of the vulnerability for perceptual metrics by employing the f-PPGD Madry al. 2017) attack (c 1.0) and the following MSE loss is used during the optimization:
max 8:1881l_Se
LMSE (z + 8), fx))]
(14)
The attack is performed on 500 randomly selected samples from the ImageNet- Ik test set and against the DreamSim Ensemble feature extractor. After optimizing the the DreamSim distance metric
7

---

Published as a conference paper at ICLR 2024
Table 2: Certified scores of LipSim given different settings. The natural and certified 2AFC scores of all variants of LipSim are shown in this figure. The LipSim DreamSim version outperforms other variants regarding certified scores. The tradeoff between robustness and accuracy compares the results for different margins in the hinge loss. A higher margin parameter leads to a higher certified score and lower natural score.
LipSim with Teacher Model
Margin in Hinge Loss
Natural Score
Certified Score
36
T2
108
265
256
0.2
84.76
64.14
34.76
10.53
LipSim DINO
0.4
84.65
65.19
40.51
18.04
0.5
81.96
66.28
44.63
22.49
0.2
83.33
61.18
34,87
13.60
LipSim OpenCLIP
0,4
80.59
63.27
42.32
21.71
0.5
81.30
64.80
45.12
25.38
0.2
85.58
62.88
35.36
11.18
LipSim Dreamsim
0.4
83.33
65.40
43.69
21.11
0.5
82.89
66.39
44.90
23.46
calculated between the original image and the perturbed image: d(x, x + 8). The distribution of distances is shown in Figure 3b We can observe shift in the mean of the distance from 0 to 0.6 which can be considered as a large value for the DreamSim distance as shown in Figure |
5.2 LIPSIM RESULTS
In this section, we aim to leverage the framework introduced in the paper and evaluate the LipSim perceptual metric. In the first step (i.e., right of Figure 2). we train a 1-L.ischhtzz network for the backbone of the LipSim metric and use the SSL. architecture which has 20 Layers of Conv-SSI. and 7 layers of Linear-SSL. For training the I-Lipschitz feature extractor, the ImageNet- Ik dataset is used (without labels) and the knowledge distillation approach is applied to utilize the state-of- the-art feature extractors including DINO, OpenCLIP, and DreamSim which an ensemble ViT- based models. To enhance the effectiveness of LipSim, we incorporate two parallel augmentation pipelines: standard and jittered. The standard version passes through the feature extractor and the teacher model while the jittered only passes through the feature extractor. Then, the RMSE loss is applied to enforce similarity between the embeddings of the jittered and standard images. This enables LipSim to focus more on the semantics of the image, rather than its colors. After training the I-Lipschitz backbone of LipSim, we further fine-tune our model on the NIGHT dataset (i.e., step see right of Figure D. During the fine-tuning process, the embeddings are produced and are projected to the unit E2 ball. In order to maintain the margin between logits, the hinge loss is employed similarly to DreamSim. However, while DreamSim has used margin parameter of 0.05, we used a margin parameter of 0.5 for fine-tuning LipSim in order to boost the robustness of the metric. Remarkably, LipSim achieves strong robustness using I-Lipschitz pipeline composed of 1-Lipschitz feature extractor and projection to the unit & ball that guarantees the 1-Lipschitzness of cosine distance. To evaluate the performance of LipSim and compare its performance against other perceptual metrics, we report empirical and certified results of LipSim for different settings.
Empirical Robustness Evaluation. We pro- vide the empirical results of LipSim against 2-- APGD in Table [ Although the natural score of LipSim is lower than the natural score of DreamSim, there is a large gap between the ad-
Table I: Alignment on NIGHT dataset for origi- nal and perturbed images using APGD. In this ex- periment, the perturbation is only applied on the reference images. LipSim demonstrates higher robustness on all perturbations.
versary scores. We can observe that LipSim outperforms all state-of tfth-atf metrics. The re- sults of more comprehensive comparison be- tween LipSim, state-of-the-art perceptual met- rics, previously proposed perceptual metrics, and pixel-wise metrics are presented in Fig- ure Ba The pre-trained and fine-tuned natu- ral accuracies comparable with the of the-art metrics and even higher in comparison to CLIP. In terms of empirical robustness, LipSim demonstrates great resilience. More comparisons
Metric/ Embedding
Natural Score
APGD
0.5
1.0
2.0
3.0
CLIP OpenCLIP
93.91
29.93
8.44
1.20
0.27
95.45
72.31
42.32
11.84
3.28
DINO DreamSim
94.52
81.91
59.04
19.29
6.35
96.16
46.27
16.66
0.93
LipSim (ours)
85.58
82.89
79.82
72.20
61.84
8

---

Published as a conference paper at ICLR 2024
LipSim
DreamSim
Figure 4: Adversarial attack impact on the performance of DreamSim and LipSim distance metrics on image retrieval application. The 1 rows show the original images and the top-1 nearest neighbors. Adversarial images generated separately for LipSim and DreamSim metrics along with their top-1 nearest neighbors are depicted in rows. More precisely, the red block shows a complete sample in the figure, where the upper and lower right images are the original and adversarial queries and the upper and lower left images are the 1-top nearest images to them respectively.
have been performed in this sense, the empirical results over APGD and f--MIA are also re- ported in Table and Table 5 in Appendix D which aligns with the l2 results and shows strong empirical robustness of LipSim. In order to evaluate the robustness of LipSim outside the classifica- tion setting, we have performed P2D attack (c L.0) using the MSE loss defined in Equation [4 and the distribution of d(xz x + 5) is shown at Figure 3b) The values of d(æ, x + 8) are pretty much close to zero which illustrates the general robustness of LipSim as discussed in proposition The histogram of the same attack with a bigger perturbation budget (e = 3.0) is shown Figure of the Appendix.
Certified Robustness Evaluation. In order to find the certified radius for data points, the margin between logits is computed and divided by the le norm distance between embeddings of distorted images ([f((z)) - [(x) ll.). The results for certified 2AFC scores for different settings of LipSim are reported in Table 2 which demonstrates the robustness of LipSim along with high natural score. The value of the margin parameter in hinge loss used during fine-tuning is mentioned in the table which clearly shows the trade-off between robustness and accuracy. larger margin parameter leads to more robustness and therefore higher certified scores but lower natural scores.
5.3
IMAGE RETRIEVAL.
After demonstrating the robustness of LipSim in terms of certified and empirical scores, the focus of this section is on one of the real-world applications of distance metric which image retrieval. We employed the image retrieval dataset proposed by Fu et al. (2023), which has 500 images randomly selected from the COCO dataset. The top-1 closest neighbor an image with respect LipSim and DreamSim distance metrics are shown at the O rows of Figure In order to investigate the impact of adversarial attacks on the performance of LipSim and DreamSim in terms of Image Retrieval appli- cation, we have performed P2D attack (c = 2.0) with the same MSE loss defined in Equation [4] separately for the two metries and the results are depicted at the @ rows of Figure 4 In adversarial rows, the LipSim sometimes generates different image the closest which is semantically similar to the closest image generated for the original image.
CONCLUSION
In this paper, we initially showed the vulnerabilities of the SOTA perceptual metries including DreamSim to adversarial attacks and more importantly presented framework for training certifi- able robust distance metric called LipSim which leverages the l-Lipschitz network as its backbone, I-Lipschitz cosine similarity and demonstrates non-trivial certified and empirical robustness. More- over, LipSim was employed for an image retrieval task and exhibited good performance in gathering semantically close images with original and adversarial image queries. For future work, It will be interesting to investigate the certified robustness of LipSim for other 2AFC datasets and extend the performance of LipSim for other applications, including copy detection, and feature inversion.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This paper is supported in part by the Army Research Office under grant number W91 1NF-2111-- 0155 and by the New York University Abu Dhabi (NYUAD) Center for Artificial Intelligence and Robotics, funded by Tamkeen under the NYUAD Research Institute Award CG010.
REFERENCES
Alexandre Araujo, Laurent Meunier, Rafael Pinot, and Benjamin Negrevergne. Advocating for multiple defense strategies against adversarial examples. In ECML PKDD 2020 Workshops, 2020.
Alexandre Araujo, Benjamin Negrevergne, Yann Chevaleyre, and Jamal Atif. On lipschitz regular- ization of convolutional layers using toeplitz matrix theory. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, 2021.
Alexandre Araujo, Aaron Havens, Blaise Delattre, Alexandre Allauzen, and Bin Hu. A unified algebraic perspective on lipschitz neural networks. In The Eleventh International Conference on Learning Representations, 2023.
Peter I. Bartlett, Dylan J Foster, and Matus Telgarsky. Spectrally-normalizedd margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017
ieee symposium on security and privacy (sp). 2017.
Nicholas Carlini, Florian Tramer, Krishnamurthy Dj Dvijotham, Leslie Rice, Mingjie Sun, and Zico Kolter. (certified!!) adversarial robustness for free! In The Eleventh International Confer- ence on Learning Representations, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pp. 9650-96600 2021.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor-
don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-inen -mage learning. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, 2023.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference machine learning, 2019.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with ensemble of diverse parameter-fre free attacks. In International conference on machine learning, 2020.
Francesco Croce and Matthias Hein. Mind the box: 1-app for sparse adversarial attacks on image classifiers. In International Conference Machine Learning, 2021.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 9185-9193, 2018.
Farzan Farnia, Jesse M Zhang, and David Tse. Generalizable adversarial training via spectral nor- malization. arXiv preprint arXiv: 118//.070 1.04577, 2018.
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning dimensions of human visual similarity using synthetic data. arXiv preprint arXiv:2306.09344 2023.
Sara Ghazanfari, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, and Alexan- dre Araujo. R-Ipips: An adversarially robust perceptual similarity metric. arXiv preprint arXiv:2307.15157 2023.
Abhijay Ghildyal and Feng Liu. Shift-tolerant perceptual similarity metric. In European Conference Computer Vision, pp. 91-107. Springer, 2022.
10

---

Published as a conference paper at ICLR 2024
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572,2 2014.
Matthias Hein and Maksym Andriushchenko. Formal guarantees the robustness of classifier against adversarial manipulation. Advances in neural information processing systems, 30, 2017.
Markus Kettunen, Erik Härkönen, and Jaakko Lehtinen. E-lpips: robust perceptual image similarity via random transformation ensembles. arXiv preprini urXiv:1906.03973, 2019.
Aounon Kumar and Tom Goldstein. Center smoothing: Certified robustness for networks with structured outputs. Advances in Neural Information Processing Systems, 34:5560-5575. 2021.
Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security. 2018.
Qiyang Li, Saminul Haque, Cem Anil, James Lucas, Roger B Grosse, and Jörn-Henrik Jacobsen. Preventing gradient attenuation in lipschitz constrained convolutional networks. Advances in neural information processing systems, 32, 2019.
Cheng Luo, Qinliang Lin, Weicheng Xie, Bizhu Wu, Jinheng Xie, and Linlin Shen. Frequency- driven imperceptible adversarial attack on semantic similarity. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15315-15324, 2022.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0000 1608.0603, 2017.
Laurent Meunier, Blaise Delattre, Alexandre Araujo, and Alexandre Allauzen. A dynamical system perspective for lipschitz neural networks. In International Conference on Machine Learning. PMLR, 2022.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint /802 2018.
Yurii Nesterov et al. Lectures on convex optimization, volume 137. Springer, 2018.
Rafael Pinot, Laurent Meunier, Alexandre Araujo, Hisashi Kashima, Florian Yger, Cédric Gouy- Pailler, and Jamal Atif. Theoretical evidence for adversarial robustness through randomization. Advances in neural information processing systems, 2019.
Bernd Prach and Christoph H Lampert. Almost-orthogonal layers for efficient general-purposs lipschitz networks. European Conference on Computer Vision, pp. 350-365. Springer, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748 8763. PMLR, 2021.
Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. Ad- vances in Neural Information Processing Systems, 32, 2019.
Huaqing Shao, Lanjun Wang, and Junchi Yan. Robustness certification for structured prediction with general inputs via safe region modeling in the semimetric output space. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2010-2022, 2023.
Oskar Sjögren, Gustav Grund Pihlgren, Fredrik Sandin, and Marcus Liwicki. Identifying and miti- gating flaws of deep perceptual similarity metrics. arXiv preprint arXiv:2207.02512. 2022.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312,6/ 2013.

---

Published as a conference paper at ICLR 2024
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Lipschitz-marce training: Scalable certi- fication of perturbation invariance for deep neural networks. Advances in neural information processing systems, 31, 2018.
Ruigang Wang and lan Manchester. Direct parameterization of lipschitz-bounded deep networks. In International Conference on Machine Learning, pp. 36093-361 10. PMLR, 2023.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. Fsim: A feature similarity index for image quality assessment. IEEE transactions on Image Processing, 20(8):2378-2388 2011.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
12

---

Published as a conference paper at ICLR 2024
A PROOFS
Proposition 1. Let f : - R2t be a 1-Lipschitz feature extractor and [lf 1| = 1, let d be a
distance metric defined as in Equation [ ann let ò E X and E IR+ such that ||ô112 S E. Then, we have,
[d(2), 12) d(21 + 82)22 s 1101|
(3)
Proof of Proposition 1 We have the following:
(f(2) + 8). f(x2))
(f(x))
[d(a1, d( + 8, x2)1 =
[f(z] 6)]ll
[[(x))]]](<
[s(s) [)) ((x2))
Kf(x)) +6) - [(2))
[[((z + 6) - f(=))1l]f(((
||ó]|
where (1) and (2) are due [1]())]] 1 for all and the fact that is 1-Lipschitz, which concludes the proof.
Theorem 1 (Certified Accuracy for Perceptual Distance Metric). Let H X + IR2 be the soft
classifier defined in Equation 4 Let 5 X and E E R$ such that [|0||2 S Assume that the
feature extractor f X R& is I-Lipschitz and that for all x, [1f(2)) 1, then have the
following result:
MH (=)) f(1))ll
MH 20
Proof of Theorem extractor f as follows:
7
First, let us recall the soft classifier H X x g² with respect to some feature
H(x) = (1),,,,)), 20)l
(15)
(f(x).f(g) where d: X x X > R is defined as: d(x,, v) = TT(']]l 1/
Let us denote Ho and Hi the first and second logits of the soft classifier. For a tuple (z, zo. æ1 and target label y. we say that H correctly classifies if arg max Hxx) y. Note that we omit the dependency on Zo and øj in the notation. Let us assume the target class y = The case for = is exactly symmetric. Let define the margin of the soft classifier H as:
MH == (x)x) - Ho(x)
(16)
We have the following:
M M 8) (1 + 8) Holx 8)
H1 8) H1(x) Holæ + 5) Ho(x) + (H)(s) Ho(c))
d(x + 8, x0) d(x,20) - d(x + +1)
-
(1
(f(x + 0), f(zo))
(1
(f(x), ((oo)
If + 0)12211
[1()))2211
(s(x ()x
MH,
[
(f( + (f(x (xx)) f(z1)) + Mn
= [f(=++))/(aa-//(000)>----- (f(x) f(x0) f(x1)) MH
+ Mn
2 ((fc))-((e+0]]]/((ca-//f)f MH
2
Mz.z
where (1) due the fact that for all ær [=)]l = Therefore, MH 2 only if Mu.x
(x))))))) + [(x1)] which conclude the proof.
13

---

Published as a conference paper at ICLR 2024
B DATASET DETAILS & ADDITIONAL FIGURES
In the section, we initially represent some details of 2AFC datasets and more specifically NIGHT dataset and show some examples of this dataset.
B. DATASET DETAILS
In order to train perceptual distance metric, datasets with perceptual judgments used. The perceptual judgments are of two types: two alternative forced choice (2AFC) test, that asks which of two variations of reference image is more similar to it. To validate the 2AFC test results, a second test, just noticeable difference (JND) test is performed. In the JND test, the reference images and one of the variations
asked to be the same different. BAPPS (Zhang et al. 2018) and NIGHT (Fu et al. (023) are two datasets organized with the 2AFC and JND judgments. The JND section of the NIGHT dataset has not been released yet, therefore
did evaluations only based on the 2AFC score. In Figure we show some instances from the NIGHT dataset, the reference is located in the middle and the two variations are left and right. The reference images are sampled from well-known datasets including ImageNet-Ik.
Figure 5: Some instances of NIGHT dataset is shown. The middle image is the reference image and right and left images are To and distortions respectively.
C CERTIFIED ROBUSTNESS ON BAPPS DATASET
In this section, we aim to show the generalization of LipSim for other 2AFC dataset available called BAPPS (Zhang et al. 2018). The BAPPS has the same structure with NIGHT dataset as explained at Appendix B However, the left and right images have been generated by adding distortions to the reference image and the label determines which of the distortions is more similar to the reference. Therefore the distribution of data for NIGHT and BAPPS datasets totally different. To provide certification for the BAPPS dataset, the labels which are are between and are rounded to have bi- nary labels for all samples. Later, leverage the finetuned verions of LipSim to provide certificates on the BAPPS dataset. Although LipSim fine-tuned on the NIGHT dataset and the distribution of images differ for these two datasets, it has good certified scores on BAPPS dataset as reported Table 3
Table 3: Certified scores of LipSim on BAPPS dataset.
Margin in Hinge Loss
Natural Score
Certified Score
Metric
DreamSim
78.47
0.0
0.0
0.0
0.2
73.47
30.0
12.96
5,33
LipSim
0.4
74.31
31.74
15.19
7.0
0.5
74.29
31.20
15.07
6.77
14

---

Published as a conference paper at ICLR 2024
D ADDITIONAL EMPIRICAL RESULTS FOR LIPSIM
In this section we aim to perform diverse set of empirical analysis on LipSim. At first we revisit the empirical robustness of LipSim using different adversarial attacks. Second we compare LipSim with recent empirical defense called R-LPIPS Ghazanfari et al. (2023) on both NIGHT and BAPPS dataset. Later we look into the general robustness property and perform stronger attack directly to the metric. Afterwards, we present another application for LipSim as perceptual metric which is the KNN Task and Finally we have the comparison between LipSim and other metrics in terms of natural score.
D.1 EMPIRICAL ROBUSTNESS RESULTS
In this part, the empirical robustness of LipSim is evaluated using & -PGG, loo -GG attack and Momentum Iterative Attack Dong al. (2018). The cross entropy loss defined in Equation 13 is used for the optimization. In the case of -PGGD, LipSim outperforms all metrics for the entire of perturbation budgets. For -PGD, the performance of DreamSim better for 0.01, however, LipSim has kept high stable score encountering PGD attacks with different perturbation sizes, which demonstrates the stability of the LipSim. The results for Momentum Iterative Attack are reported at Table [5] which is in line with the other results presented in the paper in terms of empirical robustness of LipSim towards adversarial attacks.
Table Alignment on NIGHT dataset for original and perturbed images using APGD. this ex- periment, different feature extractors are employed in combination with cosine distance to calculate the distance between reference images and distorted images, and the perturbation is only applied to the reference images. DINO, CLIP, and OpenCLIP semble is a concatenation all these three feature extractors and LipSim Backbone a 1-Lipschitz network that trained from scratch using the knowledge distillation approach.
ViT-based feature extractors. DreamSim En-
Metric/ Embedding
Natural Score
APGD
0.01
0.02
0.03
DINO
94.52
14.91
1.42
0.21
CLIP
93.91
0.93
0.05
0.00
OpenCLIP
95.45
7.89
0.76
0.05
DreamSim
96.16
2.24
0.10
0.05
LipSim
85.58
62.28
33.66
15.19
Table 5: Alignment on NIGHT dataset for original and perturbed images using Momentum Iterative Attack.
Natural Score
f2-MIA
Ex -MIA
Metric
0.5
2
0.01
0.02
0.03
DreamSim
96.16
61.79
52.85
52.69
2.08
0.05
0.0
LipSim
85.58
82.79
79.99
80.10
62.45
34.38
15.84
15

---

Published as a conference paper at ICLR 2024
LipSim
DreamSim
0.0
0.2
0.4
0.6
0.8
1.0
12
Figure 6: Distribution of d(2,, x 8) where 8 is generated using P-PPGD attack with e=3.0
D.2 GENERAL ROBUSTNESS OF LIPSIM
In order to evaluate the general robustness of LipSim in comparison with DreamSim, we optimize the MSE loss defined in Equation 14 employing Eg- 2g-PDD attack for LipSim and DreamSim sepa- rately. The distribution of d(x, x + 8) is shown in Figure The difference between this figure and the histogram in Figure 3bjis that have chosen larger perturbation budget, the fact that even for larger perturbations, LipSim is showing general robustness.
demonstrate
D.3 COMPARISON WITH AN EMPIRICAL DEFENSE
As discussed the Related Work, there exist few papers about providing defense for perceptual metrics. R-LPIPS (Ghazanfari et al. 2023) is recent paper to propose empirical defense for the LPIPS (Zhang et al.) 2018) perpetual similarity metric by leveraging Adversarial training and -PD attacks. For our evaluation
perform - and 22-MIA (Momentum Iterative attack) and provide an empirical robustness score on the BAPPS dataset as well as the NIGHT dataset in need to emphasize that R-LPIPS is trained on the BAPPS dataset while Lipsim
Table 6 Finally finetuned on the NIGHT dataset and Lipsim provides certified scores while R-LPIPS does not come with any provable guarantees.
Table 6: Alignment on NIGHT and BAPPS dataset for original and perturbed images with LipSim and R-LPIPS metrics.
Natural Score
GPG = 0.03
Metric
Dataset
1.0
NIGHT BAPPS
70.56 80.25
58.50
32.46
R-LPIPS
72.38
70.94
NIGHT
85.58
79.99
75.27
LipSim
BAPPS
73.47
60.09
42.77
16

---

Published as a conference paper at ICLR 2024
D.4 MORE APPLICATIONS: KNN TASK
For the applications of LipSim, we have presented image retrieval in Section 5 and for second application, add the KNN task this part. KNN (k-nearest neighbors algorithm) is zero-shot classification task that classifies test images based on the proximity of their feature vectors to the training images' feature vectors. We performed our experiment on Imagenettd (ImageNet dataset with 10 classes) and for h = {10, 20). The accuracy of LipSim and DreamSim for the KNN task are reported in Table 7 LipSim is providing more than 85% accuracy on classification task by leveraging its robust embeddings and its accuracy is also very close to DreamSim in terms of Top 5.
Table 7: LipSim and DreamSim comparison for the KNN task on Imagenette dataset.
10-NN
20-NN
Metric
Top
Top
Top
Top 5
DreamSim
99.03
99.82
98.82
99.89
LipSim
85.32
97.20
85.35
98.09
http://////.faaaaaa///getttttt magenette
17


Reviews:
[{"review_id": "a6h29uWT1h", "reviewer": "Reviewer_KFDY", "summary": "This paper studies how to develop a robust perceptual similarity metric. Specifically, this paper propose to leverage 1-Lipschitz neural network to provide guarded areas around each data point. The experimental results illustrate the effectiveness of the proposed method.", "questions": "See weakness.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The experimental results looks very good when against auto attacks.\n\nThe proof seems to make sense but I am not an export with this.", "weaknesses": "The presentation could be better. For example, the explanation of 2AFC is little which making it difficult to get the messages from this paper.\n\nThe experiments only conducted with Auto Attack. However, there are different kinds of attacks, and it would be good to experiment with other attack methods as well.\n\nIt would also be good to compare with other certified or non-certified defense methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3V0L6eziDO", "reviewer": "Reviewer_YdcH", "summary": "This study investigates Lipschitz Networks' robustness against small adversarial perturbations, capitalizing on Lipschitz continuity. A 1-Lipschitz function can bound a network's output to prevent it from changing excessively in response to slight input changes. Another key focus is the DreamSim perceptual distance metric, which utilizes cosine distance on combined feature vectors derived from various representation learning techniques. Furthermore, a new metric, LipSim, is introduced, aiming to be robust against adversarial disturbances. LipSim's design employs a two-step training process that first uses DreamSim as a guiding model on the ImageNet dataset, followed by fine-tuning on the NIGHT dataset to enhance its robustness.", "questions": "n/a", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- LipSim employs a 1-Lipschitz network backbone, which when combined with certain design choices, greatly enhances its resistance to adversarial perturbations.\n- When applied to the real-world task of image retrieval, LipSim effectively identified semantically close images, even when faced with adversarial image queries.\n-  LipSim excels both in empirical and certified robustness tests. This dual proficiency ensures that the metric's performance is not only observed in experimental conditions but also certified under specific robustness criteria.", "weaknesses": "- The natural score of LipSim was observed to be lower than that of some competitors like DreamSim. This might raise concerns about its general performance when not under adversarial conditions.\n- The real-world application testing of LipSim was primarily on image retrieval. It would be beneficial to see its performance on a wider variety of tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D3GmYJ9YQy", "reviewer": "Reviewer_sTNV", "summary": "Authors argue that perceptual metrics inherit both the strengths and shortcomings of neural networks. One of the important shortcomings of neural Networks is their vulnerability to adversarial examples.\nThus it makes sense to design adversarially robust nn-based perceptual metrics. They show that the DreamSim metric is not robust to adversarial examples. They then propose a robust perceptual\nsimilarity metric called LipSim (Lipschitz Similarity Metric) with provable guarantees.", "questions": "Q: Please mention why we need robust perceptual metrics??!!! In real world. In other words, how much impact such work has in real world and why it actually matters? An example may help here.\n\nQ: How about some non NN-based similarity metrics? I am guessing maybe there are not as good as NN-based ones but at least they might be more robust! Can you compare your method agains some of those m\nMetrics? Is DISTS in Fig. 3.a NN-based?\n\nQ: Other datasets than Nights dataset? What is the guarantee that these results will also generalize to other datasets?\n\n\nQ: In 5.1, you are stating “Can adversarial attacks against SOTA metrics cause: (1) misalignment with human perception?”. \nIn section 4.1, you mention “Q: Please mention why we need robust perceptual metrics??!!! In real world. In other words, how much impact such work has in real world and why it actually matters? An example may help here. Alternatively, shouldn’t two image differ significantly when perturbation is high? Have you considered what happens at high perturbations?\nShouldn’t a good perceptual metric match human judgements regardless of perturbation magnitude?\n\n\nQ: Table 2 shows the certified scores. However, the scores for perturbation 72/255 and beyond are still pretty low. How do humans behave in those perturbations?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The presented method is first in its kind to provide a provably robust perceptual metric. Authors have shown that this method is more robust than existing perceptual metrics (DreamSim). Several experiments are conducted to show the efficacy of the method. \n\nThe paper in general is well organized and well-written. There are also several experiments that show effectiveness of the method.", "weaknesses": "Although there has been some previous works on robust perceptual metrics, authors claim theirs is the first one with provable guarantees. I still think discussing why this matters in practice is important. I see the application for image retrieval, but how can someone have access to the model (white box) to actually attack it. Please elaborate.\n\nConsider changing the colors in bar plot of Fig. 3.a. They are hard to distinguish.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)