Understanding Metric Learning on Unit Hypersphere and Generating Better Examples for Adversarial Training
Authors: Yihan Wu
Heng Huang
Venue: ICLR 2022 Submitted (2022)
Keywords: Metric learning
Adversarial learning

Abstract: Recent works have shown that adversarial examples can improve the performance of representation learning tasks. In this paper, we boost the performance of deep metric learning (DML) models with adversarial examples generated by attacking two new objective functions: \textit{intra-class alignment} and \textit{hyperspherical uniformity}. These two new objectives come from our theoretical and empirical analysis of the tuple-based metric losses on the hyperspherical embedding space. Our analytical results reveal that a) the metric losses on positive sample pairs are related to intra-class alignment; b) the metric losses on negative sample pairs serve as uniformity regularization on hypersphere. Based on our new understanding on the DML models, we propose Adversarial Deep Metric Learning model with adversarial samples generated by Alignment or Uniformity objective (ADML+A or U). With the same network structure and training settings, ADML+A and ADML+U consistently outperform the state-of-the-art vanilla DML models and a baseline model, adversarial DML model with attacking triplet objective function, on four metric learning benchmarks.

Under review as a conference paper at ICLR 2022
UNDERSTANDING METRIC LEARNING ON UNIT HY- PERSPHERE AND GENERATING BETTER EXAMPLES FOR ADVERSARIAL TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Recent works have shown that the adversarial examples can improve the perfor- mance of representation learning tasks. In this paper, we boost the performance of deep metric learning (DML) models with adversarial examples generated by attacking two new objective functions: intra-class alignment and hyperspherical uniformity. These two new objectives are motivated by our theoretical and em- pirical analysis of the tuple-based metric losses the hyperspherical embedding space. Our analytical results reveal that a) the metric losses on positive sam- ple pairs are related intra-class alignment; b) the metric losses on negative sample pairs serve as uniformity regularization on hypersphere. Based on our new understanding on the DML models, we propose Adversarial Deep Metric Learning model with adversarial samples generated by Alignment or Uniformity objective (ADML+A or U). With the same network structure and training set- tings, our ADML+A and ADML+U consistently outperform the state-of-the-art vanilla DML models and the baseline model, adversarial DML model with attack- ing triplet objective function, four metric learning benchmark datasets.
INTRODUCTION
Deep metric learning (DML) has been applied to various computer vision tasks ranging from face recognition (Schroff et 2015 Liu al., 2017) to zero-shot learning (Romera- Parrdes-Paredes & Torr 2015 Bucher et al. 2016) and image retrieval (Song et al. 2016; Wu et al.) 2017). It has been proved to be one of the most effective methods for learning the distance-preservingg features of images. The intuition of DML is to pull the embedding of positive images pairs together and push the negative pairs apart, where the embedding function could be a deep neural network. Most of the metric losses in DML are tuple-based (Schroff et al. 2013 Song et al. 2016 Wu et al.) 2017 Wang et al. 2019) or classification- based (Movshovitz-Atiee Attias et al., (201) 7] Kim et al. 2020) Boudiaf et al., 2020 these different losses have been shown to achieve similar performance in the recent reviews
of DML (Roth et al.) 2020 Musgrave et al. 2020).
One common ground of existing DML. models is that the embedding space is unit hypersphere. It is widely known that achieving uniformity on hypersphere can increase the generalization of models and preserve as much information as possible (Bachman al.) 2019 Liu et al.) 2018: [02 Hjelm (et al. 2018), and the objective function that lead to uniformity is called uniformity regularization. Meanwhile, the downstream tasks in DML favor the models with small intra-class alignment (Wul et al. 2017 Wang et al.) 2019). In this work, we investigate these two properties, intra-class align- ment and hyperspherical uniformity (Wang & Isola 2020) for tuple-based metric losses. We derive the theoretical analysis for the triplet loss to prove that the triplet loss on the positive sample pairs minimizes the intra-class alignment by mapping all samples from one class to the same vector, while the triplet loss on the negative sample pairs achieves hyperspherical uniformity. We further conduct empirical studies to show that the same statement is also valid for other tuple-based metric losses.
We utilize our new understanding DML to design novel robust DML methods to enhance the
performance via improved adversarial training. Adversarial training aims at improving the robust- ness of models towards to certain types of attacks by training with perturbed samples. However, as shown in the recent work on contrastive representation learning (Jang et al.) 2020). adversarial training can also enhance the natural performance on the downstream classification task. Due to

---

Under review as a conference paper at ICLR 2022
the similarity between contrastive learning and deep metric learning, believe it's also possible to improve the nature performance of metric learning models with adversarial samples. Following new insights positive and negative metric losses, generate perturbations by attacking the alignment or uniformity objective, and create adversarial DML, models augmented with both nor- mal samples and perturbed samples. Our experimental results show that the new adversarial DMI. models can significantly boost the natural performance.
The major contributions of our paper can be summarized as follows:
We analyze the intra-class alignment and hyperspherical uniformity for tuple-based metric losses, and establish the connections between these two properties and the positive/negative metric losses.
Based on our new analysis and understanding, we propose two new adversarial DML models, ADML+A and ADML+U, via attacking the alignment or uniformity objective. ADML+A and ADML+U improve the natural performance on benchmarks significantly.
RELATED WORKS
Deep metric learning. There are mainly two kind metric losses in DML, tuple-based and classi- fication based losses. Tuple-based losses include contrastive loss (Hadsell et al. 22006), triplet loss (Schroff et al. 2013), margin loss (Wu et al. 2017), and multi-similarity loss (Wang et al., 2019). where the objective function is based the distance between positive pairs and negative pairs. In classification-base bbased losses, the learning objective is not depend on the positive or negative pairs but fixed (Boudiaf al. 2020) learnable proxy (Kim al., 2020p. In the recent reviews of metric learning methods (Roth et al., 2020) Musgrave et al.] 2020). it's concluded the improvement on the DML performance mainly due different training strategies and unfair comparison. The original contrastive loss and triplet loss still achieved comparable result with other metric losses under the same network and training strategies. In experiments we apply the training framework of (Roth et al. 2020) to ensure fair comparison.
Learning with hyperspherical uniformity. Hyperspherical learning regards learning tasks where the embedding space is a unit hypersphere. The uniformity of the hypersphere represents the diver- sity of vectors on the sphere. It encourages vectors be spaced with angles as large as possible so that these vectors can be evenly distributed on the hypersphere (Liu et al. 2018). Achieving hyper- spherical uniformity can help with preventing overfitting and improving generalization of the neural works (Liu et al., 2021). The objective functions which can lead to the uniformity on hypersphere are called uniformity regularization. Hyperspherical embedding is widely applied in representation learning tasks such as contrastive representation learning (Oord et al., 2018: [Hjelm et al., 2018) and DML (Wu et al. 2017; Liu et al. 2017). WWan & Isola (2020) showed that the objective function in contrastive representation learning optimizes for intra-class alignment and uniformity together.
Adversarial examples improves natural performance. In classification tasks, it is well known that the clean accuracy of adversarially trained model typically worse than the normal model. How- ever, Xie et al. (2020) showed that adversarial samples can be used to improve the clean accuracy of image classification models. According to (Jiang et al., 2020), training with adversarial samples can help improve the natural performance of contrastive learning models on the downstream classifica- tion tasks. The authors presented adversarial attacks based on the objective of contrastive learning and achieved improvement on both natural and robustness performance. It's believed that adversar- ial examples contain extra features, thus the generalization of models augmented with adversarial example is increased (llyas al. 22019; Salman et al., 2020; Xie et al. 2020), which contributes to better natural performance. Duan et al. (2018) utilized the triplet loss as the attacking objective to generate adversarial examples to improve DML models. To our best knowledge, this is the only one work using ad adversarial example to boost the natural performance of DML. In our work, we show that alignment and uniformity loss can generate stronger adversarial examples comparing triplet loss, and thus lead to better generalization of DML models.
3 ALIGNMENT AND HYPERSPHERICAL UNIFORMITY IN TUPLE-BASED METRIC LOSSES
In this section, we will study tuple-based metric losses on the unit hypersphere embedding space. We assume having n classes X,,..., X,, in training set and denote the encoder by f Rdd -
2

---

Under review as a conference paper at ICLR 2022
where Sk-1 is the surface of k-dimensional unit ball. Let Pdutal.) be the data distribution RR (:o be the distribution of positive pairs over Rd x R and ptri(:s...) be the distribution of triplet pairs over R I2 x IRd where the first two entries have the same label and the third entry sample from different classes. Please note that all detailed proofs included in supplementary material Appendix E) We also conduct experiments to validate our theoretical analysis, the details in Appendixx Cl
The major intuition of DML is to pull the representations of similar samples together and push dissimilar samples apart. Thus, reformulate the metric losses the combination of two parts:
Positive metric loss: minimizes the distance between embedded positive sample pairs.
Negative metric loss: maximizes the distance between embedded negative sample pairs.
Although in DML models the positive metric losses have different representations, they share one common optimal solution pattern, where samples from the same class are mapped to the same feature vector. Thus, we define the alignment loss with minimizing the intra-class distance.
Definition 1. (Intra-class alignment) The expectation of intra-class distance is given by:
Lalignment (f: X, Ppos) = ((zza))~Poo [Il[(x) - - [t)l
(1)
the minimum of this loss is achieved when the samples with the same label are encoded to the same embedding.
Proposition If the support set of the data distribution is connected and the support set of each class distribution is closed, the minimum of Colignment is reached when all samples are projected to the same vector.
In Sec. C.I we conduct the empirical studies verify our analysis. Results in Table 8 show that samples are roughly projected to the same vector if only the positive metric losses are used.
The negative metric losses aim positioning the embedding of dissimilar samples far as possible. However, because the embedding space of DML is unit hypersphere, where the maximum distance between two points is 2, it's not possible to separate all negative embeddings with large margin. Actually on 8k-1, the number of points with pairwise distance larger equal than /2 is at most 2k and the embedding dimension & is always much smaller than the number of feature vectors, thus it's impossible to make all distances between negative pairs exceed v2. Therefore investigating the properties of negative metric losses on the unit hypersphere is an interesting and important topic. We believe the negative metric losses are closely related to the uniformity on the hypersphere and our experimental results support this argument.
Definition 2. (Hyperspherical uniformity) The embedded samples should be evenly distributed on the spherical surface.
In practice, the hyperspherical uniformity can be achieved by optimizing the uniformity regulariza- tion. There exist many different representations of the regularization, and we utilize the hyperspher- ical energy (HE) (Liu et
2018):
Exrpeats N Pdata - 1 >
E(s, X)
(2)
Exopaates Paatt [ s=
- f(y)l(
and Gaussian hyperspherical energy (G-HE) Wang & Isola 2020):
(s, X) = log Br--paaee
* =I]
in the experiments for comparison. The values of HE and G-HE can also used as measurements on the uniformity of the embedded samples. We expect the value to be small in order to achieve good hyperspherical uniformity. We also want to mention that simply maximizing the distance between samples will not lead to hyperspherical uniformity, and the detailed discussion is in Sec. B.2
Because finding the optimal solution of the HE or G-HE problem is NP-hard [Lu et al. 2018). we not able calculate the exact position of vectors which are evenly distributed on the sphere. We provide a primary insight about how should finite vectors be uniformly distributed on unit hyper- sphere, and our conclusion is consist with the empirical results.
3

---

Under review as a conference paper at ICLR 2022
Since there exist many different tuple-based metric losses, analyzing all of them theoretically is impossible this work. In Sec. 3.1 we will provide the theoretical analysis of the triplet loss. The analysis of linear loss be found in See.. B.2 In Appendix C we will show the empirical results on four popularly used tuple-base metric losses to verify our statement.
3.1 TRIPLET METRIC LOSSES
In this subsection, provide our theoretical analysis the triplet metric losses with the following assumptions.
Assumption. Distributions Pdates Pposs Ptri should satisfy:
Random positive sampling: Vx, Ppos (I, y) = Pdaata (1r Pdata (y) Xx), where Pdtata (-1x2) ) is the
conditional pdf of Paota the of samples X similar i.e. Pdaata (- Pdata (bata(xx)
Pdnntal))
Random negative sampling: (tr
Ppos (2, v)Paaaa (x where Paata (x=)
Páuca I_ Paatu (x )a
Class- learning: Paato (X;) 1. then Ja- Pdaata (x )dx=
where x is negative sample w.r.t. and is the number of classes.
Definition 3. (Triplet loss)
Ceriptex (f, T) := E (2.g,= Joper [[llf(=) - f(u)l 11f(x) - +1)+))
++++]
(4)
Triplet loss can be rewritten into the form of naive linear loss with different distribution of triplets. We consider new distribution:
0, Cperi
when [1[(x>) [wv]1 - [1[(x) - f(x >>1 +
Peri
else,
where 1/B -1
then
Cerspter = Llinear (f; X, ((r))) F= [ll f()) |15())
Apparently the positive part of the triplet loss is minimizing the intra-class distance under new distribution Pere which has similar effect as the alignment loss with Ptri shown in the experiments Table 8). Now we focus on the negative part of the triplet loss.
Theorem 1. Denote the probability density function (pdf) of of (z = [1f(z) f(y) w.r.t.
y ~ Patoraa- (-1X,) by q(dl (l, Then the pdf of = [f(xx)) - (w) [ + with fixed x, x' q(u - T + d (x, x))), let S(x, x') = Jo q(u - T d(xxx))ddu E (0, 11, then
dxx
- ((.., [I] (11((())))))
f(x )1
n E
prrpaatn of ~' [1lf(z) ffxx'( (13
E (a. [1] - (
The negative triplet loss consists of two parts, where the first part dominants the second part be- cause n is always large in practice. The first part is actually a weighted unbiased regularization with weight S(x, x'). We think S(x, may help the unbiased regularization to achieve hyperspher- ical uniformity. Because the closed form of ((d"(r))) y)) is intractable, it's impossible to analyze S(x, x²) theoretically without any assumptions. We assume q(d (x,, y)) exponentially distributed and show the gradient flow of negative triplet loss is asymptotically equal the gradient of Gaus- sian hyperspherical energy. Therefore in this case the negative triplet loss can lead to hyperspherical uniformity.
Proposition Assume qxd v)) de Ledla),,) S(I,x') Le (ed((((2,2- and for the network parameter f, we have
A2 VoEo +
A
V wyy (11(()))))))))))))(()) [(
the negative triplet loss has asymptotically the same gradient Gaussian hyperspherical energy.
4

---

Under review as a conference paper at ICLR 2022
Despite the theoretical analysis, we also empirically show that the negative triplet loss achieves hyperspherical uniformity without any assumption on 5'z,z"). Besides, the negative part other metric losses also shown to achieve hyperspherical uniformity in our empirical study. The details is shown in Appendix C
In summary, the uple-based metric losses the unit hypersphere are closely related to intra-class alignment and hyperspherical uniformity. The positive metric losses target at minimizing the intra- class alignment and the negative metric losses try to keep all samples distributed uniformly on the hypersphere.
Connection to adversarial examples and adversarial training. The goal of adversarial examples is to fool the neural network by reducing the model performance. Attacking alignment loss, which positions the embedding of similar samples apart, or attacking the uniformity loss, which pulls dissimilar samples together, can definitely destroy the representation learned by DML models. Thus alignment and uniformity loss are suitable objectives for generating adversarial examples.
DESIGNING NEW ADVERSARIAL DML MODELS BASED ON BETTER UNDERSTANDING OF DML Loss
In this section, we introduce our new adversarial DML models: adversarial DML with alignment or uniformity objective (ADML+A or ADML+U), Before we introduce the details of our models, we share our motivation for designing ADML+A/U models by answering the following questions:
How can adversarial training helps improve DML models? One of the most reasonable explana- tions is that training with adversarial examples brings additional features to neural networks. For example, compared with clean images, adversarial examples make network representations more consistent with salient data features and human perception (Tsipras et al. 2018). Another possible reason is that adversarial examples can be regarded as data augmentation method, which prevents overfitting of the neural networks. Augmentation techniques which are similar to adversarial train- ing, c.g. using masking out VDevs & Taylor 2017) or adding Gaussian noise (Lopes et al. 2019) to regions in images, can help to achieve better performance on image recognizing tasks.
Why do we need new objectives for adversarial DML models? Based on our analysis in Sec. 3 the DML embedding of each image is depend all other positive and negative samples from per- spective of alignment and uniformity objective. Therefore we want to generate the adversarial sample x' for one image x; need push the adversarial sample away from the similar samples of x (maximize the alignment loss w.r.t. x²), or pull the adversarial sample close to the dissimilar samples of (maximize the uniformity loss w.r.t. x¹). Currently, the existing adversarial DML. models (Duan et al. 2018 Panum et al. 2021) generate adversarial samples by attacking the triplet loss (Definition 3). In this case, only one positive and one negative samples are used generat- objective
ing adversarial sample z', which is obviously less powerful than the alignment/unifome (which utilizes more positive or negative samples). Our experimental results in Table I also show the adversarial examples generated by alignnent/uniformity objectives is more powerful than the triplet objective. Thus it is critical to design new objectives for attacking DML models, which can take advantage of the representation information from more data samples.
Adversarial training. We first recall the standard tuple-based DML training setting. Denote the metric loss function by C(:; 0), where 0 is the model parameters, our learning objective is:
min E E -=-====>>>aa >>>>>=/>//p>> [C xt [)]
0
In the regular adversarial training framework (Madry et al. 2017), we train networks with perturbed samples from distribution p (ads)
min E (a,at _a- )opp(ad)) [C( x xz [x; ):
As our goal is to improve the DML performance on clean images by leveraging the regularization power of adversarial examples, we treat adversarial images as additional data augmentations and train networks with a mixture of adversarial examples and clean images. Our learning objective is
>xp at, x² 0)] + AE (ar,rt . (agplan))) [L((I, x+, a,,,,====>> (z)):)))
(5)
where À is the strength of the adversarial training.
5

---

Under review as a conference paper at ICLR 2022
Table 1: Performance of DMI. model evaluated adversarial samples generated by different objec-
tives. The threatened model is a pretrained Triplet-D model. Adversarial samples are generated from triplet, alignment or uniformity loss. Lower score indicates better quality of adversarial samples.
CUB200-2011
CARSI%
Online-products
Attack objective
R@1
NMI
mAP@C
R@1
NMI
mAP@C
R@1 1
NMI
mAP@C
No attack
62.40
67.21
23.56
77.59
66.64
23.83
77.53
89.98
41.12
Triplet 28.33
44.48
6.61
22.42
34.30
3.47
53.77
83.43
25.21
Alignment
8.71
23.42
0.46
10.99
17.11
0.55
8.82
80.05
1.49
Uniformity 13.47
32.87
3.54
14.30
24.07
2.05
4.68
79.90
0.51
Generate adversarial samples. We use to PGI-FSGM (Madry al., 2017) method for generating adversarial samples. We consider DML models require class-balanced batches for training, and propose to generate perturbations by maximizing the intra-batch alignment or uniformity. Given a batch S and a sample x, the loc FSGM adversarial sample of a generated by maximizing the alignment objective is given by,
ADV(x) (z.)) + a V wawaaiggmenn
where (x,e) is the projecting function the Lo ball centering with radius c, and is the attack strength. Analogously, the adversarial sample generated by maximizing the uniformity
objective (Eq. 2 and Eq. 30.3ss is
ADV(z) [., (x. + V Cnnnsforrnttt (f;
In PGD-FSGM method, we will update the adversarial samples iterative by x+++ = ADV(x()) for
L steps. The output perturbed samples (2( will be used in our adversarial training objective Eq. S In ADML-A
alignment loss (Eq.. 1) generate adversarial samples and in ADML-U
Gaussian uniformity loss G-HE (Eq. 3). We include the algorithm of ADML-A and ADML-U in the appendix (Alg. and Alg. 2}. In experiments, we apply multi-similarity losses as the metric loss L with attacking alignment and uniformity objective, both models achieve significantly better results on benchmarks (Table and Table 3).
EXPERIMENTS
In our experiments, we first conduct the empirical studies to verify the theoretical analysis results in Sec. 3 (the results are discussed in Appendix C). After that, we show that alignment and uniformity objectives can help to generate better adversarial examples than the triplet loss, then we compare the natural and robust performance of our adversarial DML model with the state-of-thh-arr art methods. The natural performance is the performance of DML models evaluated with clean samples, while the robust performance evaluated with adversarial samples.
5.1 EXPERIMENTAL SETUP
Datasets. We test our model on four DML benchmarks, CUB200-2011 (Wah et al. 2011
CARS196 (Krause et al., 2013), Online-product (Song et al., 2016). and In-shop (Liu et al. We follow the previous work (Song et al. 2016) and (Liu et al. 2016) for the train-test split. The statistics of these datasets is introduced in Sec. D.2
2016
Training Frameworks, In all experiments, we use the DML framework from (Roth et al.. 2020) for training. This framework enables us to train and evaluate DML. models under the same settings and ensure fair comparison of the model performance. The backbone network ResNet50 (He et al., 2016) with ImageNet pretrained (Krizhevsky et al.) (012)) and frozen Batch-Normalizate the embedding dimension of samples is 128. The initial learning rate is 0.00001 with no scheduling 24GB Nvidia Tesla P40.
layers,
and the batch size is 1 12. Experiments are performed
Baseline models. We compare ADML+A/U with the state-of-the-an DML models. First we select three of the best DML models according to (Roth al. 2020): margin loss with distance sampling (Wu et al. 2017). multisimilarity loss (Wang et al.) (211)) and triplet loss with distance sampling. Besides, we take another two SOTA DMI. models which were published recently but not included

---

Under review as a conference paper at ICLR 2022
Table 2: Performance metric learning models on clustering and retrieval tasks averaged over runs on CUB200-2011 and CARSI96. Our Adversarial DML models, ADML+A, and ADML+U, outperform the rest models. The model settings and training parameters are same for all models.
CUB200-2011
CARS1%
Models
R@1
NMI
mAP@C
R@1
NMI
mAP@C
ImageNet pretrain
43.77
57.56
8,99
36.39
37,96
4.93
Linear
38.42
43.28
7.64
32.45
35.12
3.48
Triplet-D
62.31 = 0.41
67.23
0.34
23.29 0.25
79.08
0.41
66.02
24.02 #: 0.31
Margin
62.42 0.36
0.449
23.54=0.21
78.11 =
66.87
23,94 0.27
Mulli-Similarity
62.73 = 0.61
67.45 0.39
22.65
79.94 + 0.28
0.228
67.59 0.43
24.12 + 0.25
Proxy-Anchor
0.446
0.37
2.321
0.33
6..31
23.86 # 0.26
Cross-Entropy
61.58
6..37
2.22
78.41
6.33
23.63 + 0.34
Info-NCE
0.51
66.91
2224
0.37
0,57
23.41 + 0.22
ADML+T
0.44
68.13
2.305
80.88
66.5
23.91 0.39
ADML+A
66.02
6.37
2.24
=1.95
67.97 0.49
24.21 = 0.28
ADML+U
65.46 0.40
68.60
24.58 0,28
# 0.36
68.21 0.35
24.82 # 0,34
ADML+A+U
64.24 # 0.38
67.73 # 0.45
23.88 + 0.26
80.95 0.41
67.64 # 0.39
23.85 + 0.30
in Roth et al.
2020 work: Proxy-Anchor loss (Kim al. 2020) and Cross-Entropy loss (Boudiaf
et al. (020). Next we apply the Info-NCE loss (Wang & Isola| (020), which is contrastive learning objective, as one of the baselines. Finally we compare our models with the only existing adversarial DML model (Duan et al. 2018 ADML+T (triplet objective), in the experiments.
Evaluation Metrics. We measure the performance of DML and ADMI, models with Recall @k (R@k) (Jegou et al., 2010), Normalized Mutual Information (NMI) (Christopher et al. (0008) and Mean Average Precision measured on recall (mAP@k) (Musgrave et al. 2020). The details of these metrics are introduced in Sec. D.3
5.2 COMPARE THE QUALITY OF ADVERSARIAL EXAMPLES WITH DIFFERENT OBJECTIVES
The threatened model a pretrained DML model with triplet loss and distance sampling (Wu al.) 2017). which is one of the most competitive DML models according Roth al. 2020). We consider three diffferent attack objectives, triplet loss (Eq. 4), alignment loss (Eq. 1.. and Gaussian hyperspherical uniformity (Eq. 3). for generating adversarial samples. We use loc PGD- FSGM attacks with strength = 0.0314, L = 7 steps and step size 0.007, we keep this settings for all three attack objectives our experiments.
Settings.
Results. In Table I the adversarial samples generated by alignment or uniformity objectives are significantly stronger than the samples generated by triplet loss. This indicates that adversarial sam- ples from alignment or uniformity contain more features that not captured by the vanilla DML models. Thus we believe the Adversarial DML model with alignment or uniformity objectives could be more generalized than the vanilla DML models ADML with triplet loss. Our experimental re- sults in Table 2] and Table 3] which show that ADML+A and ADML*U outperform the baseline models metric learning tasks, also support
analysis. We also notice that using both A and U in ADML have similar performance as ADML+T, which suggests we should use the attack objective (A or U) separately. In Appendix Fiwe also shows the T-SNE plot of the embedding generated by the vanilla DML and ADML+A, which shows ADML+A can better separate different classes.
5.3 ADVERSARIAL DML MODELS IMPROVE NATURAL PERFORMANCE
Settings. We train all DML models for 100 epochs. For our adversarial DML models we apply ADML+A and ADML+U, The adversarial training strength À in ADML+A/U is 0.1 for CUB200- 2011 and 0.15 for CARS196. For generating adversarial examples we use lax PGD-FSGM attacks with strength 0.0314, L 7 steps and step size 0.007. For Online-produch and In-shop we use À 0.005, = 0.01, L = 5, and a 0.003. The ImageNet model is the model only with ImageNet pretrain.
Results. In Table our ADML+A and ADML+U models outperform the SOTA metric learning models. ADML+AU improve the R@1 over 1% CUB200-2011 and CARS196, and also have considerable improvement on Online product and In-shop dataset. Besides, the performance of ADML+A/U under the NMI and mAP@C metrics is also comparable better than the SOTA.

---

Under review as a conference paper at ICLR 2022
Table 3: Performance of DML models on clustering and retrieval tasks averaged over runs on Online-products and In-shop. Our Adversarial DML models, ADML+A, and ADML+U, outperform the rest models in most cases. The model setrings and training parameters are same for all models.
Online-product
In-shop NMI
Models
R@
NMT
MAP@C
1
mAP@C
ImageNet pretrain
48.51
84.24
17.37
21.62
76.53
4.02
Linear
20.53
81.20
5,78
16.03
75.81
2.47
Triplet-T)
77.41 = 0.19
90.04 a 0.05
41.05 =
87.31 # 0.18
89.76 + 0.09
28.45 + 0.17
Margin
77.66 0.14
89.93
0.06
0.12
0.155
0.07
28.45
Multi-Similarity
7
90.00 0.04
41.39
87.33
89.85 0.12
# 28.11 0.11
Proxy-Anchor
0.13
89.90 0.05
00.1
8..14
89.41 0.05
Cross-Entropy
76.92
89.82 0.11
4..41
86.75 0.32
89.71 0.133
28 38 0.51
Info-NCE
0.15
8.04
39.42
8.124
89.62 0.04
27 94 0.17
A
77.13 7.13
89.59 + 0.03
0.075
87.47 0.12
89.65 + 0.10
29.05 + 0.12
ADML+A
78,12 0.16
0.15 0.04
0.19
89.93 + 0.05
30.12 0.15
ADML+U
78.01 0.12
89.97
4012
0.18
89.57 0.08
29.93 0.16
ADML+A+U
77.41 0.15
89.88 0.07
0.14
67.65
89.71 0.09
29.72 0.13
0.70
0.70
Rocal
0.68
Racall@1
0.80
Racallel
0.66
Recalle11
0.66
Recel
0.64
0.70
0.65
0.62
0.62
0.60
0.60
0.60
KMI
Recalig1
0.65
025
0.50 Lambda
0.50 Lambda
0.75
0.0
0.
3.4
0.6
0.0
0.0
0.2
0.4
36
0.0
Lambda
Lambda
(a) CUB-ADML+A
(b) CUB-ADML+U
(c) CARS-ADML+A
CARS-ADMLAU
Figure 1: Performance of ADML+ and ADML+U with different adversarial training strength on CUB200-2011 and CARS196.
Since all the models are training under the same framework and settings, we can conclude that adversarial training helps to enhance the natuaral performance of DML models.
5.4 ROBUSTNESS PERFORMANCE OF ADVERSARIAL DML MODELS
We evaluate the robustness performance of our ADML+A and ADML+U models against attacking the alignment objective, which is the strongest DMI. attack according to our experiment in Sec. 5.2 on CUB200-201 and CARSI 196 datasets, For baseline models we use margin, multi-similarity, and ADML+Triplet models. The settings of alignment attack are the same the settings in (Sec. 5.2 the settings of ADML+A/U and baseline models are the same as the settings in Sec. 5.3 As seen in Table 4 ADML+T has slightly better performance than the vanilla DML models, while ADML+A and ADML+U outperform the baseline models with large margin under the alignment attacks. Notice ADML+A takes the advantage of adversarial training with alignment loss, it's reasonable that the performance ADML+ is slightly better than +DML+U under alignment attacks.
Table 4: Robustness performance of DML models and ADML models against adversarial samples generated by attacking alignment loss.
CUB200-200 2011
CARS196
Products
Models
R@
NMI
C
R@
NMI
mAP@C
R
NMT
mAP@C
Margin
8.04
24.59
0.58
8.39
17.17
0.56
8.27
80.02
1.34
Multi- somli--imillrity
8.90
24.13
0.51
11.95
18.32
0.55
8.93
80.06
1.45
ADML+T
11.58
25.26
0.74
25.35
21.22
1.31
10.69
80.20
1.74
ADMI +A
17.37
29.16
1.27
39.97
26.05
2.54
13.98
80.40
+U
15.10
27.89
1.06
33.09
24.48
2.30
11.32
80.29
1.85
8

---

Under review as a conference paper at ICLR 2022
Table Robustness with different adversarial training strength À on CUB200-2011 under the align- ment attacks. The metric is Recall @ I.
0.2
0.4
0.6
0.8
1.0
ADML+A
8.90
19.15
22.51
24.47
25.29
25,93
ADML+U
8.90
16.21
18.36
20.23
21.06
21.55
5.5 ABLATION STUDY
Strength of adversarial training. In this part, we evaluate the effect of adversarial training strength on the performance of ADML+A and ADML+U. The backbone is DML with multi-similante loss. We expect the performance will first increase with À then decrease, because when À is close to 0, the ADMI. models can hardly learn adversarial features and the improvement small, when is large, the adversarial features will dominate the DML models and the performance on clean features will be poor. The model settings are the same as in Sec. 5.3) The experimental results illustrated in Fig. are consistent with our analysis, when increasing the adversarial training strength, the natural performance of ADML+A and ADML+U is first improved then decreased. Table 5 shows the robustness of ADML+A and ADML+U model with adversarial training strength À and alignment attacks. Both models become increasingly robust against alignment attacks.
ADML on different metric loss. In this experiment, apply our ADML approach with triplet, alignment, uniformity, and alignment+uniformity objectives on different metric losses, including triplet, margin, multi-similarity, and info-NCE losses. The metric is Recall The model settings are the same as in Sec. 5.3] From [abll 6) we can observe that all ADML methods boost the performance of all DML losses. ADML+A achieves the most significant improvement across all attacks and metric losses.
Table 6: @ different metric learning losses with ADML methods on CUB200-2011
Vanilla
ADML+T
ADML+A
ADML+U
ADML+A+U
Triplet
62.29
63.68
65.11
64.72
63.17
Margin
62,48
64.26
65.92
65.61
64.32
Multi-similarity
62.71
64.45
66.13
65.58
64.39
Info-NCE
61.42
62.74
64.01
63.85
62.91
A mixture of alignment and uniformity attacks. In this experiment, we study the effect of different weight of alignment and uniformity objectives in ADML+A+U, The backbone is DML with multi- similarity loss. The model settings are the same as in Sec. 5.3, Specifically, we assign weight (1 3) the alignment objective and the uniformity objective. The results are shown in Table 7 When 99 increases, the performance of the ADMI model first decreases and then increases, which indicates that attacking alignment and uniformity loss separately can lead to better results.
Table 7: Recall @ of ADML+(1 3)A+3
3 U CUB200-2011 with different B.
0
0.2
0.4
0.5
0.6
0.8
I
ADML+(1 (1ML=(1-B)AH33] 66.13
65.81
64.67
64.39
64.55
65.33
65.58
CONCLUSION
In this work, we investigated two important properties, intra-class alignment and hyperspherical uniformity, of tuple-based metric losses on unit sphere. According to our theoretical analysis and experimental results, the positive metric losses contribute to the intra-class alignment and the neg- ative metric losses achieve hyperspherical uniformity. Based on our new understanding, we design two novel adversarial DML models, ADML+A and ADML+U, where the perturbations are gen- erated by maximizing the alignment loss or the uniformity loss. Our ADML+A and ADML+U improve both of natural and robust DML performance by enhancing model generalization. Poten- tial future work directions include analyzing other tuple- based metrie losses theoretically, designing new metric losses based on the alignment and uniformity, and exploring the trade-off between nature and robustness performance in metric learning.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
In this paper, we design new adversarial deep metric learning (DML) models for improving the vanilla DML. Our method can boost the robustness of neural networks again adversarial attacks, which contributes the trustworthy AI and machine learning.
REPRODUCIBILITY
All datasets, baseline models, general training settings provided in [Sec. 5.1 For specific tasks we also include the detailed settings in the corresponding sections. For example, the detailed model structure and hyperparameter settings for the natural performance of ADMI, models are written in We will release the code if the paper is accepted.
Sec. 5.3
REFERENCES
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv:1906.00910, 2019.
Malik Boudiaf, Jéróme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. A unifying mutual information view of metric learning: cross-entropy vs.
pairwise losses. In European Conference on Computer Vision, pp. 548-564. Springer, 2020.
Maxime Bucher, Stéphane Herbin, and Frédéric Jurie. Improving semantic embedding consistency by metric learning for zero-shot classiffication. In European Conference on Computer Vision, pp. 730-746. Springer, 2016.
D Manning Christopher, Raghavan Prabhakar, Schütze Hinrich, et al. Introduction to information retrieval. An Introduction To Information Retrieval, 151(177):5, 2008
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint vrXiv:170..0455 2017.
Yueqi Duan, Wenzhao Zheng, Xudong Lin, Jiwen Lu, and Jie Zhou. Deep adversarial metric learn- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 2780-2789, 2018
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- tion (CVPR'06) 06), volume 2, pp. 1735-1742. IEEE, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildoo Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1808.0667, 2018.
Andrew Ilyss, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. arXiv preprint teXivv/905.022 2019.
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 33(1):117-128, 2010.
Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adversarial contrastive learning. arXiv preprint arXiv:2010.13337,, 2020.
G. A. Kabatjanskii and V. I. Levenstein. Bounds for packings on a sphere and in space. Problemy Peredachy Informatsii, 1978.
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 2238-3247, 2020.
10

---

Under review as a conference paper at ICLR 2022
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- ane-graindd categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3), Sydney, Australia, 2013.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 212-220, 2017.
Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. arXiv preprint / 09298, 2018.
Weiyang Liu, Rongmei Lin, Zhen Liu, Li Xiong, Bernhard Schölkopf, and Adrian Weller. Learn- ing with hyperspherical uniformity. In International Conference on Artificial Intelligence and Statistics, pp. 1 1180-188 PMLR, 2021.
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin Cubuk. Improv- ing robustness without sacrificing accuracy with patch gaussian augmentation. arXiv preprint 02269066061,,, 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1706.06883, 2017.
Yair dovshovitz-Attias Alexander Toshev, Thomas K Leung, Sergey loffe, and Saurabh Singh. No fuss distance metric learning using proxies. In Proceedings of the IEEE International Conference on Computer Vision, pp. 360-368, 2017.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In European Conference on Computer Vision, pp. 681-699 Springer, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 2018
Thomas Kobber Panum, Zi Wang, Pengyu Kan, Earlence Fernandes, and Somesh Jha. Exploring adversarial robustness of deep metric learning. arXiv preprini arXiv:2102.07265, 2021.
Bernardino Romera-Paredes and Philip Torr. An embarrassingly simple approach to zero-shot learn- ing. In International conference on machine learning, pp. 2152-2161. PMLR, 2015.
Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Bjorn Ommer, and Joseph Paul Co- hen. Revisiting training strategies and generalization performance deep metric learning. In International Conference Machine Learning, pp. -2252. PMLR, 2020.
Hadi Salman, Andrew Iyyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adver- sarially robust imagenet models transfer better? arXiv preprint arXiv:2007.08489 2020.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 815-823, 2015.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In IEFE Conference on Computer Vision and Pattern Recognition (CVPR). 2016.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. There is no free lunch in adversarial robustness (but there are unexpected benefits). arXiv preprint arXiv: 1805. /2/22, 2(3), 2018.
11

---

Under review as a conference paper at ICLR 2022
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD 1 Dataset. Technical report, 2011.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In International Conference on Machine Learning, pp. 9929-9939. PMLR, 2020.
Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. Multi-similarity loss with general pair weighting for deep metric learning. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 5022-5030, 2019.
Chao-Yuan Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl. Sampling matters in deep embedding learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2840-2848, 2017.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoc V Le. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2020.
12

---

Under review as a conference paper at ICLR 2022
APPENDIX A CODE
wtt. We use the open source DML training framework htttp://gittttii www.wtthucccoooooooooooooo fusez us/ / Deep Metric Learning Baselines, We include the code of our AAgg. [ Alg. in the supplementary material. All datasets used in this paper are available online. We will release the full code package if our paper accepted.
APPENDIX B THEORETICAL EXPLANATION OF POSITIVE AND NEGATIVE METRIC LOSSES
B.I NUMBER OF NEARLY ORTHOOGNNAL VECTORS IN HIGH DIMENSIONAL SPIIERE.
Lemma 1. Kaabatjankki-Levensae Levenstein bound (Kabatjanski Levenstein
1978 For a hypersphere
Sk-1 RR there exist at least kM vectors with pairwise distance in the range of v2+0(1
M
k
Lemma shows the number of nearly orthogonal vectors we can take from high dimensional sphere. k 128 and M = 3, the amount of nearly orthogonal vectors could be more than one million, which exceeds the volume of benchmark datasets X in DML lot. Following the former discussion, we know the pairwise distance cannot be larger than v2 for all negative pairs. Therefore we think the uniformity regularization with DML benchmarks will lead to an embedding space where the feature vectors nearly orthogonal. The plots of pairwise distance distribution in Fig. with hyperspherical regularization also support our analysis.
B.2 NAIVE LINEAR METRIC LOSSES
With the same Assumption Sec. 3.1 we can prove that the positive part of the linear loss minimizes the intra-class distance and the negative part maximizes an unbiased term, which doesn't 't achieve hyperspherical uniformity. The theoretical analysis results are summarized in the following theorem. Definition 4. (Naive linear loss)
Liinear (f; X. Ptri) == Ra Iupert [1lf(x) f(0)1 11f(x) f(x |
Definition 5. (Unbiased regularization)
Lunbiased ( X, Pdata) = 1[ (x)
the minimum of this loss is reached when the centroid coincide with the origin.
Theorem 2. Naive linear loss consists of alignment loss and unbiased loss with a constant multi- plier.
Positive part: Ea )>pper [11((z) [())11 Lolignment
Negative part: - E Jmpiri [llf(x)) f(x 11
(2Lunbiased 2) + L Colignment
Combining them we have Liinear n (2Cunbiased + Calignement 2).
Because the number of classes n is always large and the magnitude of Lunbiased - and Calignmentt are similar, in negative linear loss the dominant objective is the unbiased regularization. Thus simply maximizing the distance between negative pairs will lead the unbiased regularization instead of hyperspherical uniformity. In Figure 2(e) and Figure 2(c) of the appendix, we can observe the difference between the models with negative linear loss and uniformity regularization. We also show the interesting connection between the naive linear loss and linear discriminant analysis (LDA) in
Sec. E,5
Remark. Naive linear loss doesn't t work all in practice (see the experimental results of Linear model in Table 2 and Table 3). Based on our analysis, we believe it's because linear loss doesn' 't op- timize the hyperspherical uniformity. In the next section, will introduce of triplet loss, which is a simple variant of linear loss. We find triplet loss optimize the hyperspheri- cal uniformity, which could be the reason that triplet loss works well empirically.
theoretical analysis
13

---

Under review as a conference paper at ICLR 2022
APPENDIX C EMPIRICAL STUDY OF POSITIVE AND NEGATIVE METRIC LOSSES
In this subsection, study the effect of tuple-based metric losses on positive negative pairs em- pirically, and focus on the four tuple-based losses: naive linear, triplet, margin, and multi- similarity loss. In experiments train all DMI. models with either positive metric losses negative metric losses.
C.I DML MODELS WITH POSITIVE METRIC LOSSES
Settings. We train 4 DML models with linear, triplet, margin, and multi-similarity losses the positive sample pairs. The gradient flow of negative pairs is stopped. We train all models for 50 epochs. We compare the average distance between positive/negaaive/all pairs of embedded samples.
Results. In Table 8, the DML models trained with only positive metric losses have average pairwise distance close O. Therefore, minimizing the intra-class alignment will lead model which maps all samples to the same feature vector.
Table 8: Comparison of DML models trained with the positive metric losses CUB200-2011 and CARS196. We calculate the average distance (Avgdist), average distance of positive pairs (Avgdist- Pos), and average distance of negative pairs (AvgdistNeg) with the embedded samples.
CUB200- 2011
CARST96
AvgdistPos
AvgdistNeg
Avgdist
AvgdistPos
AvgdistNeg
Avgdist
Linear 2.010e-3
3.190e-3
3.178e-3
1,497e-3
3.162e-3
3.183e-3
Triplet
1.934e-3
3.187e-3
3.174e-3
1.476e-3
3.163e-3
3.184e-3
Margin
7.565e-2
8.200e-2
8.192e-2
3.195e-2
3.468e-2
3.465e-2
MS
2.558e-3
3.324e-3
3.316e-3
2.242e-3
3.264e-3
3,277e-3
C.2 COMPARE NEGATIVE METRIC LOSSES WITH UNIFORMITY REGULARIZATION
Settings. We compare 8 different models in this experiment, including the original model with only ImageNet pretrain; b) four models trained with linear, triplet, margin, and multi-similarity losses on negative pairs; c) three models trained with HE(s=0), HE(s=1) [Eq. and G-HB(s=1) [Eq. 3) regularization, those regularization functions are introduced in _ppppeeddx C We train all models with 50 epochs. For the models with negative metric losses, the gradient flow of positive pairs is stopped.
Evaluation Metrics. We use the regularization score on HE(s=0) (Eq. and G-HE(s=1) (Eq. 3)
to measure the uniformity of embedded samples on hypersphere. Smaller score indicates better uniformity. We also check the average of pairwise distance of different models, we expect it to be close to 2 for good hyperspherical uniformity.
Results. In experiments we compare the regularization strength of negative metric losses with HE and G-HE Fig. and Fig. illustrates the pairwise distance the test samples on CUB200-2011 and CARSTE 196 dataset. The DML models with negative metric losses have similar pairwise dis- tance distributions as the uniformity regularization methods, and the only exception is the negative naive linear loss, which will not lead to the hyperspherical uniformity based on our analysis in Sec. B.2 We also include the distance distribution before training (Figure 2(a)) as a reference. Be- sides, also compare those models under the hyperspherical uniformity metrics. In Table 9 the negative metric losses achieve comparable results with the uniformity regularization methods. The G-HEE(==1) outperforms the rest models.
14

---

Under review as a conference paper at ICLR 2022
Table 9: Comparison of DML models trained with the negative metric losses and uniformity regu- larization on CUB200-2011 and CARSI96, All negative metric losses except Linear achieve com-
parable performance to uniformity regularization. The details of models and training settings are in
Sec. C.2
CUB200-2011
CARSI96
Avgdist
HE(s=0)
G-HB(s=I)
Avgdist
HE(s=0)
G-HB(s=1)
ImageNet
0.7220
0.3368
0.5939
0.6557
0.4326
0.6497
Lincar
1.2398
-0.1670
0.2535
1.3194
-0.1686
0.2449
Triplet
1.3877
-0.3254
0,1496
1.4039
-0,3376
0,1420
Margin
1.3929
-0.3297
0.1465
1.4016
-0.3363
0.1426
MS
1.3825
-0.3221
0.1509
1.3985
-0.3338
0.1441
HE regularization (s=0)
1.4001
-0.3347
0.1438
1.4064
-0.3395
0,1411
HE regularization (s=1)
1. 1.4009
-0.3355
0.1432
1.4068
-0.3398
0.1407
G-HE regularization (s=1)
1.4028
-0.3369
0.1424
1.4077
-0.3406
0.1402
0.5
1.0
15
2.
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
15
dist sce
distacce
distence
distance
(a) ImageNet
(b) G-HE (s=1)
(c) HE (s=0)
(d) HE (s=1)
05
3.0 dist3/ce
15
2.0
0.0
3.5
0.0
3.5
0.0
1.0
155
dstance
disznnce
distance
(e) Linear
(I) Triplet
(g) Margin
(h) Multi-similarity
Figure 2: Illustration of pairwise distance distributions of the embedded CUB200-2011 samples
generated by DML models trained with negative metric losses or uniformity regularization. The details of models and training settings are in Sec. C.2
15

---

Under review as a conference paper at ICLR 2022
0.5
2.2
1.5
0.0
1.5
0.0
1.0 distance
diviance
distance
datance
(a) ImageNet
(b) G-HE(s=1)
(c) HE((=0)
(d) HE((=1)
0.5
1.0
1.5
20
0.0
0.5
1.0
3.5
2.0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
sista/cce
distncce
distence
distance
(e) Linear
(I) Triplet
(g) Margin
(h) Multi-similarity
Figure 3: Illustration of pairwise distance distributions of the embedded CARS196 samples with negative metric losses or uniformity regularization. The details of models and training settings are in Sec. 5.3.
APPENDIX D ADML ALGORITHMM AND EXPERIMENTAL SETTINGS
D.1 ALGORITIM OF ADML+A AND ADML+U
Algorithm ADML+A
Input: training set number epochs N; original classifier f; weight adversarial training A; PGD attack step L; PGD attack strength
Initialize class balanced sampler
for E epochs do for mini-batch (S1, ..., Sn) do
5(0) SS
Generate adversarial samples of S with PGD-FSGM attack for E 0: L do
g(I+1) IT (S (S0))) (5 + a sin Latignment (f))(())
end
Sado = SL))
Calculate objective function
Ltotat = L(f, 5) + XCf Sadv);
L can be an arbitrary metric loss, in our paper we use margin and multisimilarity loss Update network parameters of f with Lrotati
end
end
16

---

Under review as a conference paper at ICLR 2022
Algorithm 2 ADML+U
Input: training set X; number of epochs N; original classifier f: weight of adversarial training A; PGD attack step L; PGD attack strength
Initialize class balanced sampler
for E epochs do for mini-batch
3 do
=( S:
Generate adversarial samples of S with PGD-FSGM attack for E E0: do
g(++ Bx (S ()) + a sun Sus (f
end
Sado S(L)
Calculate objective function
Ciorat L(f. S) + Sodo)
C can be an arbitrary metric loss, in our paper we use margin and multisimilarity loss
Update network parameters of f with Liotal;
end
end
D.2 DATASET DETAILS
CUB200-2011 contains 200 species of birds and 11,788 images (Wah al. 2011). We use the first 100 species as training set and the rest test set.
CARS1% has 196 models of cars and 16,185 images. (Krause et al. 2013). We use the first 98 models training and the rest as test set.
Online-product includes 22,634 classes of products and 120,053 images (Song et al. 2016 We use the first 11,318 classes as training set and the rest as test set.
In-shop contains 7982 classes of clothing and 54,624 images (Liu al. 2016). We the first 3,997 classes as training set and the rest as test set. The test set is further partitioned into a query set with 14,218 images of 3,985 classes and gallery set with 12,612 images of 3,985 classes.
D.3 EVALUATION METRICS
We use the following metrics to evaluate the DML models with retrieval and clustering downstream tasks.
Recall @k. For the retrieval task we apply the Recall @k (R) metric (Jegou et al., 2010). For test set M = ((21,y))- (In, yr }.. the indices of the first & nearest neighborhood of sample o, is given by (x,) arg |
Ejes, [I/ >
[2x2)11, and
R@k 1
(x))
NMI. We use Normalized Mutual Information (NMI) (Christopher et al., 2008 to measure the quality of the clustering task. We use K-means to generate the clusters of the embedded samples, then we calculate the label assignment = {?2..... from clustering. Denote the ground truth labels by Ω [y.,.., ..., yo the NMI is computed
NMI(Ω, r) = I(2, ((2,))/R((()))) + H(T)),
where -(:,-) is the mutual information function and H(-) is the entropy function.
mAP@C C. According to (Musgrave et al. 2020), we also include mean average precision measured additional metric. We first compute the recalled samples, which determined by the nearest neighbour ranking. Then compute the mAP-score follows the standard mAP procedure. MAP@C C. is the mean over the class-wise average precision @ k where k. is the
recall (mAP@k)
17

---

Under review as a conference paper at ICLR 2022
number of samples in class c, which means only recall ke nearest neighbour. Following the notation of Recall@k, the value mAP@ C is given by
mAP@C ΣΣ dEC My=C
[x
Ske (xg)) = 84
kic
APPENDIX E MISSING PROOFS
E.1
PROOF OF PROPOSITION 1
Denote the support set the distribution of each class by , Sn. according to Definition the minimum of alignment loss is reached when the encoder f maps all samples in class to the same feature vector iee. Vi, (S,) = {u;}. For arbitrary j. because - Sg is connected and each Sk is closed, we can select set sequence Skgs--.. Sk.. such that Sk S Sk.. Sj and Sa, 0 TK * 0,1 E [m- - 1]. By Sk, 5k * 0 we have Uk, U for all / E [m - 1]. thus vi, f. (S,). So all samples are projected to the same feature vector.
E.2 PROOF OF THEOREM 2
Proof. Recall that the naive linear loss is given by
Litnear (J; Ptrss) ) - s(v)l12 [1f(x)) f(x |
Consider the positive part
(a J~men. [x) f(y) = (a.a)) [1lf(z)) - f(v)l(? = Lalignment
(8)
Consider the negative part
(B ><pm.. [1lf(())) - f(x 1 (a. >>pm..l(e)) f(x))f((")) [[x]]] -
and
(a [xx){((z))
-E(aw) [fx)) ~paaa [f(x))]
1
= -Ep-paraa lf(x)²
J2- Paaaa (x= )dx (E maace [f(z')] - Ex' puraa (f(z')
E (peppson (f(x)² T Ex reparna [f(x']] Exupaare (z) "E, Be [f('x)
71 (Ex-paaro (fx)) T E [f(x)]]
(+Xx,) Euzppani [ (x)² Pdata ( (1xx)) (f('))))
71 Ear- [f(x)) T
R Bar [f(x)] n R F (f) [(y)
n1 Em--
Eam [f(x)] (
n P (1x))
(( s())
Lunbinsed 2(n 1) (Lalignment 2)
(9)
where we denote the class of sample x by X Combining Eq. and Eq. 9 together, we have Llinear (f)
2n n Lunbiased
n (Calignment 2) -2 2 + Lalignment
(10)
(2Cunbinsed Latignment - 2)
n
n
18

---

Under review as a conference paper at ICLR 2022
E.3 PROOF OF THEOREM 2
Proof.
The triplet loss is
Liriptet (F, = Lisnear (f; Peri) = -
(u.,v. [() f(v)l -1f(2))) -
(u,ue )mpee [l]((z) ht0p:-1hh000---10122120000000110
Laana Jopens [Ilf(r)) - f
Consider the negative part,
Blagg )aptn [1lf(x))) f(
E Paate Paan [1lf(x) - f(x ) -1 -11
B Br-pesto Paata [1lf(x)) - / (a |1
n
+
Bx Bi-piate Pante
[1l[(x) - /(x')l Ex, / hww.:///a/a///a/uoooooo/oo-o---0
n
E Paata x' ~
[1lf(x) yr h1tp::/:000...aaa/aa/a/.//...a.0
71
1
1
E (x,aa) [1f((x) |1000:000000000.aa/aaaaaa/a//a-0
71
π1
B Berpestt Paain [1lf(x)) - f (xx'11B((x,²²)
n 1
1
B [1](s)) [(x') ((="))_, x'))
+
n
where X is the set of samples have the same label The last equation is based
S(z,z)= = / q(u + dZ, - r) = =Bg(P((x))) = Ey-pacca x.)1+(w2op]
E.4 PROOF OF PROPOSITION
Proof. By ((d(x,)) y)) }e (x, then
_e Ad the pdf of a d(x,,) - d(xxx') + r
S(x,2')
1 - 1rd c-Alute((x2)) A
du - A(dd"zz
A
Consider the gradient of the negative triplet loss B( )op,, [Ilf(x) - f(x f(x))11)). during training
we first sampling from ph,, then calculate the gradient. In this case the actual gradient flow is given by
B Jopp,n [x)ls(>) f(x
19

---

Under review as a conference paper at ICLR 2022
Analogous with the discussion in Sec. E.3 we have
E (u. >ap, [ollf(x) f( 11
E Papppaa upanen (5 ( Voll)((() f(=))1
n
1
" (Vol(()) fxx')1
R (0,a' (S(2,
n
enn
E Se (z,
- (0,00)
(I, d(a, ol df
ad(I, x²)
0 1
-
A(n - 1)
00
eAn n
E Barppautnn -alar"ppaaaai Paacaa Ad (m, 2d(
Od(2, x') +
+
A(n 1)
00
n
eAn
Be of daa.a"j (e Art (6, (u.,0)) Odl (2( x²)
+0
+
A (n-n) 1)
00
enn
Vo "oEx-ppaatr le Ar (r + 0
A2 10-1 1) en
n
42 +0
E.5 CONNECTION BETWEEN NAIVE LINEAR LOSS AND LDA
The intuition of multiple linear discriminant analysis (LDA) is to maximize the inter-class variance while minimizing the intra-class variance. In this section we will show linear metric loss have similar effect.
Definition 6. (Total variation) For a random vector I, the lotal variation is (V (r(E[(xx B[x))(xx B(c))?)) E[(x - [xx))) [[x]]] = E[xTTT]
E[a] B[
(11)
Definition 7. (Centroid) Define the centroid of samples in an arbitrary set Y by
Cy == Enyepast (I) PY) Byr-psuu (f())
(12)
Proposition 3. (Intra-class total variation) The within-class variation
TV Vnnaaa (X) Exppesn -xx cx.)l
cx.)
proportional to alignment loss
Caligned 2TV
(13)
Proof.
TVintra (X) = Exr - ((o cx.)
Paato (x.xxX f(x)]
cx.)
nE pe x. (x)))((2)) f(x' f(x')l lxex.))
(14)
Exrpante paraa (1)) S(x)) Σ Lexx
B(a,y) (() s())]
1
Catigned(f)
2
20

---

Under review as a conference paper at ICLR 2022
Proposition 4. (Inter-class total variation) The inter-class total variation
Tinttr (X) == Eurepenta [cxx -c) (cx, c))
where e is the centroid of all samples, is proportional triplet loss
2n
Llinear (f) = n1 Vinnter (X)
(15)
Proof.
TVinter (X) = Eempiice 1xxx cx,c c=c
(16)
Firstly,
Ex Bampaa (f()) Eurpaata [f(x)]
Pdota (d)ta(X)
Hence cf Lunbiased
Next,
Eaepaato 1x ex,e
Σ
Pdiata (x
Pdata (X, (x, CX.
Pa f(y)loex)
Bx My (1 (1)) f(y) Σ Leex.
Pu (x)x) s(v)]
Thus Ennpeer 1xx,,xx_ cx. a-1
Latigned and TV TVitter
Latigned(()
Lunbiassd (f)
Litnear (f)
2n
Proposition 5. (Total variation the dataset) The total variation of dataset X
TV (X): == Ex~paste [(x c)z (x -)c
where e is the centroid of all samples, is proportional the unbiased loss
Lunbiased 11--V -TVioota(XX)
(17)
Proof.
TVzotal (X) = PEac [fx) En- En-paete [f(x))
(18)
= Lunbiosed())
We can also check if Vooooall (X) T' V'witthin (X) + V.ettween (X) holds validate the proofs above.
APPENDIX F T-SNE EVALUATION OF ADML+A AND VANILLA DML.
In experiments, we visualize the embedding the first 10 classes CUB200-2011 generated by
vanilla DML model (with multi-similarity loss) and an ADML+A model (with multi-similarity loss). The Recall I of the DML model and the ADML+A model are 62.71 and 66.13 13, respectively. Fig. 4 (a) plots the T-SNE result for vanilla DML and Fig. 4 (b) plots the T-SNE result for ADML+A. Comparing two figures, we can see that ADML+A has better separation on the (red, green, orange)
samples.
21

---

Under review as a conference paper at ICLR 2022
dlsss
dlass
30
0.0
0.5
20
20
5.0
5.0
6.0
6.0
Dmz
-20
-30
-30
-20
-10
10
20
-40
-30
-20
-19
10
20
30
40
Diml
Dim1
(a) Vanilla DML.
(b) ADML+A
Figure 4: T-SNE visualization of embedding generated by vanilla DML and ADML+A model on CUB200-2011.
22


Reviews:
[{"review_id": "sIhgt-rBLiq", "reviewer": "Reviewer_xCnR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n--------------------------------------------\n1. Fairly well written, though a few sections, such as Sec 2 and 4 can be improved\n2. Theoretical proves are given however they are largely motivate by [1]\n\nWeakness\n-------------------------------------------\n1. Why the unit hypersphere is a nice feature space is not very well addressed in this paper. Is there any link between connected\nsets with smooth boundaries are nearly linearly separable in the hyperspherical geometry and such linear separability is used here to disentangle relation?\n2. \"... We derive the theoretical analysis for the triplet loss to prove that the triplet loss...\" Page 1, Para 2. IMHO this fact is already well studied and theoretically proved in [1]. A citation to [1] is missing on Page 1, Para 2. \n3. Ablation missing? For example, a study considering only positive, only contrastive, etc., is missing?\n4. Negative samples are hard negative or soft negatives? How are we getting negatives is not very clear.\n\nRef:\n-----------------------------------------------------\n1. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, ICML 2020", "summary_of_the_paper": "In this paper, the authors provided elegant similarities between contrastive learning and deep metric learning. They decomposed contrastive loss objectives into two quantities evaluating the geometry of the learned representation space:\n1. Alignment, and,\n2. Uniformity. Where Alignment is the closeness that defines how close two positive pairs are on the embedding space. While the uniformity ensures the scattered. The authors found the representation learned by optimizing contrastive objective indeed has these two properties compared to representation learned using supervised objective. They also showed that both alignment and uniformity are required for learning better examples for adversarial example tasks.", "summary_of_the_review": "The paper shows a closeness between contrastive learning and deep metric learning. They decomposed contrastive loss objectives into two quantities evaluating the geometry of the learned representation space:\n1. Alignment, and,\n2. Uniformity. However, the paper misses producing few ablations such as considering only positive, only contrastive, etc. How we mine negative samples is not very clear. Overall the paper seems like an extension of [1].\n\n\nRef:\n-----------------------------------------------------\n1. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, ICML 2020", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZLmD61XRHLN", "reviewer": "Reviewer_t4QZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the general idea of the paper and the hyperspherical perspective on deep metric learning. It has been a common wisdom that deep metric learning should be performed on a hyperspherical space, but an rigorous and principled understanding is yet to be found. I believe this is an interesting and extremely important direction. In such sense, this paper is tackling an significant problem.\n\nThe proposed algorithm to improve deep metric learning method is rather straightforward. Instead of generating the adversarial examples using the standard metric learning loss (e.g. triplet loss), the paper takes another approach by attacking either intra-class alignment loss or hyperspherical uniformity loss. I think the intuition behind makes senses, and the method can be viewed as attacking the deep metric learning objective from a dual space (which is equivalent to tuple-based loss space under some regularity).\n\nThe empirical results seem good to me. The proposed ADML by attacking intra-class alignment loss / hyperspherical uniformity loss shows non-trivial gain compared to the other existing methods.\n\nMy concerns and suggestions are mostly in the following aspects:\n\n(1) I understand that the decompostion to intra-class alignment and hyperspherical uniformity is from tuple-based losses, but will the generated adversarial examples also help contrastive loss / triplet loss? It will be interesting to see whether these adversarial examples are generally useful for deep metric learning methods.\n\n(2) Since attacking either intra-class alignment loss or hyperspherical uniformity loss yields a consistent performance gain, what if attack both objective at the same time? It will be more interesting to formulate the attack to both intra-class alignment loss and hyperspherical uniformity loss in a soft manner. That is to say, you have a parameter to control the balance between attacking two losses. Although you end up with an additioanl hyperparameter, it will make the method more flexible and I will be curious to see how the performance varies with this hyperparameter.\n\n(3) It will strengthen the experiments if there could be comparison among different adversarial attack methods. If every adversarial attack method works well for improving tuple-based losses, it will make the current conclusion more general.\n\n(4) The presentation could be significantly improved and I think the current paper structure may be suboptimal. I think some of the analyses on decomposing tuple-based loss to intra-class aligment and hyperspherical uniformity can be put to appendix. Only the main result is put in the main paper. Then the authors can be sufficient space to comprehensively evaluate how and why these generated adversarial examples help deep metric learning.\n\n(5) It will be helpful to visualize these adversarial examples on a 2-dim embedding space. Specifically, you can train the same model by setting the output feature dimension as 2. Then these features can be naturally visualized without any other visualization tools (such as T-SNE). Of course, using T-SNE may also be fine and interesting to see where these examples lie geometrically.\n\nI am sitting between weak accept and accept. If my concerns are properly addressed, I will be happy to increase my score.", "summary_of_the_paper": "This paper first analyzes the tuple-based metric losses on the hyperspherical space and studies two objective functions: intra-class alignment and hyperspherical uniformity. Then based on these two objective functions, the paper develops two ways of generating adversarial examples for improving deep metric learning. The empirical results show some advantages of the proposed method.", "summary_of_the_review": "This paper studies an interesting problem. Although the proposed method seems naive and incremental, its effectiveness and the significance of the topic really make up for that. Overall, I vote for weak accept, but may increase my rating based on the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U5JWA_H5tBy", "reviewer": "Reviewer_UWyt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe paper is well written and easy to follow. Though the proposed idea is interesting and the theoretical results are solid, the claimed contributions are marginal given the related works [1,2]. To be specific,\n\n1. I think the theoretical analysis that analyzes the intra-class alignment and hyperspherical uniformity for the tuple-based metric is a simple extension of [1] given the similarity between contrastive learning and deep metric learning (DML). In particular, the proof of main theoretical results in Section 3.2 in this paper inherits from the proof of Theorem 1 in [1] Appendix. Therefore, the theoretical contribution is marginal.\n\n2. The idea of using adversarial samples for DML has been well explored by [2], though with a different attacking objective (triplet loss). As proved in this paper that the triplet loss also optimizes the intra-class alignment and hyperspherical uniformity goals, so the proposed method has a marginal contribution.\n\nI have some other questions:\n1. Proposition 1 is unclearly motivated. I cannot connect it with other analysis.\n\n2. The claim that Page 3 “because the embedding space of DML is a unit hypersphere, where the maximum distance between two points is 2, it’s not possible to separate all negative embeddings with a large margin. Actually on $S^{k−1}$, the number of points with pairwise distance larger or equal than $\\sqrt{2}$ is at most k and the embedding dimension k is always smaller than the number of feature vectors, thus it’s impossible to make all distances between negative pairs exceed $\\sqrt{2}$” seems problematic. Assume k=2, then the number of points with a pairwise distance larger or equal than $\\sqrt{2}$ is supposed to be 4, imaging four points of a square in the circle. (If I am wrong, please correct me.)\n\n3. Why not consider a method ADML+A+U that includes two kinds of adversarial samples simultaneously?\n\n4. What is the lambda for experiment section 5.5?\n\n5. When increasing lambda, will the robustness of the model be better?\n\n6. In equation (7), $L_{uniformity}$ should connect to equation (2) (3).\n \n\n\nReference \n\n[1] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pp.9929–9939. PMLR, 2020\n\n[2] Yueqi Duan, Wenzhao Zheng, Xudong Lin, Jiwen Lu, and Jie Zhou. Deep adversarial metric learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.\n2780–2789, 2018.\n", "summary_of_the_paper": "The paper connects metric losses with the intra-class alignment and hyperspherical uniformity from theoretical analysis and proposes to boost metric learning augmented with adversarial samples generated by attacking alignment or uniformity objective. Experiments compare the proposed method with various baselines and the proposed method achieves the best performance.", "summary_of_the_review": "The paper has marginal contributions and limited novelty.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BULg4_6H6Gf", "reviewer": "Reviewer_MuXo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- A thorough proof of the link between alignment/uniformity on the sphere is provided for the case of the triplet loss;\n- The three variants of the proposed ADML provide a consist performance boost across various benchmarks;\n- The relevance of attacking the alignment/uniformity terms is justified empirically (Table 1);\n- The effect the hyperparameter $\\lambda$ introduces by their training scheme is studied across multiple datasets.\n\nWeaknesses:\n- As stated in this paper, separating the triplet loss into the intra-class alignment and the hypersphere uniformity terms is intractable. Therefore it is unclear whether the performance gain is a result of the different loss choice or the robustness to intra-class alignment and the hypersphere uniformity perturbations. In other words:\n    - Is the setting ADML+U+A, where both terms are adversarially attack, comparable to ADML+T? If that's the case then this would an argument in favor of attacking each one of the two terms separately. However, if that's not the case, then performance gain might be attributed to the InfoNCE loss being a better attack target;\n    - The combination of the intra-class alignment and the hypersphere uniformity terms used to generate adversarial examples is the InfoNCE loss. Would training the ADML model using the InfoNCE loss result in comparable results?\nI'm aware the gradients of the intra-class alignment and the hypersphere uniformity terms are only used to generate adversarial samples which are then used to train the model with the triplet loss. \n\n- Table 4 shows that the three ADML variants result in more robust models. However one needs to state that ADML+A has an unfair advantage compared to the rest of the models evaluated since it was explicitly trained for this scenario.", "summary_of_the_paper": "This work introduces novel adversarial deep metric learning algorithm based on the fact that contrastive losses promote intra-class alignment and uniformity on the unit sphere. The authors provide a theoretical proof that this is also true for the triplet loss under mild assumptions. \nThe adversarial examples used to regularize the training are generated by targeting the alignment and uniformity terms separately. This regularization strategy results in a significant robustness/performance boost on deep metric learning benchmarks.", "summary_of_the_review": "This work extends known properties of contrastive losses to the triplet loss. Namely, the loss can be rewritten as the sum of two terms: a intra-class alignment term and a hypersphere uniformity term. Based on this, a novel regularization term was devised for training deep metric learning models based on adversarial attacks on each one of the two terms. The three proposed variants of the ADML model acheive convincing performance gains across different image retrieval benchmarks and are more robust to subsequent adversarial attacks. However, it is unclear whether gain in performance is due to adversarially perturbing the intra-class alignment and the hypersphere uniformity terms separately or due to InfoNCE having more informative gradients compared to the triplet loss.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the paper has merits, I generally agree with negative reviewers. Among other issues, there were concerns about the theoretical contribution overlaps with prior work. While the authors argued the current work is not an extension, but rather designing ADML is. If this is the case, the paper should be rewritten to deemphasize the less novel contribution and focus more on what the authors believe to be the novel contribution. I don't believe in the practice of putting different messages (some novel and some not) into a paper with the hope that this makes the overall result "more novel". I'd suggest the author rewrite the paper and more clear about the message.
Decision: Reject