Understanding Metric Learning on Unit Hypersphere and Generating Better Examples for Adversarial Training

1 INTRODUCTION . Deep metric learning ( DML ) has been applied to various computer vision tasks ranging from face recognition ( Schroff et al. , 2015 ; Liu et al. , 2017 ) to zero-shot learning ( Romera-Paredes & Torr , 2015 ; Bucher et al. , 2016 ) and image retrieval ( Song et al. , 2016 ; Wu et al. , 2017 ) . It has been proved to be one of the most effective methods for learning the distance-preserving features of images . The intuition of DML is to pull the embedding of positive images pairs together and push the negative pairs apart , where the embedding function could be a deep neural network . Most of the metric losses in DML are tuple-based ( Schroff et al. , 2015 ; Song et al. , 2016 ; Wu et al. , 2017 ; Wang et al. , 2019 ) or classification-based ( Movshovitz-Attias et al. , 2017 ; Kim et al. , 2020 ; Boudiaf et al. , 2020 ) , these different losses have been shown to achieve similar performance in the recent reviews of DML ( Roth et al. , 2020 ; Musgrave et al. , 2020 ) . One common ground of existing DML models is that the embedding space is a unit hypersphere . It is widely known that achieving uniformity on hypersphere can increase the generalization of models and preserve as much information as possible ( Bachman et al. , 2019 ; Liu et al. , 2018 ; 2021 ; Hjelm et al. , 2018 ) , and the objective function that lead to uniformity is called uniformity regularization . Meanwhile , the downstream tasks in DML favor the models with small intra-class alignment ( Wu et al. , 2017 ; Wang et al. , 2019 ) . In this work , we investigate these two properties , intra-class alignment and hyperspherical uniformity ( Wang & Isola , 2020 ) for tuple-based metric losses . We derive the theoretical analysis for the triplet loss to prove that the triplet loss on the positive sample pairs minimizes the intra-class alignment by mapping all samples from one class to the same vector , while the triplet loss on the negative sample pairs achieves hyperspherical uniformity . We further conduct empirical studies to show that the same statement is also valid for other tuple-based metric losses . We utilize our new understanding on DML to design novel robust DML methods to enhance the performance via improved adversarial training . Adversarial training aims at improving the robustness of models towards to certain types of attacks by training with perturbed samples . However , as shown in the recent work on contrastive representation learning ( Jiang et al. , 2020 ) , adversarial training can also enhance the natural performance on the downstream classification task . Due to the similarity between contrastive learning and deep metric learning , we believe it ’ s also possible to improve the nature performance of metric learning models with adversarial samples . Following our new insights on positive and negative metric losses , we generate perturbations by attacking the alignment or uniformity objective , and create adversarial DML models augmented with both normal samples and perturbed samples . Our experimental results show that the new adversarial DML models can significantly boost the natural performance . The major contributions of our paper can be summarized as follows : • We analyze the intra-class alignment and hyperspherical uniformity for tuple-based metric losses , and establish the connections between these two properties and the positive/negative metric losses . • Based on our new analysis and understanding , we propose two new adversarial DML models , ADML+A and ADML+U , via attacking the alignment or uniformity objective . ADML+A and ADML+U improve the natural performance on benchmarks significantly . 2 RELATED WORKS . Deep metric learning . There are mainly two kind of metric losses in DML , tuple-based and classification based losses . Tuple-based losses include contrastive loss ( Hadsell et al. , 2006 ) , triplet loss ( Schroff et al. , 2015 ) , margin loss ( Wu et al. , 2017 ) , and multi-similarity loss ( Wang et al. , 2019 ) , where the objective function is based on the distance between positive pairs and negative pairs . In classification-based losses , the learning objective is not depend on the positive or negative pairs but a fixed ( Boudiaf et al. , 2020 ) or learnable proxy ( Kim et al. , 2020 ) . In the recent reviews of metric learning methods ( Roth et al. , 2020 ; Musgrave et al. , 2020 ) , it ’ s concluded the improvement on the DML performance is mainly due to different training strategies and unfair comparison . The original contrastive loss and triplet loss still achieved comparable result with other metric losses under the same network and training strategies . In experiments we apply the training framework of ( Roth et al. , 2020 ) to ensure fair comparison . Learning with hyperspherical uniformity . Hyperspherical learning regards learning tasks where the embedding space is a unit hypersphere . The uniformity of the hypersphere represents the diversity of vectors on the sphere . It encourages vectors to be spaced with angles as large as possible so that these vectors can be evenly distributed on the hypersphere ( Liu et al. , 2018 ) . Achieving hyperspherical uniformity can help with preventing overfitting and improving generalization of the neural works ( Liu et al. , 2021 ) . The objective functions which can lead to the uniformity on hypersphere are called uniformity regularization . Hyperspherical embedding is widely applied in representation learning tasks such as contrastive representation learning ( Oord et al. , 2018 ; Hjelm et al. , 2018 ) and DML ( Wu et al. , 2017 ; Liu et al. , 2017 ) . Wang & Isola ( 2020 ) showed that the objective function in contrastive representation learning optimizes for intra-class alignment and uniformity together . Adversarial examples improves natural performance . In classification tasks , it is well known that the clean accuracy of adversarially trained model is typically worse than the normal model . However , Xie et al . ( 2020 ) showed that adversarial samples can be used to improve the clean accuracy of image classification models . According to ( Jiang et al. , 2020 ) , training with adversarial samples can help improve the natural performance of contrastive learning models on the downstream classification tasks . The authors presented adversarial attacks based on the objective of contrastive learning and achieved improvement on both natural and robustness performance . It ’ s believed that adversarial examples contain extra features , thus the generalization of models augmented with adversarial example is increased ( Ilyas et al. , 2019 ; Salman et al. , 2020 ; Xie et al. , 2020 ) , which contributes to better natural performance . Duan et al . ( 2018 ) utilized the triplet loss as the attacking objective to generate adversarial examples to improve DML models . To our best knowledge , this is the only one work using adversarial example to boost the natural performance of DML . In our work , we show that alignment and uniformity loss can generate stronger adversarial examples comparing to triplet loss , and thus lead to better generalization of DML models . 3 ALIGNMENT AND HYPERSPHERICAL UNIFORMITY IN TUPLE-BASED METRIC LOSSES . In this section , we will study tuple-based metric losses on the unit hypersphere embedding space . We assume having n classes X1 , · · · , Xn in training set and denote the encoder by f : Rd → Sk−1 where Sk−1 is the surface of a k-dimensional unit ball . Let pdata ( · ) be the data distribution over Rd , ppos ( · , · ) be the distribution of positive pairs over Rd ×Rd , and ptri ( · , · , · ) be the distribution of triplet pairs over Rd × Rd × Rd , where the first two entries have the same label and the third entry is a sample from different classes . Please note that all detailed proofs are included in supplementary material Appendix E. We also conduct experiments to validate our theoretical analysis , the details is in Appendix C. The major intuition of DML is to pull the representations of similar samples together and push dissimilar samples apart . Thus , we reformulate the metric losses as the combination of two parts : • Positive metric loss : minimizes the distance between embedded positive sample pairs . • Negative metric loss : maximizes the distance between embedded negative sample pairs . Although in DML models the positive metric losses have different representations , they share one common optimal solution pattern , where samples from the same class are mapped to the same feature vector . Thus , we define the alignment loss with minimizing the intra-class distance . Definition 1 . ( Intra-class alignment ) The expectation of intra-class distance is given by : Lalignment ( f ; X , ppos ) : = E ( x , y ) ∼ppos [ ||f ( x ) − f ( y ) ||22 ] ( 1 ) the minimum of this loss is achieved when the samples with the same label are encoded to the same embedding . Proposition 1 . If the support set of the data distribution is connected and the support set of each class distribution is closed , the minimum of Lalignment is reached when all samples are projected to the same vector . In Sec . C.1 , we conduct the empirical studies to verify our analysis . Results in Table 8 show that samples are roughly projected to the same vector if only the positive metric losses are used . The negative metric losses aim at positioning the embedding of dissimilar samples as far as possible . However , because the embedding space of DML is a unit hypersphere , where the maximum distance between two points is 2 , it ’ s not possible to separate all negative embeddings with a large margin . Actually on Sk−1 , the number of points with pairwise distance larger or equal than √ 2 is at most 2k and the embedding dimension k is always much smaller than the number of feature vectors , thus it ’ s impossible to make all distances between negative pairs exceed √ 2 . Therefore investigating the properties of negative metric losses on the unit hypersphere is an interesting and important topic . We believe the negative metric losses are closely related to the uniformity on the hypersphere and our experimental results support this argument . Definition 2 . ( Hyperspherical uniformity ) The embedded samples should be evenly distributed on the spherical surface . In practice , the hyperspherical uniformity can be achieved by optimizing the uniformity regularization . There exist many different representations of the regularization , and we utilize the hyperspherical energy ( HE ) ( Liu et al. , 2018 ) : E ( s , X ) = { Ex∼pdata , y∼pdata [ ||f ( x ) − f ( y ) || −s 2 1x ̸=y ] , s > 0 Ex∼pdata , y∼pdata [ log ( ||f ( x ) − f ( y ) || −1 2 1x ̸=y ) ] , s = 0 ( 2 ) and Gaussian hyperspherical energy ( G-HE ) ( Wang & Isola , 2020 ) : EG ( s , X ) = logEx∼pdata , y∼pdata [ e−s||f ( x ) −f ( y ) || 2 2 ] , s > 0 ( 3 ) in the experiments for comparison . The values of HE and G-HE can also be used as measurements on the uniformity of the embedded samples . We expect the value to be small in order to achieve good hyperspherical uniformity . We also want to mention that simply maximizing the distance between samples will not lead to hyperspherical uniformity , and the detailed discussion is in Sec . B.2 . Because finding the optimal solution of the HE or G-HE problem is NP-hard ( Liu et al. , 2018 ) , we are not able to calculate the exact position of vectors which are evenly distributed on the sphere . We provide a primary insight about how should finite vectors be uniformly distributed on unit hypersphere , and our conclusion is consist with the empirical results . Since there exist many different tuple-based metric losses , analyzing all of them theoretically is impossible in this work . In Sec . 3.1 , we will provide the theoretical analysis of the triplet loss . The analysis of linear loss can be found in Sec . B.2 . In Appendix C we will show the empirical results on four popularly used tuple-based metric losses to verify our statement .

Summary: In this paper, the authors provided elegant similarities between contrastive learning and deep metric learning. They decomposed contrastive loss objectives into two quantities evaluating the geometry of the learned representation space: 1. Alignment, and, 2. Uniformity. Where Alignment is the closeness that defines how close two positive pairs are on the embedding space. While the uniformity ensures the scattered. The authors found the representation learned by optimizing contrastive objective indeed has these two properties compared to representation learned using supervised objective. They also showed that both alignment and uniformity are required for learning better examples for adversarial example tasks.