It Takes Two to Tango: Mixup for Deep Metric Learning

1 INTRODUCTION Classification is one of the most studied tasks in machine learning and deep learning . It is a common source of pre-trained models for transfer learning to other tasks ( Donahue et al. , 2014 ; Kolesnikov et al. , 2020 ) . It has been studied under different supervision settings ( Caron et al. , 2018 ; Sohn et al. , 2020 ) , knowledge transfer ( Hinton et al. , 2015 ) and data augmentation ( Cubuk et al. , 2018 ) , including the recent research on mixup ( Zhang et al. , 2018 ; Verma et al. , 2019 ) , where embeddings and labels are interpolated . Deep metric learning is about learning from pairwise interactions such that inference relies on instance embeddings , e.g . for nearest neighbor classification ( Oh Song et al. , ∗equal contribution 2016 ) , instance-level retrieval ( Gordo et al. , 2016 ) , few-shot learning ( Vinyals et al. , 2016 ) , face recognition ( Schroff et al. , 2015 ) and semantic textual similarity ( Reimers & Gurevych , 2019 ) . Following ( Xing et al. , 2003 ) , it is most often fully supervised by one class label per example , like classification . The two most studied problems are loss functions ( Musgrave et al. , 2020 ) and hard example mining ( Wu et al. , 2017 ; Robinson et al. , 2021 ) . Tuple-based losses with example weighting ( Wang et al. , 2019 ) can play the role of both . Unlike classification , classes ( and distributions ) at training and inference are different in metric learning . Thus , one might expect interpolation-based data augmentation like mixup to be even more important in metric learning than in classification . Yet , recent attempts are mostly limited to special cases of embedding interpolation and have trouble with label interpolation ( Ko & Gu , 2020 ) . This raises the question : what is a proper way to define and interpolate labels for metric learning ? In this work , we observe that metric learning is not different from classification , where examples are replaced by pairs of examples and class labels by “ positive ” or “ negative ” , according to whether class labels of individual examples are the same or not . The positive or negative label of an example , or a pair , is determined in relation to a given example which is called an anchor . Then , as shown in Figure 1 , a straightforward way is to use a binary ( two class ) label per pair and interpolate it linearly as in standard mixup . We call our method Metric Mix , or Metrix for short . To show that mixing examples improves representation learning , we quantitatively measure the properties of the test distributions using alignment and uniformity ( Wang & Isola , 2020 ) . Alignment measures the clustering quality and uniformity measures its distribution over the embedding space ; a well clustered and uniformly spread distribution indicates higher representation quality . We also introduce a new metric , utilization , to measure the extent to which a test example , seen as a query , lies near any of the training examples , clean or mixed . By quantitatively measuring these three metrics , we show that interpolation-based data augmentation like mixup is very important in metric learning , given the difference between distributions at training and inference . In summary , we make the following contributions : 1 . We define a generic way of representing and interpolating labels , which allows straightforward extension of any kind of mixup to deep metric learning for a large class of loss functions . We develop our method on a generic formulation that encapsulates these functions ( section 3 ) . 2 . We define the “ positivity ” of a mixed example and we study precisely how it increases as a function of the interpolation factor , both in theory and empirically ( subsection 3.6 ) . 3 . We systematically evaluate mixup for deep metric learning under different settings , including mixup at different representation levels ( input/manifold ) , mixup of different pairs of examples ( anchors/positives/negatives ) , loss functions and hard example mining ( subsection 4.2 ) . 4 . We introduce a new evaluation metric , utilization , validating that a representation more appropriate for test classes is implicitly learned during exploration of the embedding space in the presence of mixup ( subsection 4.3 ) . 5 . We improve the state of the art on four common metric learning benchmarks ( subsection 4.2 ) . 2 RELATED WORK . Metric learning Metric learning aims to learn a metric such that positive pairs of examples are nearby and negative ones are far away . In deep metric learning , we learn an explicit non-linear mapping from raw input to a low-dimensional embedding space ( Oh Song et al. , 2016 ) , where the Euclidean distance has the desired properties . Although learning can be unsupervised ( Hadsell et al. , 2006 ) , deep metric learning has mostly followed the supervised approach , where positive and negative pairs are defined as having the same or different class label , respectively ( Xing et al. , 2003 ) . Loss functions can be distinguished into pair-based and proxy-based ( Musgrave et al. , 2020 ) . Pairbased losses use pairs of examples ( Wu et al. , 2017 ; Hadsell et al. , 2006 ) , which can be defined over triplets ( Wang et al. , 2014 ; Schroff et al. , 2015 ; Weinberger & Saul , 2009 ; Hermans et al. , 2017 ) , quadruples ( Chen et al. , 2017 ) or tuples ( Sohn , 2016 ; Oh Song et al. , 2016 ; Wang et al. , 2019 ) . Proxy-based losses use one or more proxies per class , which are learnable parameters in the embedding space ( Movshovitz-Attias et al. , 2017 ; Qian et al. , 2019 ; Kim et al. , 2020c ; Teh et al. , 2020 ; Zhu et al. , 2020b ) . Pair-based losses capture data-to-data relations , but they are sensitive to noisy labels and outliers . They often involve terms where given constraints are satisfied , which produce zero gradients and do not contribute to training . This necessitates mining of hard examples that violate the constraints , like semi-hard ( Schroff et al. , 2015 ) and distance weighted ( Wu et al. , 2017 ) . By contrast , proxy-based losses use data-to-proxy relations , assuming proxies can capture the global structure of the embedding space . They involve less computations that are more likely to produce nonzero gradient , hence have less or no dependence on mining and converge faster . Mixup Input mixup ( Zhang et al. , 2018 ) linearly interpolates between two or more examples in the input space for data augmentation . Numerous variants take advantage of the structure of the input space to interpolate non-linearly , e.g . for images ( Yun et al. , 2019 ; Kim et al. , 2020a ; 2021 ; Hendrycks et al. , 2020 ; DeVries & Taylor , 2017 ; Qin et al. , 2020 ; Uddin et al. , 2021 ) . Manifold mixup ( Verma et al. , 2019 ) interpolates intermediate representations instead , where the structure is learned . This can be applied to or assisted by decoding back to the input space ( Berthelot et al. , 2018 ; Liu et al. , 2018 ; Beckham et al. , 2019 ; Zhu et al. , 2020a ; Venkataramanan et al. , 2021 ) . In both cases , corresponding labels are linearly interpolated too . Most studies are limited to cross-entropy loss for classification . Pairwise loss functions have been under-studied , as discussed below . Interpolation for pairwise loss functions As discussed in subsection 3.3 , interpolating target labels is not straightforward in pairwise loss functions . In deep metric learning , embedding expansion ( Ko & Gu , 2020 ) , HDML ( Zheng et al. , 2019 ) and symmetrical synthesis ( Gu & Ko , 2020 ) interpolate pairs of embeddings in a deterministic way within the same class , applying to pair-based losses , while proxy synthesis ( Gu et al. , 2021 ) interpolates between classes , applying to proxy-based losses . None performs label interpolation , which means that ( Gu et al. , 2021 ) risks synthesizing false negatives when the interpolation factor λ is close to 0 or 1 . In contrastive representation learning , MoCHi ( Kalantidis et al. , 2020 ) interpolates anchor with negative embeddings but not labels and chooses λ ∈ [ 0 , 0.5 ] to avoid false negatives . This resembles thresholding of λ at 0.5 in OptTransMix ( Zhu et al. , 2020a ) . Finally , i-mix ( Lee et al. , 2021 ) and MixCo ( Kim et al. , 2020b ) interpolate pairs of anchor embeddings as well as their ( virtual ) class labels linearly . There is only one positive , while all negatives are clean , so it can not take advantage of interpolation for relative weighting of positives/negatives per anchor ( Wang et al. , 2019 ) . By contrast , Metrix is developed for deep metric learning and applies to a large class of both pairbased and proxy-based losses . It can interpolate inputs , intermediate features or embeddings of anchors , ( multiple ) positives or negatives and the corresponding two-class ( positive/negative ) labels per anchor , such that relative weighting of positives/negatives depends on interpolation . 3 MIXUP FOR METRIC LEARNING . 3.1 PRELIMINARIES . Problem formulation We are given a training set X ⊂ X , where X is the input space . For each anchor a ∈ X , we are also given a set P ( a ) ⊂ X of positives and a set N ( a ) ⊂ X of negatives . The positives are typically examples that belong to the same class as the anchor , while negatives belong to a different class . The objective is to train the parameters θ of a model f : X → Rd that maps input examples to a d-dimensional embedding , such that positives are close to the anchor and negatives are far away in the embedding space . Given two examples x , x′ ∈ X , we denote by s ( x , x′ ) the similarity between x , x′ in the embedding space , typically a decreasing function of Euclidean distance . It is common to ` 2-normalize embeddings and define s ( x , x′ ) : = 〈f ( x ) , f ( x′ ) 〉 , which is the cosine similarity . To simplify notation , we drop the dependence of f , s on θ. Pair-based losses ( Hadsell et al. , 2006 ; Wang et al. , 2014 ; Oh Song et al. , 2016 ; Wang et al. , 2019 ) use both anchors and positives/negatives in X , as discussed above . Proxy-based losses define one or more learnable proxies ∈ Rd per class , and only use proxies as anchors ( Kim et al. , 2020c ) or as positives/negatives ( Movshovitz-Attias et al. , 2017 ; Qian et al. , 2019 ; Teh et al. , 2020 ) . To accommodate for uniform exposition , we extend the definition of similarity as s ( v , x ) : = 〈v , f ( x ) 〉 for v ∈ Rd , x ∈ X ( proxy anchors ) and s ( x , v ) : = 〈f ( x ) , v〉 for x ∈ X , v ∈ Rd ( proxy positives/negatives ) . Finally , to accommodate for mixed embeddings in subsection 3.5 , we define s ( v , v′ ) : = 〈v , v′〉 for v , v′ ∈ Rd . Thus , we define s : ( X ∪ Rd ) 2 → R over pairs of either inputs in X or embeddings in Rd . We discuss a few representative loss functions below , before deriving a generic form . Contrastive The contrastive loss ( Hadsell et al. , 2006 ) encourages positive examples to be pulled towards the anchor and negative examples to be pushed away by a margin m ∈ R. This loss is additive over positives and negatives , defined as : ` cont ( a ; θ ) : = ∑ p∈P ( a ) −s ( a , p ) + ∑ n∈N ( a ) [ s ( a , n ) −m ] + . ( 1 ) Multi-Similarity The multi-similarity loss ( Wang et al. , 2019 ) introduces relative weighting to encourage positives ( negatives ) that are farthest from ( closest to ) the anchor to be pulled towards ( pushed away from ) the anchor by a higher weight . This loss is not additive over positives and negatives : ` MS ( a ; θ ) : = 1 β log 1 + ∑ p∈P ( a ) e−β ( s ( a , p ) −m ) + 1 γ log 1 + ∑ n∈N ( a ) eγ ( s ( a , n ) −m ) . ( 2 ) Here , β , γ ∈ R are scaling factors for positives , negatives respectively . Proxy Anchor The proxy anchor loss ( Kim et al. , 2020c ) defines a learnable proxy in Rd for each class and only uses proxies as anchors . For a given anchor ( proxy ) a ∈ Rd , the loss has the same form as ( 2 ) , although similarity s is evaluated on Rd ×X .

Summary: The paper presents a technique for using mixup augmentation in deep metric learning training. Specifically, the dml loss function is represented in a general form so that mixup loss can be easily computed for different pairs. This loss is combined with regular dml loss for training the network. The method is evaluated on CUB200, CARS196, SOP and IN-SHOP datasets where it outperforms it's baselines and also achieves SOTA results.