Reducing Class Collapse in Metric Learning with Easy Positive Sampling

1 INTRODUCTION . Metric learning aims to learn an embedding function to lower dimensional space , in which semantic similarity translates to neighborhood relations in the embedding space ( Lowe , 1995 ) . Deep metric learning approaches achieve promising results in a large variety of tasks such as face identification ( Chopra et al. , 2005 ; Taigman et al. , 2014 ; Sun et al. , 2014 ) , zero-shot learning ( Frome et al. , 2013 ) , image retrieval ( Hoffer & Ailon , 2015 ; Gordo et al. , 2016 ) and fine-grained recognition ( Wang et al. , 2014 ) . In this work we investigate the family of losses which optimize for an embedding representation that enforces that all modes of intra-class appearance variation project to a single point in embedding space . Learning such an embedding is very challenging when classes have a diverse appearance . This happens especially in real-world scenarios where the class consists of multiple modes with diverse visual appearance . Pushing all these modes to a single point in the embedding space requires the network to memorize the relations between the different class modes , which could reduce the generalization capabilities of the network and result in sub-par performance . Recently researchers observed that this phenomena , where all modes of class appearance “ collapse ” to the same center , occurs in case of the classification SoftMax loss ( Qian et al. ) . They proposed a multi-center approach , where multiple centers for each class are used with the SoftMax loss to capture the hidden distribution of the data to solve this issue . Instead of using SoftMax , it was shown that triplet loss may offer some relief from class collapsing ( Wang et al. , 2014 ) and this is certainly true in noise-free environments . However , in this paper , we show that in real-world conditions with modest noise assumptions , triplet and other metric learning loss yet suffer from class collapse . Rather than refine the loss , we argue the key lies in an improved strategy for sampling and selecting the examples . Early work ( Malisiewicz & Efros , 2008 ) proposed per-exemplar distance representation as a means to overcome class collapsing ; inspired by this we introduce a simple sampling method to select positive pairs of training examples . Our method can be combined naturally with other popular sampling methods . In each training iteration , given an anchor and a batch of samples in the same category , our method selects the closest sample to the anchor in the current embedding space as the positive sample . The metric learning loss is then computed based on the anchor and its positive paired sample . We demonstrate the class-collapsing phenomena on a real-world dataset , and show that our method is able to create more diverse embedding which result in a better generalization performance . We evaluate our method on three standard zero-shot benchmarks : CARS196 ( Krause et al. , 2013 ) , CUB200-2011 ( Wah et al. , 2011 ) and Omniglot ( Lake et al. , 2015 ) . Our method achieves a consistent performance enhancement with respect to various baseline combinations of sampling methods and embedding losses . 2 RELATED WORK . Sampling methods . Designing a good sampling strategy is a key element in deep metric learning . Researchers have been proposed sampling methods when sampling both the negative examples as well as the positive pairs . For negative samples , studies have focused on sampling hard negatives to make training more efficient ( Simo-Serra et al. , 2015 ; Schroff et al. , 2015 ; Wang & Gupta , 2015 ; Oh Song et al. , 2016 ; Parkhi et al. , 2015 ) . Recently , it has been shown that increasing the negative examples in training can significantly help unsupervised representation learning with contrastive losses ( He et al. , 2020 ; Wu et al. , 2018 ; Chen et al. , 2020 ) . Besides negative examples , methods for sampling hard positive examples have been developed in classification and detection tasks ( Loshchilov & Hutter , 2015 ; Shrivastava et al. , 2016 ; Arandjelovic et al. , 2016 ; Cubuk et al. , 2019 ; Singh & Lee , 2017 ; Wang et al. , 2017 ) . The central idea is to perform better augmentation to improve the generalization in testing ( Cubuk et al. , 2019 ) . Apart from learning with SoftMax classification , Arandjelovic et al . ( 2016 ) propose to perform metric learning by assigning the near instance from the same class as the positive instance . As the positive training set is noisy in their setting , this method leads to features invariant to different perspectives . Different from this approach , we use this method in a clean setting , where the purpose is to get the opposite result of maintaining the inner-class modalities in the embedding space . Xuan et al . ( 2020 ) also propose to use this positive sampling method with respect to the N-pair loss ( Sohn , 2016 ) in order to relax the constraints of the loss on the intra-class relations . From a theoretic perspective , we prove that in a clean setting this relaxation is redundant for other popular metric losses like the triplet loss ( Chechik et al. , 2010 ) and margin loss ( Wu et al. , 2017 ) . We formulate the noisy-environment setting and prove that in this case the triplet and margin losses also suffer from class-collapsing and using our purpose positive sampling method optimizes for solutions without class-collapsing . We also provide an empirical study that supports the theoretic analysis . Noisy label problem . Learning with noisy labels is a practical problem when applied to the real world ( Scott et al. , 2013 ; Natarajan et al. , 2013 ; Shen & Sanghavi , 2019 ; Reed et al. , 2014 ; Jiang et al. , 2017 ; Khetan et al. , 2017 ; Malach & Shalev-Shwartz , 2017 ) , especially when training with large-scale data ( Sun et al. , 2017 ) . One line of work applies a data-driven curriculum learning approach where the data that are most likely labeled correctly are used for learning in the beginning , and then harder data is taken into learning during a later phase ( Jiang et al. , 2017 ) . Researchers have also tried on to apply the loss only on easiest top k-elements in the batch , determine by lowest current loss ( Shen & Sanghavi , 2019 ) . Inspired by these works , our method focuses on selecting only the top easiest positive relations in the batch . Beyond memorization . Deep networks are shown to be extremely easy to memorize and over-fit to the training data ( Zhang et al. , 2016 ; Recht et al. , 2018 ; 2019 ) . For example , it is shown the network can be trained with randomly assigned labels on the ImageNet data , and obtain 100 % training accuracy if augmentations are not adopted . Moreover , even the CIFAR-10 classifier performs well in the validation set , it is shown that it does not really generalize to new collected data which is visually similar to the training and validation set ( Recht et al. , 2018 ) . In this paper , we show that when allowing the network the freedom not to have to learn inner-class relation between different class modes , we can achieve much better generalization , and the representation can be applied in a zero-shot setting . 3 PRELIMINARIES . Let X = { x1 , .. , xn } be a set of samples with labels yi ∈ { 1 , .. , m } . The objective of metric learning is to learn an embedding f ( · , θ ) −→ Rk , in which the neighbourhood of each sample in the embedding space contains samples only from the same class . One of the common approaches for metric learning is using embedding losses in which at each iteration , samples from the same class and samples from different classes are chosen according to same sampling heuristic . The objective of the loss is to push away projections of samples from different classes , and pull closer projections of samples from a same class . In this section , we introduce a few popular embedding losses . Notation : Let xi , xj ∈ X , define : Dfxi , xj = ‖ f ( xi ) −f ( xj ) ‖ 2 . In cases where there is no ambiguity we omit f and simply write Dxi , xj . We also define the function δxi , xj = { 1 if yi = yj 0 otherwise . Lastly , for every a ∈ R , denote ( a ) + : = max ( a , 0 ) . The Contrastive loss ( Hadsell et al . ) takes sample embeddings and pushes the samples from the different classes apart and pulls samples from the same class together . Lfcon ( xi , xj ) = δxi , xj ·Dfxi , xj + ( 1− δxi , xj ) · ( α−D f xi , xj ) + Here α is the margin parameter which defines the desired minimal distance between samples from different classes . While the Contrastive loss imposes a constraint on a pair of samples , the Triplet loss ( Chechik et al. , 2010 ) functions on a triplet of samples . Given a triplet xa , xp , xn ∈ X , the triplet loss is defined by Lftrip ( xa , xp , xn ) = δxa , xp · ( 1− δxa , xn ) · ( D f xa , xp −D f xp , xn + α ) + The Margin loss ( Wu et al. , 2017 ) aims to exploit the flexibility of Triplet loss while maintaining the computational efficiency of the Contrastive loss . This is done by adding a variable which determines the boundary between positive and negative pairs ; given an anchor xa ∈ X the loss is defined by Lf , βmargin ( xa , x ) = δxa , x · ( D f xa , x − βxa + α ) + + ( 1− δxa , x ) · ( βxa −D f xa , x + α ) + 4 CLASS-COLLAPSING . The contrastive loss objective is to pull all the samples with the same class to a single point in the embedding space . We call this the Class-collapsing property . Formally , an embedding f : X −→ Rm has the class-collapsing property , if there exists a label y and a point p ∈ Rm such that { f ( xi ) | yi = y } = { p } . 4.1 EMBEDDING LOSSES OPTIMAL SOLUTION . It is easy to see that an embedding function f that minimizes : Ocon ( f ) = 1 n2 ∑ xi , xj∈X Lfcon ( xi , xj ) has the class-collapsing property with respect to all classes . However , this is not necessarily true for the Triplet loss and the Margin loss . For simplification for the rest of this subsection we will assume there are only two classes . Let A ⊂ X be a subset of elements such that all the elements in A belongs to one class and all the element in Ac belong to the other class . Recall some basic set definitions . Definition 1 . For all sets Y , Z ⊂ Rm define : 1 . The diameter of Y is defined by : diam ( Y ) = sup { ‖y − z‖ |y , z ∈ Y } 2 . The distance between Y and Z is : ‖Y − Z‖ = inf { ‖y − z‖ |y ∈ Y , z ∈ Z } It is easy to see that if f : X −→ Rm is an embedding , such that diam ( f ( A ) ) < 2·α+‖f ( A ) −f ( B ) ‖ , then : Otrip ( f ) = 1 n3 ∑ xi , xj , xk∈X Lftrip ( xi , xj , xk ) = 0 . Moreover , fixing βxi = α for every xi ∈ X , then : Omargin ( f , β ) = 1 n2 ∑ xi , xj∈X Lf , βmargin ( xi , xj ) = 0 . It can be seen that indeed , the family of embedding which induce the global-minimum with respect to the Triplet loss and the Margin loss , is rich and diverse . However , as we will prove in the next subsection , this does not remain true in a noisy environment scenario .

Summary: This paper proposes/adopts a simple positive sampling scheme in metric learning: only sampling the easiest positive for each anchor. Authors give a theoretical analysis of how the proposed sampling scheme can reduce class collapse. Experiments on fine-grain retrieval datasets show the effectiveness of the sampling scheme. Using the sampled easiest positive, nearly all current metric learning methods got improved performance.