INSTANCE CROSS ENTROPY FOR DEEP METRIC LEARNING

1 INTRODUCTION . Deep metric learning ( DML ) aims to learn a non-linear embedding function ( a.k.a . distance metric ) such that the semantic similarities over samples are well captured in the feature space ( Tadmor et al. , 2016 ; Sohn , 2016 ) . Due to its fundamental function of learning discriminative representations , DML has diverse applications , such as image retrieval ( Song et al. , 2016 ) , clustering ( Song et al. , 2017 ) , verification ( Schroff et al. , 2015 ) , few-shot learning ( Vinyals et al. , 2016 ) and zero-shot learning ( Bucher et al. , 2016 ) . A key to DML is to design an effective and efficient loss function for supervising the learning process , thus significant efforts have been made ( Chopra et al. , 2005 ; Schroff et al. , 2015 ; Sohn , 2016 ; Song et al. , 2016 ; 2017 ; Law et al. , 2017 ; Wu et al. , 2017 ) . Some loss functions learn the embedding function from pairwise or triplet-wise relationship constraints ( Chopra et al. , 2005 ; Schroff et al. , 2015 ; Tadmor et al. , 2016 ) . However , they are known to not only suffer from an increasing number of non-informative samples during training , but also incur considering only several instances per loss computation . Therefore , informative sample mining strategies are proposed ( Schroff et al. , 2015 ; Wu et al. , 2017 ; Wang et al. , 2019b ) . Recently , several methods consider semantic relations among multiple examples to exploit their similarity structure ( Sohn , 2016 ; Song et al. , 2016 ; 2017 ; Law et al. , 2017 ) . Consequently , these structured losses achieve better performance than pairwise and triple-wise approaches . In this paper , we tackle the DML problem from a novel perspective . Specifically , we propose a novel loss function inspired by CCE . CCE is well-known in classification problems owing to the fact that it has an intuitive probabilistic interpretation and achieves great performance , e.g. , ImageNet classification ( Russakovsky et al. , 2015 ) . However , since CCE learns a decision function which predicts the class label of an input , it learns class-level centres for reference ( Zhang et al. , 2018 ; Wang et al. , 2017a ) . Therefore , CCE is not scalable to infinite classes and can not generalise well when it is directly applied to DML ( Law et al. , 2017 ) . With scalability and structured information in mind , we introduce instance cross entropy ( ICE ) for DML . It learns an embedding function by minimising the cross entropy between a predicted instance-level matching distribution and its corresponding ground-truth . In comparison with CCE , given a query , CCE aims to maximise its matching probability with the class-level context vector ( weight vector ) of its ground-truth class , whereas ICE targets at maximising its matching probability with it similar instances . As ICE does not learn class-level context vectors , it is scalable to infinite training classes , which is an intrinsic demand of DML . Similar to ( Sohn , 2016 ; Song et al. , 2016 ; 2017 ; Law et al. , 2017 ; Goldberger et al. , 2005 ; Wu et al. , 2018 ) , ICE is a structured loss as it also considers all other instances in the mini-batch of a given query . We illustrate ICE with comparison to other structured losses in Figure 1 . A common challenge of instance-based losses is that many training examples become trivial as model improves . Therefore , we integrate seamless sample reweighting into ICE , which functions similarly with various sample mining schemes ( Sohn , 2016 ; Schroff et al. , 2015 ; Shi et al. , 2016 ; Yuan et al. , 2017 ; Wu et al. , 2017 ) . Existing mining methods require either separate time-consuming process , e.g. , class mining ( Sohn , 2016 ) , or distance thresholds for data pruning ( Schroff et al. , 2015 ; Shi et al. , 2016 ; Yuan et al. , 2017 ; Wu et al. , 2017 ) . Instead , our reweighting scheme works without explicit data truncation and mining . It is motivated by the relative weight analysis between two examples . The current common practice of DML is to learn an angular embedding space by projecting all features to a unit hypersphere surface ( Song et al. , 2017 ; Law et al. , 2017 ; MovshovitzAttias et al. , 2017 ) . We identify the challenge that without sample mining , informative training examples can not be differentiated and emphasised properly because the relative weight between two samples is strictly bounded . We address it by sample reweighting , which rescales samples ’ gradient to control the differentiation degree among them . Finally , for intraclass compactness and interclass separability , most methods ( Schroff et al. , 2015 ; Song et al. , 2016 ; Tadmor et al. , 2016 ; Wu et al. , 2017 ) use distance thresholds to decrease intraclass variances and increase interclass distances . In contrast , we achieve the target from a perspective of instance-level matching probability . Without any distance margin constraint , ICE makes no assumptions about the boundaries between different classes . Therefore , ICE is easier to apply in applications where we have no prior knowledge about intraclass variances . Our contributions are summarised : ( 1 ) We approach DML from a novel perspective by taking in the key idea of matching probability in CCE . We introduce ICE , which is scalable to an infinite number of training classes and exploits structured information for learning supervision . ( 2 ) A seamless sample reweighting scheme is derived for ICE to address the challenge of learning an embedding subspace by projecting all features to a unit hypersphere surface . ( 3 ) We show the superiority of ICE by comparing with state-of-the-art methods on three real-world datasets . 2 RELATED WORK . 2.1 STRUCTURED LOSSES BY QUERY VERSUS CLASS CENTRES . Heated-up , NormFace , TADAM , DRPR , Prototypical Networks , Proxy-NCA . These methods calculate the similarities between a query and class centres ( a.k.a . proxies or prototypes ) instead of other instances ( Zhang et al. , 2018 ; Wang et al. , 2017a ; Oreshkin et al. , 2018 ; Law et al. , 2019 ; Snell et al. , 2017 ; Movshovitz-Attias et al. , 2017 ) . In Heated-up and NormFace , the class centres are learned parameters of a fully connected layer , which is similar to Center Loss ( Wen et al. , 2016 ) . While in TADAM , DRPR , and Prototypical Networks , a class centre is the mean over all embeddings of a class . By comparing a sample with other examples other than class centres , more informative instances can contribute more in ICE . 2.2 STRUCTURED LOSSES BY QUERY VERSUS INSTANCES . NCA ( Goldberger et al. , 2005 ) , S-NCA ( Wu et al. , 2018 ) . NCA learns similarity relationships between instances . Since original NCA learns the whole training data and its time complexity is quadratically proportional to the scale of training data , S-NCA is proposed recently with linear time complexity with respect to the training data size . Instead , ICE is scalable to infinite training data by iterative learning on randomly sampled small-scale instances matching tasks . S-NCA and NCA share the same learning objective . However , they treat the event of all similar instance being correctly recognised as a whole by a sum accumulator . Instead , we maximise the probability of every similar sample being correctly identified individually . Therefore , ICE ’ s optimisation task is harder , leading to better generalisation . N-pair-mc ( Sohn , 2016 ) . The aim of N-pair-mc is to identify one positive example from N − 1 negative examples of N − 1 classes ( one negative example per class ) . In other words , only one positive and one negative instance per class are considered per loss computation by simulating CCE exactly . Instead , ICE exploits all negative examples to benefit from richer information . When constructing mini-batches , N-pair-mc requires expensive offline class mining and samples 2 images per class . According to ( Sohn , 2016 ) N-pair-mc is superior to NCA . Hyperbolic ( Nickel & Kiela , 2018 ) . It aims to preserve the similarity structures among instances as well . However , it learns a hyperbolic embedding space where the distance depends only on norm of embeddings . Instead , we learn an angular space where the similarity depends only on the angle between embeddings . Besides , Hyperbolic requires a separate sampling of semantic subtrees when the dataset is large . 2.3 SAMPLE MINING AND WEIGHTING . Mining informative examples or emphasising on them are popular strategies in DML : 1 ) Mining non-trivial samples during training is crucial for faster convergence and better performance . Therefore , sample mining is widely studied in the literature . In pairwise or triplet-wise approaches ( Schroff et al. , 2015 ; Wu et al. , 2017 ; Huang et al. , 2016 ; Yuan et al. , 2017 ) , data pairs with higher losses are emphasized during gradient backpropagation . As for structured losses , Lifted Struct ( Song et al. , 2016 ) also focuses on harder examples . Furthermore , ( Sohn , 2016 ) and ( Suh et al. , 2019 ) propose to mine hard negative classes to construct informative input mini-batches . ProxyNCA ( Movshovitz-Attias et al. , 2017 ) addresses the sampling problem by learning class proxies . 2 ) Assigning higher weights to informative examples is another effective scheme ( Wang et al. , 2019a ; c ) . Beyond , there are some other novel perspectives to address sample mining or weighting , e.g. , hardness-aware examples generation ( Zheng et al. , 2019 ) and divide-and-conquer of the embedding space ( Sanakoyeu et al. , 2019 ) . Our proposed ICE has a similarity scaling factor which helps to emphasise more on informative examples . Moreover , as described in ( Schroff et al. , 2015 ) , very hard negative pairs are likely to be outliers and it is safer to mine semi-hard ones . In ICE , the similarity scaling factor is flexible in that it controls the emphasis degree on harder samples . Therefore , a proper similarity scaling factor can help mine informative examples and alleviate the disturbance of outliers simultaneously . What makes ours different is that we do not heuristically design the mining or weighting scheme . Instead , it is built-in and we simply scale it as demonstrated in Section 3.4 . 2.4 DISCUSSION . We remark that Prototypical Networks , Matching Networks ( Vinyals et al. , 2016 ) and NCA are also scalable and do not require distance thresholds . Therefore , they are illustrated and differentiated in Figure 1 . Matching Networks are designed specifically for one-shot learning . Similarly , ( Triantafillou et al. , 2017 ) design mAP-SSVM and mAP-DLM for few-shot learning , which directly optimises the retrieval performance mAP when multiple positives exist . FastAP ( Cakir et al. , 2019 ) is similar to ( Triantafillou et al. , 2017 ) and optimises the ranked-based average precision . Instead , ICE processes one positive at a time . Beyond , the setting of few-shot learning is different from deep metric learning : Each mini-batch is a complete subtask and contains a support set as training data and a query set as validation data in few-shot learning . Few-shot learning applies episodic training in practice . Remarkably , TADAM formulates instances versus class centres and also has a metric scaling parameter for adjusting the impact of different class centres . Contrastively , ICE adjusts the influence of other instances . Furthermore , ours is not exactly distance metric scaling since we simply apply naive cosine similarity as the distance metric at the testing stage . That is why we interpret it as a weighting scheme during training .

Summary: This paper proposes a new objective function called Instance Cross Entropy (ICE) for metric learning. Compared to the triplet loss (or contrastive loss) and its many variants, the distance between points in feature space is defined to be the dot/inner product, rather than computing the euclidian distance between feature vectors. Since the L2 norm of the final features is constrained  to be 1, the dot product represents the cosine distance between 2 feature vectors. Compared to the softmax/categorical loss, this objective has the advantage that there is no need to learn a per-class weight vector in the output softmax layer and therefore this method can be used even when the number of classes during training is unbounded. This is a very useful design feature, since the output softmax layer grows linearly with the size of the classes and can quickly become prohibitively large. Furthermore, mini batches for training can be randomly sampled without requiring expensive negative data mining strategies which are necessary when using the triplet loss. This is another very useful property.