Extreme Triplet Learning: Effectively Optimizing Easy Positives and Hard Negatives

1 INTRODUCTION . Deep metric learning optimizes an embedding function that maps semantically similar images to relatively nearby locations and maps semantically dissimilar images to distant locations . A number of approaches have been proposed for this problem ( Schroff et al. , 2015a ; Sohn , 2016 ; MovshovitzAttias et al. , 2017 ; Song et al. , 2016 ; Xuan et al. , 2018 ; Kim et al. , 2018 ; Ge , 2018 ) . A common way to learn the mapping is to define a loss function based on triplets of images : an anchor image , a positive image from the same class , and a negative image from a different class . The loss penalizes cases where the anchor is mapped closer to the negative image than it is to the positive image . One common variant of these functions ( e.g . ( Wang et al. , 2018 ; Sohn , 2016 ) ) , uses a Deep Learning framework to map images to a feature vector , and computes similarity between normalized feature vectors based on the dot-product . This approach forces the features to lie on a hypersphere and has advantages of making the feature comparison intuitive and efficient . In this work we explore standard implementations of these loss functions and show there are two problems . First , when the gradient of the loss function does not consider the normalization to a hypersphere , a large part of the gradient is lost when points are re-projected back to the sphere , especially in the easy-positive/hard-negative cases of triplets including nearby points . Second , when optimizing the parameters ( the weights ) of the network , when points are already mapped close together , it may be difficult to find gradient directions that effectively separate nearby images . We give systematic derivation showing when and where these challenging triplets arise , and diagram the sets of triplets where standard gradient descent makes the loss increase . We find that this explains problems previously reported in the literature such as the difficulty in optimizing hardnegative triplets ( Harwood et al. , 2017 ) . Furthermore , these problems are mostly introduced because the loss-function of the triplets is based on the differences between the anchor-positive and anchornegative distances , so there is an equivalent effect of encouraging the positive image to be closer or the negative image to be further . We create a new loss function that breaks this symmetry and weights the importance of changing the anchor-positive and anchor-negative distances . Briefly , our main contributions are : • A systematic characterization of triplet selection strategies and a visualization that highlights regions of bad gradient behavior . • A simple modification to a standard loss function to fix bad gradient behavior . • Improvements to current state of the art results across a range of datasets . 2 BACKGROUND . There is a large body of work in distance metric learning and they are leading with two main ideas . One of the idea is to increase the intra-class cluster density and keep the inter-class clusters as far as possible . Pairwise loss functions like contrastive divergence ( Carreira-Perpinan & Hinton , 2005 ; Chopra et al. , 2005 ; Radenović et al. , 2016 ) directly optimize for this constraint , and ” No fuss metric learning ” ( Movshovitz-Attias et al. , 2017 ) , implicitly optimizes for this constraint by assigning each class to different location and penalizes the failure of any example to go to its assigned location . The other approach more directly reflects the fact that for image retrieval applications , it is not necessary for all elements of a class to be clustered , but instead that the distance to elements of the same class should be smaller than the distance to elements of different classes . Directly optimizing for this is based on triplets of images . The triplets are drawn from the training data and include an anchor image , a positive image from the same class , and a negative image from a different class . Key questions for these approaches explore how to select the triplets that are used . Choices such as hard or semi-hard triplet mining ( Schroff et al. , 2015b ; Simo-Serra et al. , 2015 ; Wang et al. , 2014 ) focus on triplets with negative examples that are closest ( hard negative mining ) or nearly as close to the anchor as positive images ( semi-hard negative mining ) and emphasize creating separations between classes in the embedding space . Recent work such as easy positive triplet mining ( Xuan et al. , 2019 ) selects the closest anchor-positive pairs and ensures that at least they are closer than the nearest negatives . The next section introduces a diagram to systematically organize these triplet selection approaches , and to explore where different loss functions fail to improve the triplets . 3 TRIPLET SCATTER DIAGRAM . Triplet loss is trained with triplets of images , ( xa , xp , xn ) , where xa is an anchor image , xp is a positive image of the same class as the anchor , and xn is a negative image of a different class . We consider a convolution neural network , f ( · ) that embeds the images on a unit hypersphere , ( f ( xa ) , f ( xp ) , f ( xn ) ) . We use ( fa , fp , fn ) to simplify the representation of the normalized feature vectors . When embedded on a hypersphere , the cosine similarity is a convenient metric to measure the similarity of anchor-positive pair Sap = fᵀa fp and anchor-negative pair San = f ᵀ a fn , and this similarity is bounded in the range [ −1 , 1 ] . The triplet scatter diagram is an approach to characterizing a given set of triplets . Figure 1 represents each triplet as a 2D point ( Sap , San ) , describing how similar the positive and negative images are to the anchor . This diagram is useful because the location on the diagram describes important features of the triplet : • Triplets that are already in the correct configuration , where the similarity between anchor and positive is greater than the similarity between anchor and negative images are below the San = Sap diagonal . Dots representing triplets in the correct configuration are drawn in blue , dots where the negative is closer are drawn in red . • Triplets that include an anchor and the most similar of the possible positive examples are the ” Easy Positives ” and are on the right side of the diagram because Sap tends to be close to 1 . We circle these with a red ring . • Hard negatives are cases where the anchor is very similar to a negative example , so San is close to 1 , depicted as red dots circled with a blue ring . • Hard negatives are cases where the anchor is very similar to a negative example , so San is close to 1 , depicted as red dots circled with a blue ring . • One very selective mining strategy is ” Easy-Positive , Semi-Hard Negative ” , where an anchor is matched with closest possible positive match , and a negative example which has a similar similarity . The blue dot circled with red dashed circle highlights one such example . • Another selective mining strategy is ” Easy-Positive , Hard Negative ” , which selects , for an anchor , the most similar positive and negative examples . The red circle surrounded by a blue dashed circle represents one such example . During the later discussion , we may show a subset area of Ω , Ωs = [ 0 , 1 ] × [ 0 , 1 ] , because it is rare that the hardest negative or positive pairs have a similarity less than 0 . Figure 1 ( right ) calls out two specific regions of points that we analyze in the next section ; the extremes of hard negatives and easy positives , and the region that only includes positive similarities Ωs = [ 0 , 1 ] × [ 0 , 1 ] , that includes nearly all triplets constructed with easy positives and hard negatives . 4 DIAGRAMMING WHY SOME TRIPLETS ARE HARD TO OPTIMIZE . The triplet scatter diagram offers the ability to understand when the gradient based optimization of the network parameters is effective and when it fails . The triplets are used to train a network whose loss function encourages the anchor to be more similar to its positive example ( drawn from the same class ) than to its negative example ( drawn from a different class ) , encouraging Sap to be greater than San . While there are several possible choices , we consider NCA ( Goldberger et al. , 2005 ) as the loss function , and denote this as L1st to differentiate it from an updated loss function introduced later : L1st ( fa , fp , fn ) = −log exp ( Sap ) exp ( Sap ) + exp ( San ) ( 1 ) All of the following derivation can also be done for the triplet loss formulation used in ( Schroff et al. , 2015a ) ; this has a very similar form and is derived in the Appendix . The gradient of triplets loss L1st ( fa , fp , fn ) can be decomposed into two parts : a single gradient with respect to feature vectors fa , fp , fn : ∆L = ∂L ∂fa ∆fa + ∂L ∂fp ∆fp + ∂L ∂fn ∆fn , ( 2 ) and subsequently being clear that these feature vectors respond to changes in the model parameters ( the CNN network weights ) θ : ∆L = ∂L ∂fa ∂fa ∂θ ∆θ + ∂L ∂fp ∂fp ∂θ ∆θ + ∂L ∂fn ∂fn ∂θ ∆θ . ( 3 ) The gradient optimization only affects the feature embedding through variations in θ , but we first highlight problems with hypersphere embedding assuming that the optimization could directly affect the embedding location . To do this we derive the loss gradient with respect to the feature vector fa , fp , fn and use this gradient to update the feature locations where should decrease the error : fp new = fp − αgp = fp − α ∂L ∂fp = fp + βfa ( 4 ) fn new = fn − αgn = fn − α ∂L ∂fn = fn − βfa ( 5 ) fa new = fa − αga = fa − α ∂L ∂fa = fa − βfn + βfp ( 6 ) where β = α exp ( San ) exp ( Sap ) +exp ( San ) and α is the learning rate . This gradient update has a clear geometric meaning : the positive point fp is encouraged to move along the direction of the vector fa ; the negative point fn is encouraged to move along the opposite direction of the vector fa ; the anchor point fa is encouraged to move along the direction of the sum of fp and negative fn . All of these are weighted by the same weighting factor β . Then we can get the new similarity of anchor-positive and anchor-negative ( The derivation is given in the Appendix ) : Snewap = ( 1 + β 2 ) Sap + 2β − βSpn − β2San ( 7 ) Snewan = ( 1 + β 2 ) San − 2β + βSpn − β2Sap ( 8 ) These gradients ga , gp , gn have components that move them off the sphere , computing the cosine similarity requires that we compute the norm of fanew , fpnew and fnnew ( the derivation for them is shown in Appendix ) . Given the norm of updated feature vector , we can calculate the similarity change after the gradient update . ∆Sap = Snewap ‖fanew‖‖fpnew‖ − Sap ( 9 ) ∆San = Snewan ‖fanew‖‖fnnew‖ − San ( 10 ) Because the gradient of the loss function does not consider this normalization , following the negative gradient can actually cause some triplets to push the anchor closer to the negative example or push the anchor away from the positive example , even assuming that you can directly push the anchor , positive and negative features vectors in any direction . The direction in which fa , fp , and fn move depends on the relative position of the fa , fp , fn on the hypersphere . We use γ ( fully defined in the Appendix ) as a term to describe their relative orientation ; when fa , fn , and fp are close enough so that locally the hypersphere is a plane , then γ is the dotproduct of normalized vector from fa to fp and fa to fn . Therefore , if fp , fa , fn are co-planer then γ = 1 , and if moving from fa to fp is orthogonal to the direction from fa to fn , then γ = 0 . Given this description of the relative positions of the anchor , positive and negative points , Figure 2 shows calculations of the change in similarity between the anchor positive and anchor negative for γ = 0.5 . There is an area along the right side of the ∆Sap plot highlighting locations where the anchor and positive are pushed farther apart ( ∆Sap < 0 ) , and along the top of the ∆San plot highlighting locations where the anchor and negative are pulled closer together ( ∆San > 0 ) . This behavior arises because the gradient is pushing the feature off the hypersphere and therefore , after normalization , the effect is lost . This discussion so far considers the derivative of the loss as a function of the position of the feature vectors , but the optimization can only control the feature vectors based on the network parameters θ . Changes the θ are likely to affect nearby points in similar ways . For example , if there is a hard negative example with easy positive where the anchor is close to both the positive and the negative image , then changing θ to move the anchor closer to the positive is likely to pull the negative example along with it . We call this effect ” entanglement ” and propose a simple model to capture its effect on how the gradient update affects the similarities . We use a scalar p and a factor q = √ SapSan to quantify this entanglement . As for the factor q , when anchor , positive and negative are nearby to each other , both Sap and San will be large and q will increase the entanglement effect ; when positive or negative is far away to anchor , one of Sap and San will be small and q will reduce the entanglement effect . The total similarity changes with entanglement will modelled as follows : ∆Stotalap = ∆Sap + p √ SapSan∆San ( 11 ) ∆Stotalan = ∆San + p √ SapSan∆Sap ( 12 ) Figure 2 shows ( in color ) problematic regions where this model of gradient entanglement indicates that anchor and positive images become less similar ( Sap < 0 ) and regions where the anchor negative images become more similar for different parameters of the entanglement ( San > 0 ) . While figure 2 captures problematic regions on the scatter diagram , we can create a more complete description . The bottom row plots of figure2 shows the vector field on the scatter diagram , indicating that triplets move based on the gradient of their loss function . The figure shows several vector field plots with ∆Stotalap and ∆S total an with γ = 0.5 and p = 0.4 , 0.8 settings and indicates the 3 types of movement direction for dots on the triplet scatter . When ∆Sap > 0 and ∆San > 0 , the movement direction will point to up-right . When ∆Sap < 0 and ∆San < 0 , the movement direction will point to bottom-left . When ∆Sap > 0 and ∆San > 0 , the movement direction will point to bottomright . In fact , the entanglement strength may be varied in different situation during the optimization . The exact value will not be discussed in this paper . We only use the entanglement phenomenon to demonstrate problems that happen optimizing triplets with Easy Positives and Hard Negatives . Problem 1 : Hard Negative Mining For a given anchor image , hard negative mining chooses the negative example that maximizes San . The vector field computed in the case of entanglement shows that most locations with large San ( near the top of the plot ) have vectors with an upward component , meaning the gradient update for a hard negative triplet will push the negative even closer to anchor . The result is that a network can not effectively separate the negative pairs and tending to make all points close to each other . Initializing a fine-grained visualization tasks ( e.g . CARS196 ) with a generic pre-trained network like ImageNet often creates a starting conditions where all points are mapped nearby to begin with , leading to an optimization failure where all triplets move towards ( 1,1 ) on the triplet scatter diagram and all images are mapped to the same feature location . Problem 2 : Easy Positive Mining For a given anchor image , easy positive mining chooses the positive example that maximizes Sap . The vector field computed in the case of entanglement shows that most locations with large Sap ( near the right of the plot ) have vectors with a strong downward component , meaning the gradient updates pushes the anchor and negative to be negative related , which leads over-training . A general idea for a pair of images to be different is their similarity to be zero , the negative similarity value still means the pair of images are a kind of ’ related ’ . In the later phase of the optimization , as the optimization proceeds the triplet scatter diagram will not effectively keep triplets close to the ideal ( 1,0 ) point .

Summary: The authors propose a new loss function as well as an adjoining visualization for improved performance of hard negative / easy positive mining for deep triplet metric learning. The authors note that under the NCA loss, if one selects an easy positive / hard negative and computes the gradient with respect to this example, this can lead to the negative example also being pulled closer to the anchor which is undesired. Similar phenomena can also be observed for easy positive / semi-hard negative mining as well. Motivated by this, the authors begin by designing a visualization to make this issue with NCA loss more apparent. Then they design what they refer to as an “entanglement factor” to quantify this issue more precisely. Using the desired dynamics of the gradients for the easy positive / hard negative mining and integrate to form what they refer to as the “second order loss.” Using this loss, they compare against the standard NCA loss on several datasets, showing modest performance gains. They also compare against a variety of other deep triplet embedding frameworks and show competitive results.