Extreme Triplet Learning: Effectively Optimizing Easy Positives and Hard Negatives

Abstract: The Triplet Loss approach to Distance Metric Learning is defined by the strategy to select triplets and the loss function through which those triplets are optimized.  During optimization, two especially important cases are easy positive and hard negative mining which consider, the closest example of the same and different classes.  We characterize how triplets behave based during optimization as a function of these similarities, and highlight that these important cases have technical problems where standard gradient descent behaves poorly, pulling the negative example closer and/or pushing the positive example farther away.  We derive an updated loss function that fixes these problems and shows improvements to the state of the art for CUB, CAR, SOP, In-Shop Clothes datasets.

Reviews and author responses:

-This trade-off between pushing away hard negatives and pulling in easy positives has actually previously been considered in the linear metric learning setting, though the terminology was different then. See the “Large Margin Nearest Neighbors” algorithm by Weinberger and Saul, for example which trains on “target neighbors” and “imposters.”

Very True!  The original LMNN paper creates a loss function that explicitly creates a separation —- much like the standard margin-based or NCA-based triplet loss function pushes and pulls to try to create the correct configuration.  What we show in this paper, however, is that when the triplet loss gradient is applied, it sometimes has the opposite effect.  This explains previous challenges observed in optimizing triplets that include hard negatives, and a very simple approach fixes this problem allowing extreme triplets to be effectively included in the optimization.

- Is projecting back to the hypersphere actually an issue? If the modelling assumption is truly that the learned representation should lie on the hypersphere, then this amounts to projected gradient descent and is a standard tool. In fact, projecting back to the sphere is correct under the model. If magnitude is truly important, then enforcing that the data be on the sphere is incorrect.

Many current embedding approaches force the points to lie on a hypersphere (or, more exactly, always compute similarity between representations first normalizing feature representations to be a unit vector, then computing the dot-product), some of the most recent papers are:

NIPS2016, Improved Deep Metric Learning with Multi-class N-pair Loss Objective(bottom of page3)
ECCV2018, Attention-based Ensemble for Deep Metric Learning(first paragraph of section 3.1)
CVPR2018, CosFace: Large Margin Cosine Loss for Deep Face Recognition

We follow this line of work, and consider points on the hyper-sphere, but demonstrate that the projected gradient descent approach struggles to optimize the extreme triplets for several reasons — among others, hard negatives consistent of two vectors that are very similar, and the gradient direction ends up roughly orthogonal to the surface of the sphere, meaning that after projection, the remaining gradient on the sphere is quite small.


- There are no error bars on plots or a discussion about how much variance a practitioner should expect running these experiments. If every single training run ever takes the same amount of time, who cares, but otherwise, we have one sample from each distribution and it’s hard to infer from that.

This is, unfortunately, the tradition among papers within this subfield (e.g. ECCV2018, Attention-based ensemble for deep metric learning; ICCV2017, Bier - boosting independent embeddings robustly), perhaps because the variation from run to run seems to be relatively small and seems to be relatively consistent across the datasets and network architectures in this subfield.  In our paper, we reported the median of 5 runs, the table below shows the range

Table 1 EPHN2nd Recall@1 result(%)

CUB
Median 65.2 
Range [64.8, 65.4]

CAR
Median 83.2
Range [82.9, 83.4]

SOP
Median 78.0
Range [77.8 78.1]

In-shop
Median 89.0
Range [88.8 89.0]



The reviewers point out paper [1], “Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning”. This is an important paper and we very much appreciate the pointer —- they provide a general framework for thinking about sampling and weighting, and use this framework to derive and algorithms with substantial improvements.

Our approach provides an alternative and complementary viewpoint.  We use a scatter plot based on similarity to the same and different classes to highlight different types of triplets.  For one common loss function, we derive the gradient of this triplet (how the updated network weights would change the relative similarities in the scatter plot).  Then we find that extreme triplets have specific problems, and we propose a weighting scheme to address these problems.  It is interesting that even though we arrive at different final solutions and algorithms, both algorithms offer approaches to change the training behavior for extremes (very easy positives or very hard negatives), and we believe this is something that should get wider attention.


Summary: The authors propose a new loss function as well as an adjoining visualization for improved performance of hard negative / easy positive mining for deep triplet metric learning. The authors note that under the NCA loss, if one selects an easy positive / hard negative and computes the gradient with respect to this example, this can lead to the negative example also being pulled closer to the anchor which is undesired. Similar phenomena can also be observed for easy positive / semi-hard negative mining as well. Motivated by this, the authors begin by designing a visualization to make this issue with NCA loss more apparent. Then they design what they refer to as an “entanglement factor” to quantify this issue more precisely. Using the desired dynamics of the gradients for the easy positive / hard negative mining and integrate to form what they refer to as the “second order loss.” Using this loss, they compare against the standard NCA loss on several datasets, showing modest performance gains. They also compare against a variety of other deep triplet embedding frameworks and show competitive results.
 
Comments, questions, and concerns:
- Overall the paper is well written and clear.
- The authors do a good job selecting reasonable baselines upon which to compare their method.
- It is worth noting, though not stated in the paper, that this trade-off between pushing away hard negatives and pulling in easy positives has actually previously been considered in the linear metric learning setting, though the terminology was different then. See the “Large Margin Nearest Neighbors” algorithm by Weinberger and Saul, for example which trains on “target neighbors” and “imposters.”
- Is projecting back to the hypersphere actually an issue? If the modelling assumption is truly that the learned representation should lie on the hypersphere, then this amounts to projected gradient descent and is a standard tool. In fact, projecting back to the sphere is correct under the model. If magnitude is truly important, then enforcing that the data be on the sphere is incorrect.
- The claim of a “systematic characterization for triplet selection strategies…” seems overly broad. It seems more correct to state this with respect to NCA loss specifically.
- NCA loss is just logistic loss penalizing the difference of similarities. The second order loss is the logistic loss penalizing (1/2 times) the squared difference of similarities, hence the cross term. Might be good to state this explicitly.
- The figures throughout are difficult to read due to the small font size.
- The y-axis in figure 4 is very different for each plot, which makes the effect seem much larger than it is.
- There are no error bars on plots or a discussion about how much variance a practitioner should expect running these experiments. If every single training run ever takes the same amount of time, who cares, but otherwise, we have one sample from each distribution and it’s hard to infer from that.
- Additionally, in Table 1, since there are no confidence regions, it is difficult to know if when the algorithm does perform well if these differences are significant. Especially important since on several datasets, the model does not perform as well as others. 


This paper uses the triplet scatter plot as a way to describe triplet selection strategies. The authors explain previously observed bad behavior for hard-negative triplet mining showing that it tends to make all points close to each other. The authors propose a simple modification to the desired gradients and derive a loss function that gives those gradients. With this modification, they show that easy positive hard negative (EPHN) gives results that exceed or are competitive with state of the art approaches. The paper is well-written and makes a convincing argument which will be of interest to a broad community.

Authors analyze the gradient of easy positive and hard negative pairs and propose a second-order loss for metric learning. Here are my concerns.
1.	The idea of weighting different pairs is not new and can be found in [1].
2.	The comparison is not convincing. Most of existing methods apply Inception as the backbone while this work adopts ResNet50, which has a better performance than Inception. Authors should adopt the same backbone for the fair comparison.
3.	The results in comparison are outdated. Please include the SOTA results, e.g., those in [1].
4.	The improvement from the second order term seems not significant. Besides, the second order term works as a data-dependent margin. Authors should compare it to a fixed margin to illustrate the effectiveness.


[1] CVPR’19: Multi-Similarity Loss With General Pair Weighting for Deep Metric Learning


Meta-review: The authors propose a novel distance metric learning approach. Reviews were mixed, and while the discussion was interesting to follow, some issues, including novelty, comparison with existing approaches, and impact, remain unresolved, and overall, the paper does not seem quite ready for publication. 

Decision: reject