Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Authors analyze the gradient of easy positive and hard negative pairs and propose a second-order loss for metric learning. Here are my concerns.
1.	The idea of weighting different pairs is not new and can be found in [1].
2.	The comparison is not convincing. Most of existing methods apply Inception as the backbone while this work adopts ResNet50, which has a better performance than Inception. Authors should adopt the same backbone for the fair comparison.
3.	The results in comparison are outdated. Please include the SOTA results, e.g., those in [1].
4.	The improvement from the second order term seems not significant. Besides, the second order term works as a data-dependent margin. Authors should compare it to a fixed margin to illustrate the effectiveness.


[1] CVPR’19: Multi-Similarity Loss With General Pair Weighting for Deep Metric Learning

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper uses the triplet scatter plot as a way to describe triplet selection strategies. The authors explain previously observed bad behavior for hard-negative triplet mining showing that it tends to make all points close to each other. The authors propose a simple modification to the desired gradients and derive a loss function that gives those gradients. With this modification, they show that easy positive hard negative (EPHN) gives results that exceed or are competitive with state of the art approaches. The paper is well-written and makes a convincing argument which will be of interest to a broad community.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary: The authors propose a new loss function as well as an adjoining visualization for improved performance of hard negative / easy positive mining for deep triplet metric learning. The authors note that under the NCA loss, if one selects an easy positive / hard negative and computes the gradient with respect to this example, this can lead to the negative example also being pulled closer to the anchor which is undesired. Similar phenomena can also be observed for easy positive / semi-hard negative mining as well. Motivated by this, the authors begin by designing a visualization to make this issue with NCA loss more apparent. Then they design what they refer to as an “entanglement factor” to quantify this issue more precisely. Using the desired dynamics of the gradients for the easy positive / hard negative mining and integrate to form what they refer to as the “second order loss.” Using this loss, they compare against the standard NCA loss on several datasets, showing modest performance gains. They also compare against a variety of other deep triplet embedding frameworks and show competitive results.
 
Comments, questions, and concerns:
- Overall the paper is well written and clear.
- The authors do a good job selecting reasonable baselines upon which to compare their method.
- It is worth noting, though not stated in the paper, that this trade-off between pushing away hard negatives and pulling in easy positives has actually previously been considered in the linear metric learning setting, though the terminology was different then. See the “Large Margin Nearest Neighbors” algorithm by Weinberger and Saul, for example which trains on “target neighbors” and “imposters.”
- Is projecting back to the hypersphere actually an issue? If the modelling assumption is truly that the learned representation should lie on the hypersphere, then this amounts to projected gradient descent and is a standard tool. In fact, projecting back to the sphere is correct under the model. If magnitude is truly important, then enforcing that the data be on the sphere is incorrect.
- The claim of a “systematic characterization for triplet selection strategies…” seems overly broad. It seems more correct to state this with respect to NCA loss specifically.
- NCA loss is just logistic loss penalizing the difference of similarities. The second order loss is the logistic loss penalizing (1/2 times) the squared difference of similarities, hence the cross term. Might be good to state this explicitly.
- The figures throughout are difficult to read due to the small font size.
- The y-axis in figure 4 is very different for each plot, which makes the effect seem much larger than it is.
- There are no error bars on plots or a discussion about how much variance a practitioner should expect running these experiments. If every single training run ever takes the same amount of time, who cares, but otherwise, we have one sample from each distribution and it’s hard to infer from that.
- Additionally, in Table 1, since there are no confidence regions, it is difficult to know if when the algorithm does perform well if these differences are significant. Especially important since on several datasets, the model does not perform as well as others. 

review_title: Official Blind Review #3

Meta-review: The authors propose a novel distance metric learning approach. Reviews were mixed, and while the discussion was interesting to follow, some issues, including novelty, comparison with existing approaches, and impact, remain unresolved, and overall, the paper does not seem quite ready for publication. 
Decision: Reject