Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper casts deep metric learning (DML) as a pairwise binary classification problem such that pairs of examples need to be classified as similar or dissimilar. The authors propose an objective function that computes a weighted sum over the pairwise losses in a mini-batch. The weight vector is selected to maximize the objective from a decision set encoding constraints. This formulation is called the distributionally robust optimization (DRO) framework.

The authors argue that the DRO framework is theoretically justified by showing how certain decision sets result in existing machine learning loss functions. This portion of the paper seemed hand-wavy. It is not clear what is the purpose of including the theorem from Namkoon & Duchi. It would be more clear in my view to just make the short point that a certain decision set recovers the DRO with f-divergence as would be expected. The claims with regard to learning theory are over-stated in the paper.

The authors proposed three variants of the general framework. They include a top-K formulation, a variance-regularized version, and a top-K version using a balance between positive and negative examples. The DRO framework and the variants are the main contributions in terms of methodology in this paper. It is also shown that the framework generalizes more complicated recently proposed losses.

The experiments demonstrate the DRO framework consistently outperforms state of the art deep metric learning methods on benchmark datasets by small margins. There is also a computational speed advantage that is shown.
Overall, this paper shows that the ideas from distributionally robust optimization work well in deep metric learning. In particular, the paper shows that by combining the DRO framework with simple loss functions, performance comparable with complicated loss functions can be obtained. This aspect, along with the generality are the main strong suits. That being said, I do not see this paper to be that significant of a contribution. The main idea in the paper seems like a rather direct application of the DRO modeling framework and it does not provide too significant of improvement over the MS loss.  The paper was not written super clearly and was too long. Reviewers were instructed to apply a higher standard to papers in excess of 8 pages and this paper would have been presented more effectively if it was shorter. For these reasons, I recommended a weak reject.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The authors address the (increasingly popular) problem of learning a metric from a given multi-dimensional data set. They consider the deep metric learning setup, where target distances are defined as the euclidean distances in an artificial feature space (created by a deep neural network). Main focus of the paper is to cases where the data set is affected by a substantial imbalance between the amount of examples that are similar to each other and the total number of examples.

I would tend to accept the paper because handling the imbalance problem in metric learning is important and both the theoretical analysis and the experiments show that the proposed method may have some impact.

The idea of reducing the problem to a binary classification between similar and dissimilar examples may look too simple but i) is a common approach in deep metric learning, ii) helps to handle the implicit imbalance problem and iii) suggests possible generalisations to other network-based problems (for example, where similarity is naturally defined by the existence of absence of a link). Showing that many complicated losses are equivalent to DRO may also help the general understanding of the metric learning task.

My main concerns are about the net contribution of the paper. Tackling the imbalance problem is important but it is not clear whether the full metric learning setup is really needed. The authors could have stated more precisely in what sense the metric learning unbalanced problem they consider is different from usual unbalanced binary classification. Otherwise, as DRO is well known, it is hard to identify the real novelty of their method.

Questions: 
- how does the specific metric learning setup make the considered DRO different from usual unbalanced classification? 
- how the network architecture affects the performance? For example, would the size of the embedding space change the recall/imbalance plot? 
- Is the choice of euclidean distances standard in deep metric learning? Would a choice of more general distances be incorporated in the proposed method?
 

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a framework for deep metric learning. Using ideas from distributionally robust optimization, the loss (in each batch) is the worst case weighted average of all pairwise classification losses, taken over an uncertainty set of possible weights. The framework is shown to be general and encompass various previous approaches. Based on it, the authors propose several new algorithms, which are shown to outperform the SOTA on image retrieval data sets in terms of recall.

The main contribution of the paper is a unification of previous deep metric learning algorithms, which would be helpful to the community and could inspire new approaches. I found the empirical observation that the proposed algorithms are able to reduce the computation time by nearly half to be compelling. However, apart from DRO-TopK-PN, the proposed algorithms appear to be minor modifications of existing algorithms. 

Questions about the experimental protocol:
1. Are the results from one run, or averaged over several? Standard errors of the evaluation metrics would be very helpful to judge the improvements made by the algorithms, especially as the algorithms are stochastic due to batching. 
2. The proposed algorithms seem to be similar to those of Fan et al. (2017) and Namkoong and Duchi (2017). Is there a particular reason why they weren’t included in the experiments? 
review_title: Official Blind Review #2

Meta-review: The reviewers agree that this is a reasonable paper but somewhat derivative. The authors discussed the contribution further in the rebuttal, but even in light of their comments, I consider the significance of this work too low for acceptance.
Decision: Reject