Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper deals with the problem of metrics learning. It proposes to extend Mixap data augmentation from calssification to metrics learning. This Mixap data augmentation approach interpolates two or more examples (either directly in the input or the corresponding embeddings - eq.4) and corresponding target labels at a time. This task is challenging because unlike classification, the loss functions used in metric learning are not additive over examples and hence the authors claim that the idea of interpolating target labels is not straightforward.
The authors suggest a general formulation to accomodate mixup for many losses currenly used in metrics learning (see eqs. 3, 7, 9, 10 for generic formulation  and  eqs. 13, 14,15 for formulation of specific losses). The authors provie extensive experiments that show the effectiveness of their techqniue (Table 2)  on 4 different dataset (Table 4) 


main_review: Pros:
1) This is a good paper which has techincal novelty  - generalized loss for metrics learning, which allows to incorporate mixap augmentation. The authors also illustrate the effectiveness of the suggested technique for various losses previously used for Metric Learmning(Table 2). 
2) The paper also suggest theoretical analysis of the "positivity", which is used for loss definition, (Section 3.6 and Figure 1) which aligns with empirical results.
3) The paper also performs comparison to different mixup methods and shows its superiority (Table 3)


Cons:
1) The paper is hard to read because many important formulations are found in the appendix (e.g evaluation used for Table 2 and Table 3 Recall @ K, and what K stands for is only explained in appendix B1, the discussion about influence of the approach on the training time is only found in B3,  a more eleborative discussion on positivity is presented in A2 and there are more examples). So the paper might benefit from  substantial revision or maybe  a resubmission to a journal might be considered.
2) The paper introduces new evaluation metric, utilization, validating that a representation more appropriate for test classes is implicitly learned during exploration of the embedding space in the presence of mixup. What is not clear to me is if this metrics is measured on the classes that particiapte in training (After eq.12 -  Utilization measures the average, over the test set Q, of the minimum distance of a query q to a training example x ∈ X in the embedding space of the trained model f (lower is better)). If this is the case this metrics is not really interesting with respect to the actual goal of metrics learning (to be able to generalize for unseen classes) -- it would be helpful if the authors can clarify this point and also elaborate why this metrics is more interesting than other available metrics. 
3) Minor issue: In Table 4 SOP dataset - how can we get 5 examples per class if there are 22, 634 classes and 60, 026 images in training (so if we assume balanced dstribution of classes we will get ~2.65 images per class) -- are you using the same images?
4) Are you normalizing embeddings ? 
 


summary_of_the_review: I recommend to accept this paper because I think that the paper is relevant to the ICLR community and has some innovation supported by experiments (and some minor theory).  It also serves as a good survey for the community with many useful details for AI practicionaries in the field of metric learning, which are outlined in appendix. However, my concern is that the paper is not well organized and maybe better suited for a journal venue.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a generalized loss function for the purpose of using mixup in deep metric learning, which extends the some existing loss functions for deep metric learning without mixup. Extensive experiments are conducted to show the superior performance of the proposed method.

main_review: **Strengths**
1. The paper is very clearly written and easy to follow. I really like the part on building the loss function in (9) step-by-step. 
2. I really appreciate the extensive experiments that have been done in the paper. 
3. The generalization of the loss function seems to be pretty standard, but the simple solution turns out to be very effective.

**Weaknesses**
1. My main concern is in terms of the error function in (10). Is there any explanation why the first clean loss must be there? What if only the mixed loss term is used as an objective function.
2. I appreciate the authors’ effort on the analysis in section 3.6 although I am not convinced what is the message we can get from (11). Moreover, the objective function also has the clean loss term and the analysis is only in terms of the mixed loss.

Minor Issue
1. In Section 4.1 in mixup settings, M(a) only uses positive-negative and anchor-negative pairs, while in section 3.5 in the description of M(a) positive-positive pairs are also defined as a subset of M(a), would it hurt the experiment results if positive-positive pairs are also used?
2. In the utilization section, the word "significantly" is used which is very dangerous in statistics. The uncertainty is not quantified, therefore there isn't enough evidence to use this word.

summary_of_the_review: The paper is overall very clearly written and easy to follow, the authors made a great effort to conduct the experiments in the paper. I am not an expert in this area and may have missed some related work in this area. My main concern is in terms of the error function in (10) as described in the main review.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper mentions the missing of studying both metric loss function and data augmentation techniques for the metric learning problem and proposes to use a mixup strategy for the improvement.  The authors claim the better results over the state-of-the-art using the mixup and use a new metric (utilization) to claim their method is exploring new space. 
main_review: The paper proposes a mixup strategy to augment the data, interpolate the labels and design a new loss function for deep metric learning. Overall, the paper is well written and easy to follow. And the results are improved compared with other models. 

I am not an expert in this area so the following concerns might not be important but I wish the authors could clarify. From my point of view, the results are impressive. The datasets used in the experiment are also popular enough for supporting the claims. 

However, there are two concerns:

1) Although the authors propose the new general mixup loss function, the key elements of mixup from the paper follows from Zhang et al., 2018; Verma et al., 2019 and Venkataramananetal.,2021. In particular, the authors mentions "the idea of interpolating target labels is not straightforward." However, it seems that label interpolation is natraully derived from Zhang et al., 2018 in the paper. Could the authors please elaborate the key contributions from the paper? Is it the new generic loss function for mixup?

2) The authors propose a utilization method to support the claim of exporing new space using L2 distance in embedding space, which is not very convicing to me since L2 distance hardly captures information at the distribution level. From my point of view, the lower of the utilization score does not necessarily mean the model is exploring new and meaningful space. So could the author please elaborate on what is the key reason of using L2 distance in this case? 

summary_of_the_review: The authors propose a new and generic loss function in mixup scenario for deep metric learning. The experiments on popular datasets mostly support the claims in the paper with the needs of explaining some of ideas mentioned in the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a technique for using mixup augmentation in deep metric learning training. Specifically, the dml loss function is represented in a general form so that mixup loss can be easily computed for different pairs. This loss is combined with regular dml loss for training the network. The method is evaluated on CUB200, CARS196, SOP and IN-SHOP datasets where it outperforms it's baselines and also achieves SOTA results.
main_review: Strengths:
------------
1) A general form of loss function for DML that can be easily converted to specific losses like Contrastive, Proxy-anchor, Multi-similarity etc. + incorporating mixup to this.
2) Good experimental validation showing the effectiveness of the proposed approach compared to the baseline and the SOTA 

Weaknesses: 
-----------------
1) Limited contribution/significance: The two main contributions of the paper are representing the loss function in a general form and incorporating mixup into this loss function (using continuous labels instead of discrete labels as used in the existing dml). I think the idea of using a general dml loss function is inspired from Wang et. al 2019 which uses Multi-similarity loss as a general dml loss function. The loss function in Eqn. (3) is a modified version of MS loss Eqn. (2). Mixup has been extensively used in classification tasks and three different types (input/feature/embedding) have also been proposed previously. Although this is the first time mixup has been applied to dml, the contribution of the paper seems to be limited.

2) Limited scope: The proposed mixup approach is tailored to the dml problem in this work. Hence, its scope beyond dml is limitted. Although the paper mentions that the proposed approach can be extended to transfer/few-shot/continual learning, "our work may have applications to other such problems, including transfer learning, few-shot learning and continual learning", it would be difficult because the method assumes a particular form of loss function which may not hold true in other cases.

3) Experiments: It would interesting to see some images for the following cases:
a) the mixup training (for input mixup) images for different values of \lambda. 
b) how the tsne plots for the mixup samples would look like? 
c) the nearest neighbors for different values of \lambda (obtained by comparing the features in the embedding space)


summary_of_the_review: Although the proposed approach seems to be novel (applying mixup to dml), the scope of the technique is limited to the dml applications only and the contributions are marginal. Also, showing some images of the retrieval, tsne plots would be helpful and can provide more insights into the problem.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work explores the way of mixing both examples and target labels for deep metric learning. It considers metric learning loss and data augmentation technique (e.g., mixup) together when handling two or more examples at a time. A generalized formulation of loss function was modified to accommodate for the mixup technique. Also, a new metric called utilization is introduced for evaluating representation improvements. Some theoretical analysis are provided. A number of experiments were conducted on four benchmarks to show the superiority of the proposed method.
main_review: Pros.
1) It adapts a generalized formulation of loss function to the mixup technique, such that both examples (inputs) and target labels (outputs) can be integrated to yield better embeddings or representations.
2) It proposes a new metric named utilization to examine the improvements of representation in the embedding space.
3) It provides convincing empirical results on benchmark data sets.


Cons.
1) Previous works have shown that the effectiveness of the mixup for interpolating embeddings and labels (Zhang et al., 2018; Verma et al., 2019). This work borrows the same idea on input samples and labels rather than using embeddings. Such exploration should be encouraged, but the novelty is a bit weak. Other mixup variants might be better choices to improve novelty.
2) More analysis is required on error function in Eq. (10).
3) Some qualitative comparison results can be shown to further verify the improvement brought by mixup.
summary_of_the_review: This paper is generally well written and easy to read. It provides sound technique details with theoretical analysis. Also it presents lots of experimental results to support its claim that the mixup of inputs and outputs improves the representation quality. However, the novelty is of a bit limited. Overall, it is an acceptable paper that will attract attention from relevant researchers.
review_title: null

Meta-review: This paper adapts the mixup data augmentation strategy to the case of metric learning. The main challenge addressed is the fact that in metric learning, the loss function does not treat each example as an IID sample. The paper takes the view of metric learning as learning over positive and negative pairs (those belonging to the same/different classes) and uses this to develop a fairly general metric-mixup formulation. To measure the effectiveness of the approach for metric learning, the paper introduces a new measure called utilization that looks at the distance of a query point to its nearest training point in embedding space.

The reviewers (5 of them) all favour acceptance on the grounds of novelty, and the performance of the method. During the discussion, some issues were raised around whether utilization is a useful measure, improvements to the paper clarity, whether the clean loss in eq. 10 is necessary, and potential limitations on the generality of the approach. However, additional experiments and clarification during the discussion period has resolved these issues to their satisfaction.
Decision: Accept (Poster)