Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper tackles the question of why Deep Metric Learning (DML) approaches are unable to make more than incremental improvements on classical datasets and tasks. Specifically, the trained models fail to recognize similar samples and tend to make the same kind of generalization errors. To better understand the problem, the authors examine the particular training examples that contribute to the generalization errors that the model makes, and discover that many of these examples are labelled incorrectly. This leads the authors to conclude that the major barriers for DML performance is not model design, but rather the significant presence of incorrectly labelled examples in the classical datasets. To address this issue, they propose an empirical influence function (EIF), which identifies these examples and quantifies their contribution to the generalization error. Moreover, they propose a relabelling technique based on the EIF for the problematic examples. 
strengths_and_weaknesses: This is an interesting paper - identifying examples that are mislabelled in large datasets is clearly an important challenge. Figure 2a-c of the appendix is quite compelling evidence for the claims made about these well known datasets in the introduction, and it might be worth adding these panels to the main text. 

The authors propose a novel empirical influence function, that tackles the significant run time cost of the existing influence function (IF) and an issue that the authors identify with the accuracy of an approximation made for the calculation of group-pair confusion. I am not an expert in the area of influence functions, but to my knowledge this is a novel proposal and its efficacy is supported by the experimental results that the authors present in the paper. 

Identifying mislabelled examples that prevent performance on benchmark tasks from improving is clearly a significant service to the community. 
questions: My main questions are around the improvement that EIF provides over IF. 

In the DML training experiment, it would be interesting to know how the statistics describing the improvement of EIF over IF change as the number of confusing pairs considered is varies. Table 1 of the manuscript reports the statistics when the 50 most confusing pairs are considered - but what happens if the number of pairs is increased or decrease? The statistics for each dataset could be plotted as a function of the number of pairs considered. Similarly the number of confusion groups could be varied from the value 10 for which statistics are currently reported. 

Lines 167-168 states 'Compared to IF, EIF identifies samples with which retraining the model can increase larger distance for the confusion pair.'. To make this achievement more concrete for the reader, it would be helpful for the authors to include some examples, perhaps in the appendix, and to refer to these examples at this point in the text. There are some examples of potential mislabelling given in the appendix, but it is not clear how these examples were identified (I couldn't find information about them in the appendix or the main text). 

I understand that removing the need to calculate the Hessian greatly improves the runtime over IF, but it would be helpful if the authors can provide some intuition for why the accuracy also improves for the individual pairs. What is the dependence on the choice of N_theta for the DML training experiment, and why in Figure 2 is N_theta relabelled as M? 

In Figure 2, for the noisy relabelling experiment, please could the authors comment on why the performance of EIF and IF saturate for the middle tranche of training examples, before rapidly increasing as the last few training examples are checked? 

The field study should be carried out for the IF pairs in addition to the EIF pairs, so that Table 4 can be augmented with this data. 

Overall there are a few places where the writing could be improved - the meaning is generally more or less clear, but the paper would benefit from a close read though - in some places words or even parameters/variables appear to be missing from sentences, and have to be inferred by the reader from context. 
limitations: It would be helpful if the authors could include a discussion of the limitations of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose an empirical influence function which:
(a) Quantifies the impact of training samples on generalization errors in deep metric learning and proposes a relabeling framework in the context of DML.
(b) Proposes a faster version of the traditional influence function which is predominantly slow.
strengths_and_weaknesses: Strengths:

(1) The work is very significant practically as there exists very few works concerning understanding influence functions in contexts other than supervised learning (e.g., DML). To the best of my knowledge, this is one of the only works which investigate model interpretability through the lens of the training data in DML.

(2) While influence function is quite useful for downstream tasks such as relabeling, it is very expensive to compute. The work is also significant in the sense that they circumvent the expensive inverse-Hessian-vector product approximation which is a bottleneck in influence computation.

(3) I have reviewed an earlier version of this paper in another conference. This draft is very well organized and easy to follow.  The authors have also updated the list of references which was inadequate in the earlier draft. 

(4) The authors cover a wide range of experiments on most of the commonly used datasets in DML. The empirical analysis supports the results well.

Weaknesses:
(1) While speeding up the influence function via a light weight Newton step approximation, it is not technically novel. However it's still impactful as often it reduces the running time by 10-20 times at the least. 

(2)  The authors primary focus on the DML setup and it is fair to not be able to cover different areas of applications. However it would strengthen the paper on how the EIF fares in the supervised setup too. A good idea could be to compare the run-times and also compare the quality of influence values obtained. Considering one of the major contribution is the fast version of IF, it would be beneficial to observe its universality.

(3) Does the selection of backbone architectures play a role in determining the quality of influence values? 
questions: Please refer to my questions in the Weaknesses section.


limitations: I feel the paper is a good contribution, though the technical contribution is a bit limited. However, I still feel that the paper should be accepted due to : (A) Application of IF in a new domain such as DML; (b) Good coverage of experiments; (c) Simple strategy to scale up influence functions.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 
The authors propose EIF, which is a modification of IF, for DML.  Specifically, it differs from IF in two key ways:
- It doesn't require computing the Hessian, which leads to faster calculations. 
- It allows for the possibility that the "retrained" parameters may be far away from the original parameters, which is useful when a large portion of the dataset is being re-weighted. 

They argue for the efficacy of this method with several sets of results based on datasets with deliberately mislabeled points:
- Showing that the confusion of the most confused pairs  decreases more (statistically significantly) when the model is retrained on points reweighed by EIF than IF (Table 1)
- Showing that EIF is able to identify those mislabeled points (Figure 2)
- Showing that EIF+KNN is able to correctly relabel those mislabeled points (Table 2)

They show that this has computational benefits compared to IF (Table 3)

Additionally, they argue for the efficacy of this method on benchmark tasks:
-  Showing that people often agree with the model's mistakes and, as a result, that the problem is the data labeling (Table 4)
- Analyzing how often EIF identifies different types of data labeling errors
strengths_and_weaknesses: Originality:  
- As far as I know, IF has not been applied to DML before
- EIF is, technically, a novel extension of IF

Quality:  Overall, the methodology seems solid
- Having 2 students check the authors conclusions for Table 4 is a little unusual
- Where is the claim in Line 244 supported?

Clarity:  
- Line 70-71, it's worth clarifying what "L" is here because, as written, it sounds a lot like you are going to be doing normal supervised learning. 
- Equations (1) and (2) are somewhat difficult to parse.  My understanding is that the objective is to solve the 2nd line of these equations and that the first line is like a constraint.  
- Equation (2):  Should "l:X->Y" be "R:X->Y"?
- Line 113:  what are "delta d" and "delta L"?  I'm assuming that they are the difference in these metrics for the new and original parameters
- Relatedly, is "delta L" assumed to be negative?  Otherwise, I'm not seeing the intuition for the interpretation of its sign in Lines 114-115.
- Line 156:  how is the distance normalized?
- Clarifying what it means for EIF+KNN to "recommend a new label" would be helpful (especially for understanding table 2).  My guess is that it is "EIF says a point is harmful and KNN assigns it a label different from the provided one" but I am not certain.  

Significance: 
- This seems like a potentially important problem
- Beyond the the computational benefits, it isn't overwhelming clear the EIF is more effective than IF (see Questions)
- The claims about problems facing DML datasets don't seem especially insightful (some classes have similar images, classes are sometimes similar, classes often have outliers)
questions: How should readers compare/interpret the results in Table 1 and Figure 2?  
- Table 1 shows that, for the most confused pairs, EIF is better able reduce their confusion than IF.  However, EIF only really appears to beat IF on CUB200 in Figure 2.  What is causing the disagreement between these results?
- Relatedly, all of the influence estimators show a high recovery rate when the fraction of the dataset checked is near 0 or near 1.  Were the influences scores sorted by absolute value?  

Doesn't merging similar classes simply make the problem easier?  Fundamentally, it seems that a good model should be able to differentiate similar classes.  So this doesn't seem like a dataset problem that we should try to "fix".  
limitations: Yes
review_title: null

Meta-review: metareview: This paper approaches the problem of debugging failures in deep metric learning, developing and applying a more efficient version of influence functions called EIF (empirical influence functions) that is shown to effectively help root-cause failures to ambiguous or poorly labeled examples in standard training sets.

The reviewers found this paper to be deeply interesting, appreciating the efficiency but moreover the novelty of applying IF-style approaches to the problem of DML.  From my perspective, I think the novelty point is even stronger than some of the reviewers called out -- yes, it there is some novelty in the development of EIF, but the real novelty is in creating and evaluating effective debugging strategies for learned metric spaces.  This kind of analysis and debugging is sorely lacking in the overall literature, and I am very happy to see this work present a compelling and intuitive approach -- both for its own merit, and also for the similar ideas it may well inspire in adjacent areas of research.
confidence: Certain
Decision: Accept