Debugging and Explaining Metric Learning Approaches: An Influence Function Based Perspective
Authors: Ruofan Liu
Yun Lin
XIANGLIN YANG
Jin Song Dong
Venue: NeurIPS 2022 Accept (2022)
Keywords: metric learning
influence function
noisy data

Abstract: Deep metric learning (DML) learns a generalizable embedding space where the representations of semantically similar samples are closer. Despite achieving good performance, the state-of-the-art models still suffer from the generalization errors such as farther similar samples and closer dissimilar samples in the space. In this work, we design an empirical influence function (EIF), a debugging and explaining technique for the generalization errors of state-of-the-art metric learning models. EIF is designed to efficiently identify and quantify how a subset of training samples contributes to the generalization errors. Moreover, given a user-specific error, EIF can be used to relabel a potentially noisy training sample as mitigation. In our quantitative experiment, EIF outperforms the traditional baseline in identifying more relevant training samples with statistical significance and 33.5% less time. In the field study on well-known datasets such as CUB200, CARS196, and InShop, EIF identifies 4.4%, 6.6%, and 17.7% labelling mistakes, indicating the direction of the DML community to further improve the model performance. Our code is available at https://github.com/lindsey98/Influence_function_metric_learning.

Debugging and Explaining Metric Learning Approaches: An Influence Function Based Perspective
Ruofan Liu Shanghai Jiao Tong University National University Singapore ruw.anuuannnnn nus. edu
Yun Lin"
Shanghai Jiao Tong University National University of Singapore
11mhyy@gmail. com
Xianglin Yang
Jin Song Dong
National University Singapore kianglin@u nus edu
National University of Singapore dcsdjs@nus wew.suuusueuu....... sg
Abstract
Deep metric learning (DMLL) learns generalizable embedding space where the representations of semantically similar samples are closer. Despite achieving good performance, the state-of-the-art models still suffer from the generaliza- tion errors such as farther similar samples and closer dissimilar samples in the space. In this work, we design empirical influence function (EIF), a debugging and explaining technique for the gemeralization errors of the state-of-the art met- ric learning models. EIF designed to efficiently identify and quantify how subset of training samples contribute to the generalization errors. Moreover, given a user-specific error, EIF can be used to relabel a potentially noisy train- mitigation. In quantitative experiment, EIF outperforms the traditional baseline in identifying more relevant training samples with sta- tistical significance and 33.5% less time. In the field study the well-known datasets such as CUB200, CARS196, and InShop, EIF identifies 4,4%, 6.6%, and 17.7% labelling mistakes, indicating the direction of the DML community to further improve the model performance. Our code is available at lttps; // gecithub.com/iinncii httpe//eeeeeeeettttttoooootorrrr
ing sample
Introduction
Deep metric learning (DML) learns generalizable embedding space of dataset, where semantically similar samples are mapped closer [1]. It has been widely applied face recognition [27]. image retrieval [38), and clustering [5] Recently, the record-breat brorddbbeaking methodologies have been generally
evolving from pairwise- based approaches (e.g.. triplet-based [22 and wairdbased IT]) to proxy- proxy-based
approaches 20 [4] 31 10
[ However, many recent works 28 25 begin achieve only marginal
improvements on the classical datasets 337 18 19.. Thus, the explanation approaches of DML are in
need for understanding why the trained model confuses the dissimilar samples and fails recognize the similar samples.
This research starts with our investigation popular classical datasets (i.e., CUB200, CARS196, and InShop) that, state-of-the-arr metric learning approaches, (1) different approaches share not only similar performance metries (e.g., Recall @ 1), but the same types of generalization errors, and (2) the human inspection sometimes has no less generalization errors on existing DML datasets. We
* *Correspondigg author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

The observation leads us to design an influence function based explanation framework to investigate the existing datasets, consisting of:
report the results at Section 5 and our website [2]
Scalable training-samplee attribution: We propose empirical influential function (EIF) to (1) identify what training samples contribute to the generalization errors, and (2) quantify how much contribution they make to the errors. Technically, we replace the Hessian matrix in traditional influence function [|5 with light weighted Newton step estimation to improve both its effectiveness and efficiency.
Dataset relabelling recommendation: We further aim to identify the potentially "buggy" training samples with mistaken labels and generate their relabelling recommendation.
Technique Evaluation. We design one sample-locating experiment and one sample-relabelling experiment on datasets to evaluate our framework. the sample-locating experiment, we evaluate EIF's performance locating the influential training samples for both individual- confusion (i.e., model is confused on one pair of samples) and group-pair confusion (i.e., model is confused on a class of samples). The results show that, compared to the traditional influence function [13], EIF identifies root-cause training samples which can significantly mitigate the confusion distances with 33.5% less time. As for the sample-relabelling experiment, we inject 10% noisy training samples to train a DML model. Our results show that we can accurately recommend on average 88.2% of the mis-labelled training samples.
Empirical Investigation. Based on the proposed framework, we investigate the classical datasets such as CUB200 (37.. CARSI96 [8] and InShop [T] We find that the labels annotated in the datasets are more unreliable than expected. We summarize taxonomy of dataset problems (e.g., see Section 5). We further conclude that the major barriers for DML performance might not be the model design, but the confusing labels in the classical datasets.
In summary, this work makes the following contributions:
We propose empirical influence function (EIF) for DML approaches, which can attribute root-cause training samples for arbitrary number of pairs of unseen test samples.
We propose sample-relabelling technique based on EIF for mitigating potential dataset problems.
We identify and categorize labelling problems of the well-known classical datasets for DML, indicating the potential direction to further improve the performance of DML approaches.
2 Problem Setting
We denote the input space as X C Rd (d the input dimension), the embedding space as Z C R" (m the embedding dimension), and the class label space y C -=+ DML network f(.) parameterized by 0 is denoted as fo X - Z. Given distance measure d(.,.) where d 2 x 2 - {0, R+ J. we can calculate the distance between any input pairs (x,, x,) E X by (x)o(x;). fx(x))) Typically, cosine distance and Euclidean distance are common choices of d(.,.). In addition, we define labelling function R(.) where R : + +.
The DML techniques aim to optimize O such that VX,, Xj, X E X (gi Mys E y + yk). < d( (x;), fo(xx)) Pair-based losses try to optimize the inequality directly on pairs/ / triplets [1 29 However, studies have shown that the optimization suffer from (1) slow and noisy convergence, (2) high computational complexity. Therefore, proxy- based losses have been proposed to address these issues 20 RI 14 23 6 which significantly outperform the pair-based losses. Most of proxy-based losses are defined on a per-sample basis 20 23 Therefore, in this work, we follow the loss of form L(x;0)
Given the training dataset set Xrain with labels Virola and the testing dataset Xiess with labels Viea where training and testing are class disjoint, i.e., J'main n Vreer = 0. The generalization error can be defined as a testing sample not sharing the same class label its nearest neighbor in the space, i.e.,
Definition 2.1. We define testing- sample pair p (x,, x;) (x;, Xy X as confusion pair if:
1. 2j fo(x,) is the nearest neighbour of z, fx(x;) in the distance space;
2. V # yy and yis Yj Veen

---

Given set of confusion pairs Pe = {PI. Pe. we aim to achieve the following two goals:
G1. Influential Sample Identification We aim to locate the of root-cause training samples X, C Xrain such that retraining with re-weighted X, can increase the average distance of confusion pairs in the most.
X, arg max X, &asis
Σ
d(xi, Xji xx; subject
Pal
(x.xx))E.
(1)
0, = arg min
Σ
U1: 01;
+
Σ X,EX,
e,((x, 91; 0)]
Ayaai
X/E
In Equation 0, is the retrained model by re-weighting the training sample x; by e, Specifically, le, is the re-weighting magnitude, if x₃ is helpful training sample (i.e. helpful de-confusion). then E is set to be greater than one; otherwise for harmful training, is set to be less than one. Under certain choice of X, the average distance of the confusion pair set Pa is maximized.
G2. Influential Sample Relabelling We aim to find the set of root-cause training samples di C Kraai and relabelling function R X' - y such that retraining 0 by changing the labels of X; with R()) can increase the average distance of confusion pairs in Pa the most. Specifically,
X arg max
Σ
d(xi, xj; 0.) subject
Pel
NCC. M.C.
(x.,))EP.
(2)
0, = arg min
1
Σ
[(xi, yz 0)+ 01;
Σ X.EX
L(x,, R(x:):0)
Xyrin
3 Approach
Recaping Influence Function Given that learned model parameterized by 0, and its DML loss [(x; 0). The influence of up-weighting a training sample Nyrain on a testing sample Xtest is 15
I (Xtrain; Xtest) L( ((xeest; 0') L(xtest; 0) ~ -V 0) THT''V L
In Equation 3 is the original model, 0' the retrained model after Xtrain is adjusted, Ho!
Xmntal 1=1 VX 0). The first term can be interpreted the testing sample's 's sensitivity the model's parameters, the second term estimates the interaction between training samples, and the third term is the influence of this training sample to the model. In DML settings, testing loss is undefined according to Equation since testing classes are unseen. Given confusion pair Pa (X;, X₃) with
Amate
distance d(pe; O), we can replace the testing loss with d(pe; 0). i....
I(Xtrain Pc) - Vad(pe; 0) H1. " C(xtrain 0)
However, computing I(Xtrain: P.) still suffers from two drawbacks, i.e., (1) high computational cost and (2) inaccurate approximation for group-pair confusion.
Computing the Hessian function Ho1 is non-trivial, which requires the complexity of (mp² + p²) where is training dataset size and is the parameter size. In [15], the complexity further reduced to (mp + rlp) where ~ O(n). With the increase of parameter size (c.g., millions) and the training set, the runtime cost is still considerably large.
High Computational Cost

---

cop(Ad(P, Al(x s..)
Sphere(0,, r)
0u
cop(ád(P,. ALix o-))
cop(Ad(P,, Al(x;0))]eee
Figure 1: Illustration of EIF calculation by sampling most representative g(m²) on the hypersphere.
Accuracy of Influence of Group-pair Confusion When we are locating the influential training samples for group of confusion pairs, the assumption of Eqqution 4 cannot hold in practice because the derivation of the Hessian matrix Hzg in Equation 3 requires that 0' ~ so that Taylor expansion can be applied [5] However, the number of influential training samples to group of confusion pairs (i.e., X, in Equation 1) can be large, indicating that the adjusted parameters é' could be very different from the original 0. Thus, such approximation can lead inaccurate estimation.
Empirical Influence Function (EIF) In this work, we design empirical influence function (EIF) to address the above challenges. Our rationale lies in that, (1) fitting harmful samples contribute to more generalization errors, and (2) fitting helpful samples contribute to mitigating the generalization errors. We estimate the influence scores by the empirical co-change product between the distance of confusion pairs Ad(Pe. Pr; 0) and the training loss AC(x; 0) (x E Xirain) as Equation 5)
(M(PP;))A((<x) 0)
I(x, P.) Eolcop (e 0), AL(x;
p(0)do
11-00.1 0
Assume that we train the network by upweighting x to have 0g - 0, the co-change product is defined as cop( (dd((d(( 0), AL(x; 0)) = 10-8ol1
Ad(r,.0)A(xx:0) cop( Ad( (P; 0). AC(x; 0)) > 0 indi-
(with ((x: 0)) < 0) co-happens with more confused pairs (with Ad( Pc; 0) < 0) or (2) less fitting on (with (x; f)) > 0) co-happens with less confused pairs (with (d(P:: 0) > 0). On the other hand, cop( Ad( : 0), (x O)) < 0 indicates that x is helpful. However, retraining each individual
cates that is harmful. Intuitively, is harmful distinguish P. if (1) more fitting
sample prohibitively expensive, therefore we need to estimate the integral by carefully sampling representative 0's only.
As shown in Figure assume that Ou is the model parameter, θ is the parameter after re-retraining, the distribution of 9 forms a nigh-dimensional Sphere p(0) ~ Sphere(Oo, with a radius r. where is some pre-defined bound to limit the perturbation step. The intractable integral in Equation S estimated by drawing Monte Carlo samples (e.g., ((1), ((2) ((m) on the sphere. i.e., we sample C Sphere(Oo.r)
1
Ad(P:;
(())
(P)
(10(m)
00012
((m)
In this work, instead of random sampling, heuristically construct O regarding how sensitive each ((m) E to the change of d(P.::) 0). Specifically, we first sample @ maaz E Sphere(On. r) pointing to the steepest ascent direction of d( Pa; 0), which can be considered as "repairing" the network. Symmetrically, Ouzin taken as the steepest descent direction of d(( Pc; 0), which can be considered as "worsening" the network. Next, given user-defined threshold No. we repetitively sample (m)) E Sphhreeee..r (9u. r) which is orthogonal to 40 E O until No exhausts. θ is calculated once and shared across the whole training set. Our experiment shows that even using θ = (e (mmmr) achieve accurate influence estimation, with well improved runtime efficiency.
The complexity of our empirical influence function on the whole training set is O(p) + O(Ntrain X Ne x p), where p is the parameter size. The first term stands for the complexity of constructing O

---

Table Paired t-test results for alternative hypothesis on individual confusion pairs H₃ (dFIF(pR) > (prr(Pa). And Wilcoxon signed-rank test results for alternative hypothesis on groups confusion pairs H1 AdEIF
Adrr Cm
Individual confusion
CUB200
CARSI96
InShop
confusion pains
average improvement
p-value
average improvement
p-value
average improvement
P value
30
2.29E-02
3.47-112
1.47E-422
8.16E-06
1.526-02
199E-05
50
2.12E-02
1.35-11
1,202402
L04E-07
1.395-02
2.98E-07
4.02F.19
1.310-02
306E.11
1.59F-02
90
1988-02
1888-22
1.25E:02
1.52--12
1.006-02
2. 2.10--1
100
1.96E 02
5.25E.27
1,27E.02
2.9.3. 13
1.51E.02
9.271.
Group confusion
CUB200
CARS196
InShop
# confusion groups
improvement
p-value
umprovement
p-value
improveneen
p-valoc
10
2.23E-02
9.77E-04
187E400
9.77E-04
-7.83E-04
9.35E-01
1,95E-02
3.SE..0
6.22BB-04
20
2.43E-02
9.54F-07
1833-02
9.54F-07
4.72F.04
7.63F-01
25
2,44E-00 02
2,98E 08
1,72E/02
08
2,08E 03
5.1TE-01
30
2.44E--02
9.31B-10
1.68E-02
9.31B-10
3.27B-03
3.35E-01
and the second term is performing forward pass training data with O. As result, attach each x E Xtrain with an influence function score T(x, Pa), which can be used to rank Xrain and select the most influential training samples.
Algorithm Training Sample Relabelling
Input Training set Xtrain K Output :eeabbeledd set {(x11) 1)
Xharm (x| Pa) 0, Merain S=0
3
for x E Nharm do
m(xmii Yani)) < (INN($(x))
Eee corpl porp-dti x(. (1))) VI Set((mna)
(ynoww 1]] En 2n- expl d(x, [xm0))()
S = SU (0x((x max, P(yner = 1))}
return S
Sample Relabelling Recommendation Based on the empirical influence function, we propose the data relabelling as data cleansing strategy. Given the identified influential harmful training samples Kharm from EIF, Assuming that the noisy samples are the minority in the training dataset, recommend a label E Viuin according to the labels of its neighbors through weighted-KNN KNN algorithm (See Algorithm M. Intuitively, the label supported by more weighted neighbours is recommended to correct harmful sample The neighbours are weighted by their proximity to Note that, we use hard label re-assignment (one-hot label) in Algorithm [ (line 5). We can also revise it to soft label re-assignment by using the probability score P( the calibration label.
4 Experiment
Experimental Settings this study, we use Proxy-NCA++ BI and SoftTriple 23 loss to train models
models with ResNet-50 model architecture three datasets, i.e., CUB200 37 CARS196
18] and InShop [1]] The influential sample location capability is evaluated by DML training experiment. The sample relabelling capability is evaluated by the noisy data relabelling experiment. In the DML training experiment, we evaluate whether retraining the model by upweighting or downweighting the reported influential samples relabelling experiment, we flip 1%. 5%, and 10% of the labels in the above training datasets, and evaluate whether EIF identify and correct those noisy samples. We choose the influence function with modified testing loss function (as in Equation 4) as our baseline. More details of training configuration be referred on website
mitigate the confusion pairs. In the noisy data
DML Training Experiment the experiment, each dataset, we select the top-K K (K=30, 50. 70, 90, 100) most confusing pairs in its testing dataset. For each sclected confusion pair Pk.
5

---

Table 2: Mislabelled Sample Recommendation
Dataset
Mislabelling Ratio
CUB200
CARS196
InShop
Method
ProxyNCA++
75.68%
84.09%
89.74%
1%
SoftTriple
75.00%
79.49%
90.00%
ProxyNCA++
82.49%
86.84%
90.29%
5% SoftTriple
86.14%
90.05%
87.03%
ProxyNCA++
83.28%
91.69%
88.37%
10%.
SoftTriple
89.86%
89.95%
86.27%
Table 3: Average Runtime Statistics (in seconds)
Mislabelled Sample
Dataset
App
Detection
Recommendation
EIF
260.20 29.28
1.85 0.01
CUB200
IF
375.56 68.06
EIF
453.61 28.76
3.74 0.03
CARSI96
IF
588.11 # 23.23
EIF
1306.43 + 27.31
43.24 # 0.11
InShop
IF
2454.11 9.55
evaluate the identified influential training samples (cither helpful or harmful) by comparing :
Ad(pk) d(px: 0') d(pk: 0%)
In Equation 7 Op represents the original network, 0' represents the network actually trained by down- weighting harmful samples and up- ap-weighting helpful samples for one epoch, and d(pu; 0') represents the L2-normalized embedding distance of the confusion pair Pe on O'. We test the hypothesis H1 Adgir > (pk) i.e., the selected influential samples by EIF can de-confuse the confusion pairs in more significant way than IF average. We the paired t-test for the statistical hypothesis testing, with the assumption that the normality assumption can hold when K >= 30 [13.
Furthermore, to evaluate whether EIF can work well on groups of confusion pairs, we select the top-M (M=10, 15, 20, 25, 30) testing classes with the most generalization errors, denoted as 9M = (2,, c2. CM J. We call each em E 9M as a confusion pair group. We evaluate the change in average distance after retraining the influential training samples for each confusion pair group.
Ad(cm)
Σ Cmml PeEcm.
1
d(po; d(pci 0o)
We test the hypothesis H1 AdgIF (cm) > Adjr(cm) over GM- We use non con-parametric hypothesis testing method, i.e. Wilcoxon signed-rank test since M <= 30.
Noisy Data Detection Experiment In the experiment, we evaluate how effective EIF can detect and recommend to relabel the noisy training samples.
Results: Influential Sample Identification Table shows the comparison of the score Ad(p) between EIF and the original influence function (IF). We can see that EIF outperforms IF on the average improvement on "deconfusing" the confusion pair. Compared to IF, EIF identifies samples with which retraining the model can increase larger distance for the confusion pair. Moreover, the improvement is of statistical significance (all p-values are smaller than 0.001 but the group confusion on the InShop dataset). Compared to other datasets, InShop is few-shot dataset with more noises (i.e., 50K over 8K classes), which may require a larger training batch size for EIF to be more effective.
Results: Relabelling Recommendation Figure shows the performance of EIF on detecting 10% noisy data samples on the three datasets. The x-axis represents the ranked training samples regarding

---

Poction of micaneee dra acocres
Fradir mn misanolles oara Desecto
Facton mmmoannoo
Faciion of trining data checked
Faction traiiing data checked
Haction trainin cata chocked
(a) CUB200
(b) CARS196
(c) InShop
Figure 2: The performance of detecting 10% mislabelled samples (see Section
their influence score in descending order (from positive to negative values), the y-axis represents the ratio of mislabelled samples being detected. We plot random choice (in green) and EIF with different No (in blue and orange). Readers refer to Section for the definition of No. We can see that the samples influence score with small magnitude contribute very few noisy samples; in contrast, either reported harmful helpful training samples constitute almost all the noisy samples. Our investigation shows that the harmful and the helpful noisy samples are influential in a different way. Comparing those harmful noisy samples, helpful noisy samples largely ignored by the model. Thus, they have much larger training loss than the other samples, making them sensitive to model perturbation.
Further, Table shows the performance of recommending the relabelling suggestion. Overall, the sampling relabelling algorithm performs well in predicting and re-calibrating the labels. Note that, with the increase in mislabelling ratio, the relabelling algorithm can still preserve its performance.
shows the runtime cost of EIF and IF in the above experiment. We Ng 1 for (see Section 4 recording the runtime cost of EIF. In Table 3 report the std runtime for mislabelled sample experiments. Overall, we can see that EIF can boost the runtime efficiency of IF by ~3.55% average.
Results: Runtime Performance Table
Testing contusion pai
Cuace
Dis
Clace
Org #t
The =.
Cin
ve oF
Figure Case study CUB200 example
Qualitative Result
We qualitatively investigate the top harmful samples identified by EIF v.s. IF
One example is shown in Figure 3 Overall, we have the following two observations (1) EIF identifies top harmful training samples more visually similar to the testing pair (compared those from and (2) distributed in concentrated classes. More examples can be found in Figure IO

---

Table 4: Agreeable and Disagreeable Confusion Pairs
#Confusion
#Mis-similar
Dataset
#Agrecable #Disagreeable
#Agrecable
#Disagrecable
CUB200
15
5
11
9
CARST96
6
14
2
18
InShop
10
10
5
15
(a)
under class
143 with
(confused
(b) under class 140 (confused
(c) under class 100 (con-
(d) under class 143 (con- fused with Figure 4c
Figure 4b
with Figure 4a
fused with Figure 4d
Figure 4: Figure 4a and Figure 4b are reported as a confusion pair but human investigator agree with the mode decision; Figure 4ç and Figure 4d are reported as confusion pair and human investigator disagree with the mode decision (the birds can be distinguished by the color of their heads).
5 Field Study on Popular Datasets
Based on our EIF framework, we further investigate the generalization errors of the statc-of-the-an models make on the popular datasets such as CUB200 (37] CARS196 [ and InShop [191. We investigate (1) what does the generalization errors look like and how human agree with the errors? (2) what are the root causes for the erroneous model decision? In this study, we choose Proxy-NCA++ 31 in this study as its leading performance in metric learning community.
Study Design For each dataset, we investigate two types of generalization errors: confusion pair and mis-similar pair. The mis-similar pair corresponds to the alternative definition in Section i.c., the semantically similar pair with large distance. For each erroneous pair, we manually evaluate and classify them into: (1) agreeable error and (2) disagreeable error. For agreeable errors, humans agree with the model's decision. For disagreeable errors, humans disagree with the model. In this study, we recruit two volunteers (university graduate students majoring in computer science) to independently verify the reported confusion and mis-similar pairs. For the pairs where they disagree with each other, we let them discuss and reach consensus. In addition, we use Algorithm to identify and generate relabelling suggestions for the harmful training samples the erroneous pairs. Based on the recommendationn we further confirm the recommendation and qualitatively analyze the reported problems in training datasets.
Generalization Errors and Their Agrecability number of generalization errors with the model. Overall, human agrees with 5 51.6% (31 out of 60) of the confusion pairs and 30% (18 out of 60) of the mis-similar pairs. We show agreeable and disagreeable confusion pairs in Figure 4 Overall, human investigators agree with many of the website
Table 4 shows that human may share considerable
"erroneous" model decisions. Readers can check mis-similar pairs
Root Cause of Erroneous Decision Table 5 shows that EIF generates relabelling suggestion for 41% (41 out of 100) training classes in CUB200 dataset, 46.9% (46 out of 98) training classes in the CARS196 dataset, and 22.5% (901 out of 3997) training classes in the InShop dataset. Moreover, we further investigate the training classes with more than 10% of their samples recommended to change their labels. We can see that, compared to CUB200 and CARS196 196, the InShop dataset has more confusing training classes. Figure 5 show some relabelled training samples in the InShop dataset.
We further manually sample the training classes with relabelling recommendations, regarding the following criteria:
Centralized Relabelling: The training classes with more than 10% samples are recommended to be relabelled, and the recommended labels lean towards single label.

---

Table 5: Relabelling suggestions on the popular datasets.
#class with relabelling suggestion
Dataset
#class
Total
Centralized Relabelling
Diversilied Relabelling
Individual Relabelling
Others
CUB2000
100
41
2
28
8
CARSI96
98
46
8
26
8
InShop
3997
901
602
201
19
79
(a) class 24
(b) class 652
(c) class 28
(d) class 2961
(e) class 99
(I) class 2216
Figure Samples in the InShop training dataset, which look similar but under different class.
Figure 5a is recommend to relabel to class 652, Figure 5c is recommend to relabel to class 2961,
Figure 5e is recommend to relabel to class 2216,
Diversified Relabelling: The training classes with more than 10% samples are recommended be relabelled, and the recommended labels lean towards diversified labels.
Individual Relabelling: The training classes with less than 5% samples are recommended to be relabelled.
We distinguish centralized and diversified relabelling cases by introducing a threshold Lhy (we use 0.35 this study). Given the entropy of the relabelling-clas distribution class C as He. He < thh, we consider C' as a centralized relabelling class; otherwise, we consider C' as diversified relabelling class. Generally, a dataset, if with centralized relabelling requirements, needs have its relevant training classes merged. In contrast, dataset, if with diversified and individual relabelling requirements, needs clean the samples under the relevant training classes.
We sample 3 training classes from each category on each dataset and report the confirmed relabelled suggestions Table Overall, EIF achieves high recommendation accuracy the training classes with centralized and diversified relabelling in CUB200 and CARSI96; and acceptable accuracy on the training classes with individual relabelling classes in CARS196 and InShop. We provide more details on our website 22)
Summary and Discussion In this study, we conclude that the following problems are universal among the popular DML training and testing datasets:
The testing dataset includes number of arguably confusing samples, thus "erroneous" model decision by the labels of testing samples may not necessary be really erroneous.
Some classes are confusing with each other, i.e., over 10% of the samples have the potential to be merged into other classes.
Many training classes involve outliers that look very different from other samples in the same class, but similar to other classes.
We provide more detailed examples are one of the most important barriers to future work of DML should revolve around dataset cleaning and merging improve the metrics in more significant manner.
website We conclude that those dataset problems further improve new state-of-the-art DML approaches. The
Related Work
Deep Metric Learning Deep metric learning (DML) learns an embedding space such that intra- class samples are located closer than inter-class samples. Loss functions are usually the key for learning such an embedding, which has been evolved from pairwise-based loss such as
and 20 to proxy-bases loss such as 20
BI
2
[4]

---

Table 6: Manually verified relabelling suggestions on the datasets
Manual Evaluation
Type
CUB200
CARST96
InShop
Centralized Relabelling
100%
100%
66.67%
Diversified Relabelling
92.59%
100%
50.00%
Individual Relabelling
66.67%
85.71%
80.00%
While new approaches emerge to outperform the state-of-thed ohe-the-rrrtttt art with marginal improvements [21 e4l 40.. few work has been proposed towards understanding of generalization error in DML, from the dataset perspective. This work makes the first step, and our findings shed light on the potential problems of the datasets. Moreover, EIF technique can further facilitate their fixes.
Model Explanation Despite its great successes in multiple disciplines, the deep learning model has remained to be black-box mystery for decades. There are Iwo types of explanations: feature- level and instance-level. While feature-level explanations would like to interpret the semantics and importance of features, instance-level explanations would like to quantify the individual training sample's contribution to prediction. In 15 the idea of influence function is introduced to measure the change testing loss upon removal of certain training sample. Variations of influence functions are later developed to solve the overestimation for outliers [ the low diversity in high-influence points RPS [39 proposes an alternative view from the Representer Point Theorem: they use the weighted kernels of training points as the influence measure. Since RPS has restrictions on model regularizers, RPS-LJE [BOO] has been later proposed to generalize RPS to models without regularization. However, its definition has subtle differences to the original influence function 15
4. the high computaional cost in Hessian estimation 26 22
Influence function has been applied to various tasks such as VAE 17 GAN 32 data poisoning ()) causal inference [I] data subsampling [
As far as we know, we the first work which designs influence function catering to DML problems.
36 55
data relabelling [
7 Conclusion
In this work, we design an empirical influence function (EIF) to debug and understand the generaliza- tion errors in state-of-the-arr deep metric learning models. Comparing to the traditional influence function, EIF can (1) guide us to locate the influential harmful and helpful training samples and (2) recommend the potential relabelling suggestion for the harmful training samples. Our extensive experiments have proved its effectiveness. With the support of EIF, we further identify the problems of existing datasets for metric learning, which suggests the improvement of the dataset for achieving further world-record performance.
Acknowledgement
This research is supported in part by the Minister of Education, Singapore (T1-251RES1190 T2EP20120-001 MOET3302000004), NUS-NCS Joint Laboratory for Cyber Security, Singapore, the National Research Foundation, Singapore, and Cyber Security Agency of Singapore under its National Cybersecurity Research and Development Programme (Award No. NRF-NCR TAU EAU_2021- 0002) and A *TAAR CISCO Systems (USA) Pte. Ltd and National University of Singapore under its Cisco-NUS Accelerated Digital Economy Corporate Laboratory (Award 121001E0002).
References
[1] A. Alaa and M. Van Der Schaar. Validating causal inference models via influence functions, In International Conference on Machine Learning, pages 191-201. PMLR, 2019.
[2] A. Anonymous. Anonymous webiste for empirical influence function. https //sstess//eee
wttp:///w////mmpaaaaaaaaiaaaeeoe hww.:eiooooeoccocioooooooooooooo Accessed: 2022-01-28.
10

---

[3] E. Barshan, M.-E. Brunet, and G. K. Dziugaite. Relatif: Identifying explanatory training samples via relative influence. In International Conference on Artificial Intelligence and Statistics, pages 1899-1909. PMIR, 2020.
[4] U. Bhatt, I. Chien, M. B. Zafar, and A. Weller. Divine: Diverse influential training points for data visualization and model refinement. arXiv preprini urXiv:2107.05978 2021.
[5] D. Bouchacourt, R. Tomioka, and S. Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[6] M. Boudiaf, J. Rony, I. M. Ziko, E. Granger, M. Pedersoli, P. Piantanida, and I. B. Ayed. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In European Conference on Computer Vision, pages 548-564. Springer, 2020.
[7] W. Chen, X. Chen, J. Zhang, and K. Huang. Beyond triplet loss: a deep quadruplet network for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 403-412, 2017.
[8] Deng, Guo, N. Xue, and S. Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 4690-4699, 2019.
[9] M. Fang, N. Z. Gong, and Liu. Influence function based data poisoning attacks to top-n recommender systems. In Proceedings of The Web Conference 2020, pages 3019-3025, 2020.
[10] G. Gu, B. Ko, and H.-G. Kim. Proxy synthesis: Learning with synthetic classes for deep metric learning, arXiv preprint arXiv: 103.15440 2021.
[11] R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 06), volume 2, pages 1735-1742. IEEE, 2006.
[12] E. Hoffer and N. Ailon. Deep metric learning using triplet network. In International workshop on similarity-buse -assed pattern recognition, pages 84-92. Springer, 2015.
[13] R. Hogg, E. Tanis, and Zimmerman. Probability and statistical inference, volume 993. Macmillan New York, 1977.
[14] S. Kim, D. Kim, M. Cho, and S. Kwak. Proxy anchor loss for deep metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2238-3247, 2020.
[15] P. W. Koh and P. Liang. Understanding black-box predictions via influence functions. In International Conference Machine Learning, pages 1885-1894, PMLR, 2017.
[16] S. Kong, Y. Shen, and L. Huang. Resolving training biases via influence- data relabeling. In International Conference on Learning Representations, 2021.
[17] Z. Kong and K. Chaudhuri. Understanding instance- based interpretability variational auto- encoders. Advances in Neural Information Processing Systems, 34, 2021.
[18] Krause, M. Stark, Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
[19] Z. Liu, P. Luo, S. Qiu, X. Wang, and X. Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2016.
[20] Y. Movshovitz-Attias A. Toshev, T. K. Leung, S. loffe, and S. Singh. No fuss distance metric learning using proxies. In Proceedings of the IEFE International Conference on Computer Vision, pages 360-368, 2017.

---

[21] Y. Patel, G. Tolias, and J. Matas. @ecall k surrogate loss with large batches and similarity mixup. arXiv preprint arXiv:2108. 11179, 2021.
[22] G. Pruthi, F. Liu, S. Kale, and M. Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 13910-9--1993, 2020.
[23] Q. Qian, L. Shang, B. Sun, Hu, H. Li, and R. Jin. Softtriple loss: Deep metric learning without triplet sampling. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6450-6458, 2019.
[24] K. Roth, T. Milbich, S. Sinha, P. Gupta, B. Ommer, and J. P. Cohen. Revisiting training strategies and generalization performance deep metric learning. In International Conference on Machine Learning, pages 8242-8252. PMLR, 2020.
[25] K. Roth, O. Vinyals, and Z. Akata. Integrating language guidance into vision-based deep metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16177-16189, 2022.
[26] A. Schioppa, P. Zablotskaia, D. Vilar, and A. Sokolov. Scaling up influence functions. arXiv preprint arXiv:21 2021.
[27] Schroff, D. Kalenichenko, and Philbin. Facenet: unified embedding for face recogni- tion and clustering. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 815- 823, 2015.
[28] J. Seidenschwarz, I. Elezi, and L. Leal- Taixé. Learning intra-batch connections for deep metric learning. arXiv preprint arXiv:2102.0775: 2021.
[29] K. Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Advances in neural information processing systems, pages 1857-1865, 2016.
[30] Y. Sui, G. Wu, and S. Sanner. Representer point selection via local jacobian expansion for post-hoe classifier explanation of deep neural networks and ensemble models. Advances in Neural Information Processing Systems, 34, 2021.
[31] E. W. Teh, T. DeVries, and G. W. Taylor. Proxynca++: Revisiting and revitalizing proxy neigh- borhood component analysis. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIV 16, pages 448-464. Springer, 2020.
[32] N. Terashita, H. Ohashi, Y. Nonaka, and Kanemaru. Influence estimation for generative adversarial networks. arXiv preprint arXiv:2101.0883 2021.
[33] D. Ting and E. Brochu. Optimal subsampling with influence functions. Advances in neural information processing systems, 31, 2018.
[34] H. Wang, Y. Wang, 7. Zhou, X. Ji, D. Gong. 7hou, 7. Li, and W. Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5265-5274, 2018.
[35] T. Wang. J. Huan, and B. Li. Data dropout: Optimizing training data for convolutional neural networks. In 2018 IEEE 30th International Conference on Tools with Artificial Intelligence (ICTAI), pages 39 46. IEEE, 2018
[36] Z. Wang, H. Zhu, 7. Dong, X. He, and S.-L. Huang. Less is better: Unweighted data subsampling via influence function. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pages 6340-6347, 2020.
[37] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UGe UCCD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.
[38] C.-Y. Wu, R. Manmatha, A. J. Smola, and P. Krahenbuhl. Sampling matters in deep embedding learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 2840-2848, 2017.
12

---

[39] C.-K. Yeh, J. S. Kim, I. E. Yen, and P. Ravikumar. Representer point selection for explaining deep neural networks. arXiv preprint arXiv:1811.09720, 2018.
[40] W. Zhao, Y. Rao, Z. Wang, J. Lu, and J. Zhou. Towards interpretable deep metric learning with structural matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9887-9899 2021.
13

---

8 Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes
(b) Did you describe the limitations of your work? Yes
(c) Did you discuss any potential negative societal impacts of your work? N/A
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? Yes
(b) Did you include complete proofs of all theoretical results? Yes
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? Yes
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? Yes
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? Yes
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? Yes
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? Yes
(b) Did you mention the license of the assets? Yes
(c) Did you include any new assets either in the supplemental material or as a URL? Yes
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? Yes
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? N/A
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? N/A
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? N/A
14


Reviews:
[{"review_id": "n1-Xvezn70t", "reviewer": "Reviewer_5yXY", "summary": "The authors propose an empirical influence function which:\n(a) Quantifies the impact of training samples on generalization errors in deep metric learning and proposes a relabeling framework in the context of DML.\n(b) Proposes a faster version of the traditional influence function which is predominantly slow.", "questions": "Please refer to my questions in the Weaknesses section.\n\n", "limitations": "I feel the paper is a good contribution, though the technical contribution is a bit limited. However, I still feel that the paper should be accepted due to : (A) Application of IF in a new domain such as DML; (b) Good coverage of experiments; (c) Simple strategy to scale up influence functions. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1) The work is very significant practically as there exists very few works concerning understanding influence functions in contexts other than supervised learning (e.g., DML). To the best of my knowledge, this is one of the only works which investigate model interpretability through the lens of the training data in DML.\n\n(2) While influence function is quite useful for downstream tasks such as relabeling, it is very expensive to compute. The work is also significant in the sense that they circumvent the expensive inverse-Hessian-vector product approximation which is a bottleneck in influence computation.\n\n(3) I have reviewed an earlier version of this paper in another conference. This draft is very well organized and easy to follow.  The authors have also updated the list of references which was inadequate in the earlier draft. \n\n(4) The authors cover a wide range of experiments on most of the commonly used datasets in DML. The empirical analysis supports the results well.\n\nWeaknesses:\n(1) While speeding up the influence function via a light weight Newton step approximation, it is not technically novel. However it's still impactful as often it reduces the running time by 10-20 times at the least. \n\n(2)  The authors primary focus on the DML setup and it is fair to not be able to cover different areas of applications. However it would strengthen the paper on how the EIF fares in the supervised setup too. A good idea could be to compare the run-times and also compare the quality of influence values obtained. Considering one of the major contribution is the fast version of IF, it would be beneficial to observe its universality.\n\n(3) Does the selection of backbone architectures play a role in determining the quality of influence values? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_FrlEqtfmu", "reviewer": "Reviewer_cNAe", "summary": "This paper tackles the question of why Deep Metric Learning (DML) approaches are unable to make more than incremental improvements on classical datasets and tasks. Specifically, the trained models fail to recognize similar samples and tend to make the same kind of generalization errors. To better understand the problem, the authors examine the particular training examples that contribute to the generalization errors that the model makes, and discover that many of these examples are labelled incorrectly. This leads the authors to conclude that the major barriers for DML performance is not model design, but rather the significant presence of incorrectly labelled examples in the classical datasets. To address this issue, they propose an empirical influence function (EIF), which identifies these examples and quantifies their contribution to the generalization error. Moreover, they propose a relabelling technique based on the EIF for the problematic examples. ", "questions": "My main questions are around the improvement that EIF provides over IF. \n\nIn the DML training experiment, it would be interesting to know how the statistics describing the improvement of EIF over IF change as the number of confusing pairs considered is varies. Table 1 of the manuscript reports the statistics when the 50 most confusing pairs are considered - but what happens if the number of pairs is increased or decrease? The statistics for each dataset could be plotted as a function of the number of pairs considered. Similarly the number of confusion groups could be varied from the value 10 for which statistics are currently reported. \n\nLines 167-168 states 'Compared to IF, EIF identifies samples with which retraining the model can increase larger distance for the confusion pair.'. To make this achievement more concrete for the reader, it would be helpful for the authors to include some examples, perhaps in the appendix, and to refer to these examples at this point in the text. There are some examples of potential mislabelling given in the appendix, but it is not clear how these examples were identified (I couldn't find information about them in the appendix or the main text). \n\nI understand that removing the need to calculate the Hessian greatly improves the runtime over IF, but it would be helpful if the authors can provide some intuition for why the accuracy also improves for the individual pairs. What is the dependence on the choice of N_theta for the DML training experiment, and why in Figure 2 is N_theta relabelled as M? \n\nIn Figure 2, for the noisy relabelling experiment, please could the authors comment on why the performance of EIF and IF saturate for the middle tranche of training examples, before rapidly increasing as the last few training examples are checked? \n\nThe field study should be carried out for the IF pairs in addition to the EIF pairs, so that Table 4 can be augmented with this data. \n\nOverall there are a few places where the writing could be improved - the meaning is generally more or less clear, but the paper would benefit from a close read though - in some places words or even parameters/variables appear to be missing from sentences, and have to be inferred by the reader from context. ", "limitations": "It would be helpful if the authors could include a discussion of the limitations of their work. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is an interesting paper - identifying examples that are mislabelled in large datasets is clearly an important challenge. Figure 2a-c of the appendix is quite compelling evidence for the claims made about these well known datasets in the introduction, and it might be worth adding these panels to the main text. \n\nThe authors propose a novel empirical influence function, that tackles the significant run time cost of the existing influence function (IF) and an issue that the authors identify with the accuracy of an approximation made for the calculation of group-pair confusion. I am not an expert in the area of influence functions, but to my knowledge this is a novel proposal and its efficacy is supported by the experimental results that the authors present in the paper. \n\nIdentifying mislabelled examples that prevent performance on benchmark tasks from improving is clearly a significant service to the community. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NgyRns6mm9v", "reviewer": "Reviewer_7LQB", "summary": "\nThe authors propose EIF, which is a modification of IF, for DML.  Specifically, it differs from IF in two key ways:\n- It doesn't require computing the Hessian, which leads to faster calculations. \n- It allows for the possibility that the \"retrained\" parameters may be far away from the original parameters, which is useful when a large portion of the dataset is being re-weighted. \n\nThey argue for the efficacy of this method with several sets of results based on datasets with deliberately mislabeled points:\n- Showing that the confusion of the most confused pairs  decreases more (statistically significantly) when the model is retrained on points reweighed by EIF than IF (Table 1)\n- Showing that EIF is able to identify those mislabeled points (Figure 2)\n- Showing that EIF+KNN is able to correctly relabel those mislabeled points (Table 2)\n\nThey show that this has computational benefits compared to IF (Table 3)\n\nAdditionally, they argue for the efficacy of this method on benchmark tasks:\n-  Showing that people often agree with the model's mistakes and, as a result, that the problem is the data labeling (Table 4)\n- Analyzing how often EIF identifies different types of data labeling errors", "questions": "How should readers compare/interpret the results in Table 1 and Figure 2?  \n- Table 1 shows that, for the most confused pairs, EIF is better able reduce their confusion than IF.  However, EIF only really appears to beat IF on CUB200 in Figure 2.  What is causing the disagreement between these results?\n- Relatedly, all of the influence estimators show a high recovery rate when the fraction of the dataset checked is near 0 or near 1.  Were the influences scores sorted by absolute value?  \n\nDoesn't merging similar classes simply make the problem easier?  Fundamentally, it seems that a good model should be able to differentiate similar classes.  So this doesn't seem like a dataset problem that we should try to \"fix\".  ", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:  \n- As far as I know, IF has not been applied to DML before\n- EIF is, technically, a novel extension of IF\n\nQuality:  Overall, the methodology seems solid\n- Having 2 students check the authors conclusions for Table 4 is a little unusual\n- Where is the claim in Line 244 supported?\n\nClarity:  \n- Line 70-71, it's worth clarifying what \"L\" is here because, as written, it sounds a lot like you are going to be doing normal supervised learning. \n- Equations (1) and (2) are somewhat difficult to parse.  My understanding is that the objective is to solve the 2nd line of these equations and that the first line is like a constraint.  \n- Equation (2):  Should \"l:X->Y\" be \"R:X->Y\"?\n- Line 113:  what are \"delta d\" and \"delta L\"?  I'm assuming that they are the difference in these metrics for the new and original parameters\n- Relatedly, is \"delta L\" assumed to be negative?  Otherwise, I'm not seeing the intuition for the interpretation of its sign in Lines 114-115.\n- Line 156:  how is the distance normalized?\n- Clarifying what it means for EIF+KNN to \"recommend a new label\" would be helpful (especially for understanding table 2).  My guess is that it is \"EIF says a point is harmful and KNN assigns it a label different from the provided one\" but I am not certain.  \n\nSignificance: \n- This seems like a potentially important problem\n- Beyond the the computational benefits, it isn't overwhelming clear the EIF is more effective than IF (see Questions)\n- The claims about problems facing DML datasets don't seem especially insightful (some classes have similar images, classes are sometimes similar, classes often have outliers)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept