Interactive Label Cleaning with Example-based Explanations
Authors: Stefano Teso
Andrea Bontempelli
Fausto Giunchiglia
Andrea Passerini
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: learning under noise
interactive learning
example-based explanation
influence functions
fisher information matrix

Abstract: We tackle sequential learning under label noise in applications where a human supervisor can be queried to relabel suspicious examples. Existing approaches are flawed, in that they only relabel incoming examples that look "suspicious" to the model. As a consequence, those mislabeled examples that elude (or don't undergo) this cleaning step end up tainting the training data and the model with no further chance of being cleaned. We propose CINCER, a novel approach that cleans both new and past data by identifying \emph{pairs of mutually incompatible examples}. Whenever it detects a suspicious example, CINCER identifies a counter-example in the training set that - according to the model - is maximally incompatible with the suspicious example, and asks the annotator to relabel either or both examples, resolving this possible inconsistency. The counter-examples are chosen to be maximally incompatible, so to serve as \emph{explanations} of the model's suspicion, and highly influential, so to convey as much information as possible if relabeled. CINCER achieves this by leveraging an efficient and robust approximation of influence functions based on the Fisher information matrix (FIM). Our extensive empirical evaluation shows that clarifying the reasons behind the model's suspicions by cleaning the counter-examples helps in acquiring substantially better data and models, especially when paired with our FIM approximation.

Interactive Label Cleaning with Example-based Explanations
Stefano Teso
Andrea Bontempelli University of Trento Trento, Italy
University of Trento Trento, Italy stefano teso@unitn.it
andrea. Drrrea..ontempeelle ienliiiiinntniim it
Fausto Giunchiglia
Andrea Passerini
University Trento
University of Trento
Trento, Italy
Trento, Italy
fausto., ceasto.giunchii haat:oogguunchgglaummitta
cadrea.passeriniii.. ientriiiii@unninnm .it
Abstract
We tackle sequential learning under label noise in applications where human supervisor can be queried to relabel suspicious examples. Existing approaches are flawed, in that they only relabel incoming examples that look "suspicious" to the model. As consequence, those mislabeled examples that elude (or don't 't undergo) this cleaning step end up tainting the training data and the model with no further chance of being cleaned. We propose CINCER, a novel approach that cleans both and past data by identifying pairs of mutually incompatible examples. Whenever it detects suspicious example, CINCER identifies a counter-exampl in the training chat-according to the model-is maximally incompatible with the suspicious example, and asks the annotator to relabel either or both examples, resolving this possible inconsistency. The counter-examples maximally incompatible, so to serve as explanations of the model's suspicion, and highly influential, so convey as much information as possible if relabeled. CINCER achieves this by leveraging an efficient and robust approximation of influence functions based on the Fisher information matrix (FIM). Our extensive empirical evaluation shows that clarifying the reasons behind the model's ' suspicions by cleaning the counter-examples helps in acquiring substantially better data and models, especially when paired with our FIM approximation.
chosen to be
Introduction
Label noise is a major issue in machine learning as it can lead to compromised predictive performance and unreliable models 112] 2. We focus sequential learning settings in which human supervisor, usually a domain expert, can be asked to double-check and relabel any potentially mislabeled example. Applications include crowd-sourced machine learning and citizen science, where trained researchers can be asked to clean the labels provided by crowd-workers 3 4. and interactive personal assistants [5]. where the user self-reports the initial annotations (e.g., about activities being performed) and unreliability is due to memory bias (6), unwillingness to report (7), or conditioning [8].
This problem is often tackled by monitoring for incoming examples that are likely to be mislabeled,
aka suspicious examples, and ask the supervisor to provide clean (or at least better) annotations for them. Existing approaches, however, focus solely cleaning the incoming examples (9 4 10 5 This means that noisy examples that did not undergo the cleaning step (e.g., those in the initial
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Tue lebrli "4" Arnotated *g"
Tue lebel "2" Annotated as Predicted as *7
Thue label 14*
Thue abel nqr
Thue labe! "4" Annotated as
Thue label "2"
Thue abel *3*
Tue label "O"
Predicted as 14"
Annotatied as 14"
Annotated as 14"
Annotated as
Annotated as
Annitated as "?"
4
4
4
2
7: 1-MN
O
CINGER
1-NN
CINCER
Figure 1: Suspicious example and counter-examples selected using (from left to right) CINCER, 1-NN and influence functions (IF), on noisy MNIST. Left: the suspicious example mislabeled, the machine's suspicion is supported by clean counter-example. Right: the suspicious example is not mislabeled, the machine is wrongly suspicious because the counter-cxample mislabeled. CINCER's counter-xxamle is contrastive and influential; 1-NN's is not influential and IF's is not pertinent, see desiderata DI-D3 below.
bootstrap data set) or that managed to clude it are left untouched. This degrades the quality of the model and prevents it from spotting future mislabeled examples that fall in regions affected by noise.
We introduce CINCER (Contrastive and InflueNt CounterExample stRategy), a new explainable interactive label cleaning algorithm that lets the annotator observe and fix the reasons behind the model's 's suspicions. For every suspicious example that finds, CINCER identifies a counter-example, Le., training example that maximally supports the machine's suspicion. The idea is that the example/counter-exampll pair captures a potential inconsistency in the data-as viewed from the model's perspective- which is resolved by invoking the supervisor. More specifically, CINCER asks the user to relabel the example, the counter-example, or both, thus improving the quality of, and promoting consistency between, the data and the model. Two hypothetical rounds interaction on noisy version of MNIST are illustrated in Figure
CINCER relies a principled definition of counter-cxamples derived from few explicit, intuitive desiderata, using influence functions [1 12 The resulting counter-example selection problem is solved using simple and efficient approximation based on the Fisher information matrix [13] that consistently outperforms more complex alternatives in our experiments.
Summarizing, we: 1) Introduce CINCER, an explanatory interactive label cleaning strategy that leverages example-based explanations identify inconsistencies the data-as as per- ceived by the model- and enable the annotator to fix them. 2) Show how to select counter-examples that at the same time explain why the model suspicious and that are highly informative using (an efficient approximation of) influence functions. 3) Present an extensive empirical evaluation that showcases the ability of CINCER building cleaner data sets and better models.
Contributions:
Background
We concerned with sequential learning under label noise. In this setting, the machine receives a sequence of examples & := (x,, (.), for t 1,2. ...., where X, E IR' an instance and ÿr E [c] is corresponding label, with [c] : (1....,ch, c}. The label Út is unreliable and might differ from the ground- label y"- The key feature of our setting is that a human supervisor can asked to double-check and relabel any example. The goal is acquire clean dataset and a high-quality predictor while asking few relabeling queries, so to keep the cost of interaction under control.
The state-of-the -hth-ar-t for this setting is skeptical learning (SKL) 1O 5. SKI. is designed primarily for smart personal assistants that must learn from unreliable users. SKL follows a standard sequential learning loop: in each iteration, the machine receives example and updates the model accordingly. However, for each example that it receives, the machine compares (an estimate of) the quality of the annotation with that of its own prediction, and if the prediction looks more reliable than the annotation by some factor, SKL asks the user to double-check his/her example. The details depend on the implementation: in 110 label quality is estimated using the empirical accuracy for the classifier and the empirical probability of contradiction for the annotator, while in [5] the machine measures the margin between the user's 's and machine's labels. Our approach follows the latter strategy.
Another very related approach is learning from weak annotators (LWA) 9 which focuses on querying domain experts rather than end-users. The most recent approach jointly learns

---

prediction pipeline composed of classifier and a noisy channel, which allows it to estimate the noise rate directly. Moreover, the approach in [4] identifies suspicious examples that have a large impact on the learned model. A theoretical foundation for LWA is given in [ I.AA is however designed for pool-based scenarios, where the training set is given rather than obtained sequentially. For this reason, in the remainder of the paper, we will chiefly build on and compare to SKI..
Limitations of existing approaches. A major downside of SKL is that it focuses on cleaning the incoming examples only. This means that if a mislabeled example manages to elude the cleaning step and gets added to the training set. is bound to stay there forever. This situation is actually quite common during the first stage of skeptical learning, in which the model is highly uncertain and trusts the incoming examples-eve even if they are mislabeled. The same issue occurs if the initial training set used to bootstrap the classifier contains mislabeled examples. As shown by our experiments, the accumulation of noisy data in the training set may have a detrimental effect on the model's performance (cf. Figure 2). In addition, it can also affect the model's ability to identify suspicious examples: a noisy data point can fool the classifier into trusting incoming mislabeled examples that fall close to it, further aggravating the situation.
3 Explainable Interactive Label Cleaning with CINCER
We consider a very general class of probabilistic classifiers f Rd + [c] of the form f(; 0) ==
arlmaxyele] Ply x; 0), where the conditional distribution P(Y X; 0) has been fit on training data by minimizing the cross-entropy loss {(xx y). 6) - Li implementation, we also assume to be a neural network with a softmax activation at the top layer, trained using some variant of SGD and possibly early stopping.
y} log P(i x, 8). In our
3.1 The CINCER Algorithm
The pseudo-code of CINCER is listed in Algorithm 1 At the beginning of iteration f, the machine has acquired training set Dz-1 = {=1,..., 2 and trained a model with parameters f on it. At this point, the machine receives new, possibly mislabeled example & (line() and has to decide whether to trust it.
Following skeptical learning [3]. CINCER does so by computing the margin ul 0--1). 1). i.e., the difference in conditional probability between the model's prediction d == argmax, P(y x2, f) and the annotation Tt. More formally:
(12,, 01-1 P(ù |x, 02-1) P(ue xx, 01-1)
The margin estimates the incompatibilityy between the model and the example: the larger the margin, the more suspicious the example. The example & is deemed compatible if the margin is below a given threshold 7 and suspicious otherwise (line 4); possible choices for 7 are discussed in Section 3.5
If & is compatible, it added to the data set as-is (line 5 Otherwise, CINCER computes a counter- example z E D that maximally supports the machine's s suspicion. The intuition is that the pair (ž,, 2x.) captures a potential inconsistency in the data. For instance, the counter-example might be correctly labeled example that is close or similar to ε, but has different label, distant noisy outlier that fools the predictor into assigning low probability to Gt- How to choose an effective counter-example is major focus of this paper and discussed in detail in Section 3.2 and following.
Next, CINCER asks the annotator to double-check the pair (&e, 2x) and relabel the suspicious example, the counter-example or both, thus resolving the potential inconsistency. The data set and model are then updated accordingly (line D and the loop repeats.
3.2 Counter-example Selection
Counter-exampe are meant to illustrate why particular example غ is deemed suspicious by the machine in a way that makes it easy to elicit useful corrective feedback from the supervisor. We posit that a good counter-example
should be:
DI. Contrastive: 2 should explain why &, is considered suspicious by the model, thus highlight- ing potential inconsistency in the data.
3

---

Algorithm Pseudo-code of CINCER. Inputs: initial (noisy) training set Do; threshold т.
1: fit 8g Do
2: for = 11,2... .... do
3:
receive new example & (xe,
4: if (( 0,-1) ) < then
5:
D. + D D (z)
ān is compatible
6: else
7:
find counterexample ze using Eq. 12
D 2t is suspicious
8:
present E,, 24 to the user, receive possibly cleaned labels s/. v/. D, - (De-1 1 {2+}) U {(x, v(),, (xk, }
9:
10:
fit Og on Dr
D2. Inftuential: if 2 is mislabeled, correcting it should improve the model as much possible, so to maximize the information gained by interacting with the annotator.
In the following, show how, for models learned by minimizing the cross-entropy loss, identify counter-examples that satisfy both desiderata.
What is a contrastive counter-example? We start by tackling the first desideratum. Let 0,-1 be the parameters of the current model. Intuitively, 2 E D;-1 is a contrastive counter-cxample for suspicious example žn if removing it from the data set and retraining leads to model with parameters 0, that assigns higher probability to the suspicious label Ør. The most contrastive counter-example then the one that maximally affects the change probability:
argmax { ((e xx; 0, - P 1x; (x;0)
While intuitively appealing, optimizing Eq. directly is computationally challenging as it involves retraining the model |D,-1 times. This is impractical for realistically sized models and data sets, especially in our interactive scenario where a counter-example must be computed in each iteration.
Influence functions. We address this issue by leveraging influence functions (IFs), computational device that can be used to estimate the impact specific training examples without retraining [2] Let Og be the empirical risk minimizer on D. and 0z(2, e) be the minimizer obtained after adding an example z with weight to Dr, namely:
8, argming Eim ((2x,0)
(,(2,c)
argming J (Ei=1 (rrgmin=aggming ((2.
0)) + cl(2, 0)
(3)
Taking first-order Taylor expansion, the difference between O, 0z(z, 0) and 0, e) can be written as (z(2,e) (=, = e. side is the so-called intluence function, denoted (z). It follows that the effect on O, of adding (resp. removing) an example z to D, can approximated by multiplying the IF by = 1/t (resp. 1/tt, Crucially, if the loss is strongly convex and twice differentiable, the IF can be written as:
The derivative appearing on the right hand
Io, (2) (0.)
- Vol(z,0.)
(4)
where the curvature matrix H(0.) + Ei=n VZl(ZK. 0z) is positive definite and invertible. IFs were shown to capture meaningful information even for neural networks and other non-convex models [12
Identifying contrastive counter-example with influence functions. To see the link between con- trastive counter-exanee exammles and influence functions, notice that the second term of Eq. iis independent of Zg. while the first term can be conveniently approximated with IFs by applying the chain rule:
1
d
d de
P(je x1; 0 (1k, <))
--
VoP(i X 0 10-1)
t-1
de
(5)
t-1 VoP(ii xx; 0--1) TT Io. (2k))
(6)
The constant can dropped during the optimization. This shows that Eq, 2 is equivalent to:
argmax+e(r-1) VoP(i, xx 00-1) (x -Vel=z))-Voolz...l

---

Eq. Eq.77can can be solved efficiently by combining two strategies [[]] i) Caching the inverse Hessian vectoor product (HVP) P( x,; (,;
so that evaluating the objective on each 24 becomes simple dot product, and ii) Solving the inverse HVP with an efficient stochastic estimator like LISSA [4]. This gives us an algorithm for computing contrastive counter-examples.
(4::0-))"(((()-)>>>>>> (()-)))
Contrastive counter-examples are highly influential. Can this algorithm be used for identifying influential counler-examples? It turns out that, long as the model is obtained by optimizing the cross-entropy loss, the answer is affirmative. Indeed, note that if {(z, 0) - log Ply x; 0), then:
Vo PoP X2; 0c-1)
V 8 (ūt x 11) p(u Xx; 111)
(8)
P(u x1; 011)
P(i X1; 11-1) Vo log P(ū x,; 01; -P(ie
(9)
Hence, Eq. can be rewritten as:
P( Xri Xr; Vol H
(10)
x - -= ((0:--)--Voll 02-1)
(11)
It follows that, under the above assumptions and as long as the model satisfies P(ee ( 02-10 0, Eq. is equivalent to:
argmmxxc[[-1] -
Vol(2,,
(12)
This equation recovers exactly the definition of influential examples given in 12 Eq. 2] and shows that, for the large family of classifiers trained by cross-entropy highly influential exampes- are highly contrastive and vice versa, so that no change the selection algorithm is necessary.
3.3 Counter-example Selection with the Fisher information matrix
Unfortunately, we found the computation of IFs to be unreliable in practice, ef. [3]. This leads to unstable ranking of candidates and reflects on the quality of the counter-examples, in Figure] The issue is that, for non-convex classifiers trained using gradient-based methods (and possibly early stopping), 0-11 is seldom close to local minimum, rendering the Hessian non-positive definite. In our setting, the situation is further complicated by the presence of noise, which dramatically skews the curvature of the empirical risk. Remedies like fine-tuning the model with L-BFGS [12 16]. preconditioning and weight decay [15] proved unsatisfactory in our experiments.
We take different approach. The idea is to replace the Hessian by the Fisher information matrix (FIM). The FIM F F6) of a discriminative model P(Y X, 0) and training set D,-1 1 is [1 18):
F(0) = - E Ey-P(Y|xx |x. .8) [ log Ply XR, V) log P(yl Xk, 0)
(13)
can be shown that, if the model approximates the data distribution, the FIM approximates the Hessian, cf. [19 20). Even when this assumption does not hold, as is likely in our noisy setting, the FIM still captures much the curvature information encoded into the Hessian 17 Under this approximation, Eq. 12 can be rewritten as:
argmax 0, (Vel-z,))Voo((z,,, 0.-1)
(14)
The advantage of this formulation is twofold. First of all, this optimization problem also admits caching the inverse FIM-vector product (FVP), which makes it viable for interactive usage. Second, and most importantly, the FIM is positive semi-definite by construction, making the computation of [4 much more stable.
The remaining step is how to compute the inverse FVP. Naïve storage and inversion of the FIM, which |d) x [0] in size, is impractical for typical models, so the FIM usually replaced with a simpler matrix. Three common options are the identity matrix, the diagonal of the FIM, and a block-diagone approximation where interactions between parameters of different layers are set to zero [17]. Our best results were obtained by restricting the FIM to the top layer of the network. We refer to this approximation as "Top Fisher". While more advanced approximations like K-FAC 17 exist, the Top Fisher proved surprisingly effective in experiments.
5

---

3.4 Selecting Pertinent Counter-examples
So far, we have discussed how to select contrastive and influential counter- exaamless. Now we discuss how to make the couter-examples easier to interpret for the annotator. To this end, we introduce the additional desideratum that counter-examples should be:
D3 Pertinent: it should be clear to the user why 2 is counter-example for a.
We integrate D3 into CINCER by restricting the choice of possible counter-examples A simple strategy, which do employ in all of our examples and experiments, is restrict the search to counter-examples whose label in the training set is the same as the prediction for the suspicious example, i.e., y/ Dt- This way, the annotator can interpret the counter-example being in support of the machine's suspicion. In other words, if the counter-example is labeled correctly, then the machine's ss suspicion is likely right and the incoming example needs cleaning. Otherwise, if the machine is wrong and the suspicious example is not mislabeled, it is likely the counter-example which backs the machine's suspicions that needs cleaning.
Finally, one drawback of -e-selected counter-examples is that they may be perceptually different from the suspicious example. For instance, outliers are often highly influential as they fool the machine into mispredicting many examples, yet they have little in common with those examples (20). This make it difficult for the are meant to explain. This is not necessarily an issue: first, a motivated supervisor is likely to correct mislabeled exxmpless regardless of whether they resemble the suspicious example; second, highly influential outliers are often identified (and corrected if needed) in the first iterations of CINCER (indeed, we did not observe significant amount of repetitions among suggested counter-examples in our experiments). Still, CINCER can be readily adapted to acquire more perceptually similar counter-examples. One option is to replace IFs with relative IFs [20], which trade-off off influence with locality. Alas, the resulting optimization problem does not support efficient caching of the inverse HVP. A better alternative is to restrict the search counter-examples 2g that are similar enough & in terms of some given perceptual distance ||-]| [2 by filtering the candidates using fast nearest neighbor techniques in perceptual space. This is analogous FastlF (22), except that the motivation is to encourage perceptual similarity rather than purely efficiency, although the latter is a nice bonus.
understand their relationship with the suspicious examples they
3.5 Advantages and Limitations
The main benefit of CINCER that, by asking human annotator to correct potential inconsistencies in the data, it acquires substantially better supervision and, in turn, better predictors. In doing so, CINCER also encourages consistency between the data and the model. Another benefit that, by explaining the reasons behind the model's skepticism, CINCER allows the supervisor to spot bugs and justifiably build or, perhaps more importantly, reject 23 24
trust into the prediction pipeline.
CINCER only requires to set a single parameter, the margin threshold т, which determines how frequently the supervisor invoked. The optimal value is highly application-speiffe but generally speaking, it depends on the ratio between the cost of relabeling query and the cost of noise. If the annotator is willing to interact (for instance, because paid to do so) then the threshold can be quite generous.
4 Experiments
Do counter-examples contribute to cleaning the data? Q2: Which influence-based selection strategy identifies the most mislabeled
We empirically address the following research questions: Q1:
counter-cxamples? Q3: What contributes to the effectiveness of the best counter-example selection strategy?
We implemented CINCER using Python and Tensorflow 25 on top of three classifiers and compared different counter-example selection strategies five data sets. The IF code is adapted from [26]. All experiments were run on a 12-core machine with 16 GiB of RAM and no GPU. The code for all experiments is available at: http::///gittuu..cccaabboeeeeee
Data sets. We used diverse set of classification data sets: Adult (77 data set of 48,800 persons, each described by 15 attributes; the goal is to discriminate customers with an income above/below

---

Drop
CINCER (Top Fisher)
10
20
14-
20
300
22
42
0.25
0.75
0.40
0.75
0.28
6.7
200
Barral
Figure 2: CINCER using Top Fisher vs. drop CE and no CE. Left to right: results for FC on adult, breast and 20NG, CNN on MNIST, Top row: # of cleaned examples. Bottom row: F1 score.
$50K. Breast 27 data of 569 patients described by 30 real-valued features. The goal is to discriminate between benign and malignant breast cancer cases. 20NG [27]: data set of newsgroup posts categorized in twenty topics. The documents were embedded using a pre-trained Sentence- BERT model [28] and compressed to 100 features using PCA. MNIST (2]]: handwritten digit recognition data set from black-and-white, 28 28 images with pixel values normalized in the [o, 1] range. The data set consists of 60K training and 10K test examples. Fashion [30]] fashion article classification dataset with the same structure as MNIST. For adult and breast, random 80 20 training-test split is used while for MNIST, fashion and 20 NG the split provided with the data set used. The labels of 20% of training examples were corrupted at random. The experiments were repeated five times, each time changing the seed used for corrupting the data. Performance was measured in terms of F1 score on the (uncorrupted) test set. Error bars in the plots indicate the standard error. All competitors received exactly the same examples in exactly the same order.
Models. We applied CINCER to three models: LR, a logistic regression classifier; FC, feed-forward neural network with two fully connected hidden layers with ReLU activations; and CNN, feed- forward neural network with two convolutional layers and two fully connected layers. For all models, the hidden layers have ReLU activations and 20% dropout while the top layer has softmax activation. LR was applied MNIST, FC to both the tabular data sets (namely: adult, breast, german, and 20NG) and image data sets (MNIST and fashion), and CNN to the image data sets only. Upon receiving new example, the classifier is retrained from scratch for 100 epochs using Adam [31] with default parameters, with early stopping when the accuracy and CNN, and 70% for LR. This helps substantially to stabilize the quality of the model and speeds up the evaluation. Before each run, the models are trained on a bootstrap training set (containing 20% mislabeled examples) of 500 examples for 20NG and 100 for all the other data sets. The margin threshold is set
the training set reaches 90% for FC
0.2. Due to space constraints, and three tabular data, and we focus on FC and CNN. The other results are consistent with what is reported below; these plots are reported in the Supplementary Material.
report the results
image data set
4.1 Q1: Counter-examples improve the quality of the data
To evaluate the impact of cleaning the counter-examples we compare CINCER combined with the Top fisher approximation of the FIM, which works best in practice, against two alternatives, namely: No CE: an implementation of skeptical learning [5] that asks the user to relabel any incoming suspicious examples identified by the margin and presents no counter-examples. Drop CE: a variation of CINCER that identifies counter-exampee considers the incoming example correctly labeled. The results are reported in Figure 2 The plots show that CINCER cleans by far the most examples on all data sets, between 33% and 80% more than the alternatives (top row in Figure 2 This translates into better predictive performance as measured by Fi score (bottom row). Notice also that CINCER consistently outperforms the drop CE strategy in terms of F, score, suggesting that relabeling the counter-examples provides important information for improving the model. These results validate our choice of identifying and relabeling counter-examples for interactive label cleaning compared to focusing on suspicious incoming examples only, and allow us to answer Q1 in the affirmative.
using Top Fisher but drops them from the data set if the user
7

---

precision
precisseen
precisie precisson
presiiiin seciiiion
Ex
Practical Fisher
Top Fisher
Full Fisher
Pr@5
Pr@10
Pr@S
Pr@10
Pr@
Pr@10
Pr@S
Pr@10
Figure 3: Counter-example @r@5 and Pr@ 10. Standard error information is reported. Left to right:
results for FC on adult, breast and 20NG, and CNN on MNIST.
NN
Practical Fisher
Top Fisher
10
20-
10
1.-
20
0.75
0.40
0.28
0.72
0.55
10 10.00.000 onG
Brrotore
I0cracions
Serations
Figure 4: Top Fisher vs. practical Fisher vs. NN. Left to right: results for FC on adult, breast and 20NG, CNN on MNIST. Top row: # of cleaned examples. Bottom row: F1 score.
4.2
Q2: Fisher Information-bases strategies identify the most mislabeled counter-examples
Next, compare the ability of alternative approximations of IFs to discover mislabeled counter- examples. To this end, we trained a model on a noisy bootstrap data set, selected 100 examples from the remainder of the training set, and measured how many truly mislabeled counter-examples are selected by alternative strategies. In particular, we computed influence using the IF LISSA estimator of (12), the actual FIM (denoted "full Fisher" and reported for the simpler models only for computational reasons) and its approximations using the identity matrix (aka "practical Fisher" (32). and Top Fisher. We computed the precision at k for {5, 10), i.e, the fraction of mislabeled counter- examples within five or ten highest-scoring counter-example retrieved by the various alternatives, averaged over 100 iterations for five runs. The results Figure 3 show that, in general, FIM-based strategies outperform the LISSA estimator, with Full Fisher pertorming best and Top Fisher close second. Since the full FIM is highly impractical store and invert, this confirms our choice of Top Fisher as the best practical strategy.
4.3
Q3: Both influence and curvature contribute to the effectiveness of Top Fisher
Finally, we evaluate the impact of selecting counter-examples using Top Fisher on the model's performance, in terms of use of influence, by comparing (NN), and modelling of the curvature, by comparing it to the Practical Fisher. NN simply selects the counter-example that is closest the suspicious example. The results can be viewed in Figure 4 Top Fisher is clearly the best strategy, both in terms of number of cleaned examples and F1 score. NN is always worse than Top Fisher terms of Fi, even in the case of adult (first column) when cleans the same number of examples, confirming the importance of influence in selecting impactful counter-examples. Practical Fisher clearly underperforms compared with Top Fisher, and shows the importance of having the curvature matrix. For each data set, all methods make a similar number of queries: 58 for 20NG, 21 for breast, 31 for adult and 37 for MNIST. general, CINCER detects around 75% of the mislabeled examples (compared to 50% of the other methods) and only about 5% of its queries do not involve a corrupted example or counter-exanne examee. The complete values are reported in the Supplementary Material. As a final remark, we note that CINCER cleans more suspicious examples than counter-examples (in roughly suspicious examples vs. counter-examples reported in the Supplementary Material. Comparing this the curve for Drop CE shows that proper data cleaning improves the ability of the model of being suspicious for the right reasons, expected.
intuitive nearest neighbor alternative
ratio), as shown by the number of cleaned

---

5
Related Work
Learning under noise. Typical strategies to learning from noisy labels include discarding or down- weighting suspicious examples and employing models robust to noise [33 34 21. often requiring non-trivial noise ratio estimation step [35]. These approaches make no attempt to recover the ground- truth label and are not ideal in interactive learning settings characterized by high noise rate/cost and small data sets. Most works on interactive learning under noise are designed for trowd-sourcing applications in which items are labelled by different annotators of varying quality and the goal is to aggregate weak annotations into a high- nugh-quality consensus label [3]. Our work is strongly related to approaches to interactive learning under label noise like skeptical learning TO 5 and learning
from weak annotators [9 4. These approaches completely ignore the issue of noise in the training set, which can be quite detrimental, shown by our experiments. Moreover, they are completely black-box and do not attempt to explain to the supervisor why examples are considered suspicious by the machine, making it hard for him/her to establish or reject trust in the data and the model.
Inftuence functions and Fisher information. It is well known that mislabeled examples tend to exert a larger influence the model 336 12 37 20 and indeed IFs may be valid alternative to the
margin for identifying suspicious examples. Building on the seminal work of Koh and Liang [12], we instead leverage IFs to define and compute contrastive counter-example machine is suspicious. The difference is that noisy training examples influence the model as a whole, whereas contrastive counter-examples influence a specific suspicious example. To the best of our knowledge, this application of IFs is entirely novel. Notice also that empirical evidence that IFs recover noisy examples is restricted to offline learning [12 37). Our experiments extend this to less forgiving interactive setting in which only one counter-example is selected per iteration and the model trained on the whole training set. The idea of exploiting the FIM to approximate the Hessian has ample support in the natural gradient descent literature [17] [8]. The FIM has been used
that explain why the
for computing example-bur based explanations by Khanna et al. (37] However, their approach is quite different from ours. CINCER is equivalent to maximizing the Fisher kernel [32] between the suspicious example and the counter-example (Eq. I4) for the purpose of explaining the model's margin, and this formulation is explicitly derived from two simple desiderata. In contrast, Khanna et al. maximize a function of the Fisher kernel (namely, the squared Fisher kernel between 24 and & divided by the norm of 25 in the RKHS). This optimization problem is not equivalent to Eq. 14 and does not admit efficient computation by caching the inverse FIM-vector product.
Other works. CINCER draws inspiration from explanatory active learning, which integrates local [24] 38 39 40
global 4T) explanations into interactive learning and allows the annotator supply corrective feedback on the model's explanations. These approaches differ from CINCER in that they neither consider the issue of noise nor perform label cleaning, and indeed they explain the model's predictions rather than the model's suspicion. Another notable difference is that they rely on attribution-baal based explanations, whereas the backbone of CINCER is example-base -aaee explanations, which enable users to reason about labels in terms of concrete (training) examples [2 4 Following these works, saliency maps which provide complementary information about relevant attributes could potentially be integrated into CINCER to provide more fine-grained explanations and control.
6 Conclusion
We introduced CINCER, an approach for handling label noise in sequential learning that asks human supervisor to relabel any incoming suspicious examples. Compared to previous approaches, CINCER identifies the reasons behind the model's skepticism and asks the supervisor to double-check them too, This is done by computing a training example that maximally supports the machine's 's suspicions. This enables the user to correct both incoming and old examples, cleaning inconsistencies in the data that confuse the model. Our experiments shows that, by removing inconsistencies in the data, CINCER enables acquiring better data and models than less informed alternatives.
Our work can be improved in several ways. CINCER can be straightforwardly extended to online active and skeptical learning, in which the label of incoming instances is acquired on the fly [4 10.. CINCER can also be adapted to correcting multiple counter-examples as well as the reasons behind mislabeled counter-examples using "multi-round" label cleaning and group-wise measures of influence 45 46 47). This more refined strategy is especially promising for dealing with systematic noise, in which counter-examples are likely affected by entire groups of mislabeled examples.

---

Potential negative impact. Like most interactive approaches, there is a risk that CINCER annoys the user by asking an excessive number of questions. This is currently mitigated by querying the user only when the model is confident enough in its own predictions (through the margin-based strategy) and by selecting influential counter-examples that have high chance to improve the model upon relabeling, thus reducing the future chance of pointless queries. Moreover, the margin threshold allows to modulate the amount of interaction based on the user's commitment. Another potential issue is that CINCER could give malicious annotators fine-graines control over the training data, possibly leading to poisoning attacks. This is however not an issue for our target applications, like interactive assistants, in which the user benefits from interacting with high-quality predictor and is therefore motivated to provide non-adversarial labels.
Acknowledgments and Disclosure of Funding
This research has received funding from the European Union's 's Horizon 2020 FET Proactive project "WeNet The Internet of us", grant agreement No. 823783, and from the "DELPhi DiscovEring Life Patterns" project funded by the MIUR Progetti di Ricerca di Rilevante Interesse Nazionale (PRIN) 2017 - DD n. 1062 del 31.05.2019 The research of ST and AP was partially supported by TAILOR, project funded by EU Horizon 2020 research and innovation programme under GA No 952215.
References
[1] Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: survey. IEEE transactions on neural networks and learming systems, 25(5))845-88) 869, 2014.
[2] Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. arXiv preprint arXiv:2007.08199 2020.
[3] Jing Zhang, Xindong Wu, and Victor S Sheng. Learning from crowdsourced labeled data: survey, Artificial Intelligence Review, 46(4) 46(4):543-576, 2016.
[4] Jan Kremer, Fei Sha, and Christian Igel. Robust active label correction. In International conference on artificial intelligence and statistics, pages 308-316. PMLR, 2018
[5] Andrea Bontempelli, Stefano Teso, Fausto Giunchiglia, and Andrea Passerini. Learning in the Wild with Incremental Skeptical Gaussian Processes. In Proceedings of the 29th International Joint Conference on Artificial Intelligence, 2020.
[6] Roger Tourangeau, Lance Rips, and Kenneth Rasinski. The psychology of survey response. Cambridge University Press, 2000.
[7] Louise Corti. Using diaries in social research. Social research update, 2(2), 1993.
[8] Brady T West and Jennifer Sinibaldi. The quality of paradata: A literature review. Improving Surveys with Paradata, pages 339-359, 2013.
[9] Ruth Umner, Shai Ben-David, and Ohad Shamir. Learning from weak teachers. In Artificial intelligence and statistics, pages 1252-1260. PMLR, 2012.
[10] Mattia Zeni, Wanyi Zhang, Enrico Bignotti, Andrea Passerini, and Fausto Giunchiglia. Fixing mislabeling by human annotators leveraging conflict resolution and prior knowledge. Pro- ceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(1):32, 2019.
[11] R Dennis Cook and Sanford Weisberg. Residuals and influence in regression. New York: Chapman and Hall, 1982.
[12] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference Machine Learning, pages 1894. JMLR. org, 2017.
[13] Erich I. Lehmann and George Casella. Theory of point estimation. Springer Science & Business Media, 2006.
10

---

[14] Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization for machine learning in linear time. The Journal of Machine Learning Research, 18(1):4148-411) 41877 2017.
[15] Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning fragile. arXiv preprint arXiv:2006. 14651, 2020.
[16]
Yeh, Joon Sik Kim, lan EH Yen, and Pradeep Ravikumar. Representer point selection for explaining deep neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 931 1311-932,, 2018.
[17] James Martens and Roger Grosse. Optimizing neural networks with cronecker-factoree approx- imate curvature. In International conference machine learning, pages 2015.
PMLR,
[18) F Kunsttnr, L Balles, and P Hennig. Limitations of the empirical fisher approximation for natural gradient descent. In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), pages 133-4144. Curran Associates, Inc., 2020.
[19] Daniel Ting and Eric Brochu. Optimal subsampling with influence functions. Advances in Neural Information Processing Systems, pages 3650-3659, 2018
[20] Elnaz Barshan, Marc-Etienne Brunet, and Gintare Karolina Dziugaite. Relatif: Identifying explanatory training examples via relative influence. In The 23rd International Conference on Artificial Intelligence and Statisties, 2020.
[21] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Proceedings of the s1 International Conference on Neural Information Processing Systems, pages 6629- 6640, 2017.
[22] Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. Fastif: Scalable influence functions for efficient model interpretation and debugging. arXiv preprini arXiv:2012. 15781, 2020.
[23] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-21) 215, 2019.
[24] Stefano Teso and Kristian Kersting. Explanatory interactive machine learning. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 239-245, 2019.
[25] Martín Abadi et al. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org
[26] Timothy Ye. Example based explanation in machine learning. Timothy-Ye/ wwwpppeeeeaaeedeeeee eaasatiexplanaatoon anation, 2020.
https://github. com/
[27] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
[28] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. EMNLP-IJCNLP, 2019.
[29] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.. lecun. www.:ll.w.lelcca.... com/exdb/nnistt 2, 2010.
[30] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
[31] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprins 1rXiv:1412.69880 2014.
[32] Tommi S Jaakkola, David Haussler, et al. Exploiting generative models in discriminative classifiers. Advances in neural information processing systems, pages 487-493, 1999.
11

---

[33] Dana Angluin and Philip Laird. Learning from noisy examples. Machine Learning, 2(4):343-
370, 1988.
[34] Nitika Nigam, Tanima Dutta, and Hari Prabhat Gupta. Impact noisy labels in learning techniques: survey. In Advances in data and information sciences, pages 403-411. Springer, 2020.
[35] Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373-144 1411,, 2021.
[36] Mike Wojnowicz, Ben Cruz, Xuan Zhao, Brian Wallace, Matt Wolff. Jay Luan, and Caleb Crable. "influence sketching": Finding influential samples in large-scale regressions. In 2016 IEEE International Conference on Big Data (Big Data), pages 3601-3612. IEEE, 2016.
[37] Rajiv Khanna, Been Kim, Joydeep Ghosh, and Sanmi Koyejo. Interpreting black box predictions using fisher kernels. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2392,,39,, 2019.
[38] Ramprasaath R Selvaraju. Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry Heck, Dhruv Batra, and Devi Parikh. Taking hint: Leveraging explanations to make vision and language models more grounded. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2591-26000 2019.
[39] Piyawat Lertvittayakumjjrn, Lucia Specia, and Francesca Toni. Human-in-the-loog debugging deep text classifiers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 332-348, 2020.
[40] Patrick Schramowski. Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska Herbert, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kristian Kersting. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 2(8):476-486 2020.
Interactive learning with global explanations. arXiv preprint arXiv:2009.0977 2020. [41] Teodora Popordanoska, Mohit Kumar, and Stefano Teso. Machine guides, human supervises:
[42] Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 2018
[43] Jeya Vikranth Jeyakumar, Joseph Noor, Yu-Hsi Cheng, Luis Garcia, and Mani Srivastava. How Can I Explain This to You? An Empirical Study of Deep Neural Network Explanation Methods. Advances Neural Information Processing Systems, 33, 2020.
[44] Luigi Malago, Nicolo Cesa-Bianchi, and J Renders. Online active learning with strong and weak annotators. In NIPS Workshop Learning from the Wisdom of Crowds, 2014.
[45] Pang Wei W Koh, Kai-Siang Ang, Hubert Teo, and Percy S Liang. On the accuracy of influence functions for measuring group effects. In Advances in Neural Information Processing Systems, pages 5254-5264, 2019.
[46] Samyadeep Basu, Xuchen You, and Soheil Feizi. Second-order group influence functions for black-box predictions. arXiv preprint arXiv: 1.00004, 2019.
[47] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pages 2242-2251 PMLR, 2019.
12


Reviews:
[{"review_id": "yP_I8kO7pl4", "reviewer": "Reviewer_j1iQ", "summary": "The paper introduces an approach called CINCER which tackles label noise in a sequential learning task by querying a human annotator whenever it finds a suspicious example. In support of its suspicion, the model also presents the user a maximally-incompatible counter-example from the already accumulated training set. The primary contribution of the paper lies in choosing an informative counter-example using Fischer information matrix as an approximation of influence functions. CINCER enables the user to correct both incoming and old examples.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written with adequate representation given to prior work. I would, however, like the authors to include the highly-relevant and recent work on Confident learning in their literature work (Northcutt, Curtis, Lu Jiang, and Isaac Chuang. \"Confident learning: Estimating uncertainty in dataset labels.\" Journal of Artificial Intelligence Research 70 (2021): 1373-1411.). \n\nThe idea of presenting the human annotator with a maximally-incompatible example as a means to explain the model's suspicion is an exciting idea. The authors admit that the counter-examples produced by CINCER may not be perceptually similar, which may impact the their explainability. However, I agree that future work may focus on some reasonable tweaks to incorporate that aspect. Another recommendation is having the ability to incorporate batching since it seems inefficient if a model is trained every time a new training example is obtained.\n\nI feel that the claim that CINCER can also clean the already accumulated dataset is slightly stretched, primarily because the discovery of old mislabelled data is somewhat driven by the new incoming data. Therefore, there's a significant chance that some of the old mislabelled data may never come to light. In this regard, the experiments are also a bit unclear to me. For instance the paper mentions that \"20% of the training examples are corrupted at random\" and \"before each run, the models are trained on an bootstrap training set of 100/500 examples\". It is unclear as to whether (1) the bootstrap training set has mislabelled data points, and (2) what is the proportion of mislabelled data in the initial bootstrap sample. Explicit control on the noise in the bootstrap might allow us to test the model's ability to reveal mistakes in the \"old\" dataset. Since the paper claims that this is a key differentiating factor from prior work on skeptical learning, I would be interested in seeing such experiments. \n\nIn Section 4.1 Q1, I believe it would be interesting to have NN as another baseline because from Section 4.2 it doesn't seem to do significantly poorly. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have addressed the limitation of their works remarkably well. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "fItaKTmGTKS", "reviewer": "Reviewer_hs1w", "summary": "The paper introduces a new data cleaning technique (CINCER), which selects potentially mislabeled samples to present to an expert for relabelling. CINCER works by identifying so-called `suspicious' samples and selecting a counter-example for each of them. The suspicious sample and the counter-example are both provided to the human expert for re-annotation. Samples are selected by maximizing the margin for a probabilistic classifier, while counter-example selection is performed via an approximation of the influence function which uses the Fisher information matrix. The use of the approximation is theoretically justified (mostly by pointing to relevant work), and experimentally shown to be a better choice than the alternatives from either the perspective of computational cost or performance (data cleaning / F1 score of resulting model).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduces a new and pertinent way of selecting samples for label cleaning. The method is clearly presented, with every step being explained, most importantly the need for approximating the influence function. The choice of approximations using the FIM does rely on past work, albeit its use in this context is ingenious. The authors are also as upfront about the (unavoidable) limitations of their approach as they are of the advantages. The authors have also nicely structured the experiments in support of each of their hypotheses, making section 4 clear and compelling. Overall, the method is sufficiently sophisticated and of demonstrated usefulness to be of interest to the community. \n\nA weak point of the paper is related to the claims around interpretability: The authors indicate that the counter-examples serving as 'explanations of the model's suspicions'. This isn't strictly speaking the case, according to many percepts of explainability/interpretability. While the counter-examples could provide some clues as to why the model might have made a mistake (or, conversely, point to a labeling error), it is far from providing a proper explanation (an example of an explanation, in the case of figure 1, would be 'the sample was selected because itt was labeled as a 9 yet there's a gap in the top part').  This doesn't diminish what the method actually achieves, however, statements that CINCER explains `the reasons behind the model’s skepticism' should be toned down, to avoid over-claiming wins on the interpretability side, especially since no actual user study was conducted.\n\nAnother limitation of the paper is that all the data corruption was performed in the same way (20% of training samples were corrupted at random in all datasets). This, of course, means that mislabeled samples are relatively frequent (thousands, in some cases, out of which the model cleans around 20-30). What happens if the mislabeled samples aren't quite that easy to find? (i.e. the percentage of mislabeled samples is lower) What happens if the mislabeling is systematic?\nIt would be interesting to see how many of the cleaned samples are 'counter-examples' and how many are the suspicious samples themselves as an indication of how effective CINCER is in dealing with past data.\n\nNeedless to say, the experimental section would be a lot stronger if a real  crowdsourced dataset was selected, with real human expert reviewers to correct the labels. If two labels are provided per iteration and there are a hundred of iterations, such a study is perfectly doable.\n\n\nClarity/presentation issues:\n\n- Figure 1 mentions counter-examples being 'contrastive and influential' when neither of these terms have been clearly defined yet.\n\n- the related work in section 5 is somewhat repetitive of the introduction\n\n- typo: \"details depends\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the work are addressed. In terms of negative societal impact, this work is on par with other data cleaning techniques requiring the intervention of human users -- for instance, it is vulnerable to malicious annotators. However, this is due to the nature of the task, not a drawback of the paper itself.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "UGjIoyA5v4l", "reviewer": "Reviewer_ikcA", "summary": "The paper tackles the problem of label cleaning, where labels for incoming data points might be flawed. The authors propose an interactive learning approach, where the system builds counter examples for data points, which consist of pairs of data points which the model finds incompatible. The human expert is then asked to relabel either or both of these presented data points.\n\nThe empirical evaluation applies the method to tabular- and image data, comparing the proposed approach with skeptical learning and a variation of the proposed approach without counter examples. The results show that the approach outperforms its competitors in terms of F1 score. The authors also validate that using the fisher information matrix supports finding good counter examples.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive points:\n* The paper is well-written and has clear design decisions\n* Although the paper reuses ideas, the resulting application is sufficiently novel and original\n\nNegative points:\n* Empirical evaluation could ablate margin parameter and use more challenging data sets\n\nThe paper proposes an interesting and well-motivated solution to label cleaning. The approach conceptually extends the setting of cleaning novel data points by also enabling to clean the available dataset, which is sensible in online settings with little to no initial knowledge.\n\nThe solution to finding expressive counter examples for expert labeling is also well-motivated, building on the concept of influence functions and fisher information. The paper also contributes with transferring the use of influence functions to the online setting.\n\nWith respect to the elaborated related works, I only have one comment/question. The provided list of papers and explained differences are sensible, however: the authors state that LWA assums a pool-based setting, where data is not perceived fully online. Could you please elaborate why label cleaning requires the full online setting? It would be good to thoroughly motivate it, as it is stated as main reason to build on / compare against skeptical learning only. I understand that the training set would not be built fully sequentially in the pool-based setting, but could one repeatedly use LWA on smaller batches?\n\nI find the evaluation to be convincing, as diverse data sets have been used and the method has been partially ablated and compared against skeptical learning. It would, however, also be interesting to test approach on challenging image datasets (other than MINST-based datasets), but it is sensible to start with those. Also, did you also ablate the margin threshold parameter? What would happen if one decrease/increased the parameter? How easy is it to tune it for new data?\n\nLastly, I was wondering in what (adversarial) cases the approach could break in the online setting. Could the approach converge to not asking the expert sufficiently often or to never find a large portion of noisy labels? If yes, could one categorize such cases? What if the amount of noise is very large?\n\n### Update after author response:\nI thank the authors for the thorough answers to my questions! After reading the other reviews, my assessment of the paper remains utterly positive. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed potential negative societal impact in the conclusion. The authors could additionally address limitations of the system in terms of potential (special) situations where the approach might break.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "NJnwRsuTHFV", "reviewer": "Reviewer_CVDx", "summary": "The article introduces a novel approach to interactively cleaning the erroneous labels for an ML task. It works by identifying highly-influential, mutually-incompatible examples. The authors show that, over time, correcting the newly acquired example (or the incompatible old one, or both), does improve the accuracy of the learned model. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose a novel approach to an important problem: improving the learned models by detecting and leaning the labels of mis-labeled examples. The paper is well written, easy to follow, and makes a compelling case for the proposed approach. The contribution appears to be original. The authors make a strong case for the use of influence functions in conjunction with a few explicit criteria (contrastive, influential, and pertinent).\n\nIn this reviewer's opinion, the paper could be improved by a few additions:\n- adding an error analysis section: what proportion of the mislabeled examples can you detect? what proportion of the detected  contrastive pairs do NOT contain at least one example for which you did NOT corrupt the label? \n- adding to Figures 2 & 4 the upper-bound performance of the model (ie, if it were trained w/o any corrupted labels)  \n\nOTHER COMMENTS:\n- line 35:         replace \"computes\" by \"identifies\n- line 73:         can't one make the argument that each example in a sequence is a pool of size 1?\n- line 81:         can --> may/will\n- line 81:         please point to the exact section/paragraph/experiment that illustrates the detrimental effect\n- line 158:       please provide details on unstable behavior \n- line 174:       please quantify HOW \"MUCH MORE\"\n- line 223:       this is a bit \"out of the blue\"; please provide details or remove (as it is highly speculative, and does not belong in a advantages/limitations section)\n- lines 7/12/200-202 make this approach similar in spirit with the \"aggressive co-testing\" one in the paper below (including the value of detecting the outliers); while your approach is completely original, the similarity may be worth noting.\n\nMuslea, I., Minton, S., & Knoblock, C. A. (2006). Active learning with multiple views. Journal of Artificial Intelligence Research, 27, 203-233.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Four knowledgeable referees have thoroughly reviewed this paper and all recommended that it is accepted. I agree with their recommendation.
Decision: Accept (Spotlight)