Repairing Neural Networks by Leaving the Right Past Behind
Authors: Ryutaro Tanno
Melanie F. Pradier
Aditya Nori
Yingzhen Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: model repairment
interpretability
continual learning
data deletion
debugging
interpretability

Abstract: Prediction failures of machine learning models often arise from deficiencies in training data, such as incorrect labels, outliers, and selection biases. However, such data points that are responsible for a given failure mode are generally not known a priori, let alone a mechanism for repairing the failure. This work draws on the Bayesian view of continual learning, and develops a generic framework for both, identifying training examples which have given rise to the target failure, and fixing the model through erasing information about them. This framework naturally allows leveraging recent advances in continual learning to this new problem of model repairment, while subsuming the existing works on influence functions and data deletion as specific instances. Experimentally, the proposed approach outperforms the baselines for both identification of detrimental training data and fixing model failures in a generalisable manner.


Repairing Neural Networks by Leaving the Right Past Behind
Ryutaro Tanno d Melanie F. Pradier'
Aditya Nori'
Yingzhen Li? IImpeiil College London, UK
Miicosooft Health Futures, Cambridge, UK
Abstract
Prediction failures of machine learning models often arise from deficiencies in training data, such incorrect labels, outliers, and selection biases. However, such data points that are responsible for given failure mode are generally not known a priori, let alone a mechanism for repairing the failure. This work draws on the Bayesian view of continual learning, and develops a generic framework for both, identifying training examples which have given rise the target failure, and fixing the model through erasing information about them. This framework naturally allows leveraging recent advances in continual learning to this new problem of model repairment, while subsuming the existing works on influence functions and data deletion as specific instances. Experimentally, the proposed approach outperforms the baselines both identification of detrimental training data and fixing model failures in a generalisable manner.
1 Introduction
Machine learning (ML) models often exhibit unexpected failures once deployed in the "wild". Recent lines of research aim to alleviate different well-known shortcomings in supervised models, such as vulnerability to annotation errors and adversarial attacks (2), sensitivity to data shifts [3. and biases to underrepresented subgroups 4 However, it is often challenging anticipate beforehand all plausible failure scenarios, and protect against them pre-emptively. This motivates developing technique that is able repair model on demand, new failure cases arise in practice.
Undesirable behaviours of ML models commonly stem from defects the training data. However, is un- clear how to detect the causes of such failures automatically, rendering manual troubleshooting necessary. Furthermore, once the problems are uncovered, one would still need to design fixes, which typically involve further data curation/collection. and model retraining/redesinning from scratch. Executing the above steps demand not only time, but also mature expertise in the relevant MI. areas, a scarcity in the present job market.
This work introduces approach to identifying of most detrimental training examples that have caused failure cases observed test time, and to subsequently repairing the model on these failures by deleting those culprits. At the basis of both cause identification and repairment steps is the approxination of "coun- terfactual" posterior distribution where some training examples are assumed absent. We formalise this as a Bayesian continual (un (un)lannnigg problem [7], where the above counterfactual posterior is estimated by delet- ing the evidence of selected training data from the current posterior. We note that, while other factors (e.g. model class and optimisation) may play role, we focus on "data debugging" and investigate, what extent, prediction failures could be remedied by only intervening on data and updating the model accordingly.
Fig. D gives overview of the proposed approach for model repairment, which operates in two steps by 1) identifying causes of failure among training data, and 2) updating the model by erasing the "memories" of those harmful examples. Importantly, the proposed framework is agnostic to why a particular datapoint hurts model performance, handling both, issues in the input data and/or labels. We only require specification of the set of failed cases for which we wish to improve performance.
Now at DeepMind, UK. Correspondence to Maanoogoogglecomm com and yingzhen. li@imperial. .aa.eriaaal..................... uk
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

(a) Training Data with Problems & Resultant Failures at Deployment
(b) Identify the "Causes", C. of Target Failures, FF
(c) Repair Model by Erasing Information of Failure Causes, C
Tratning dana D
Undenirnall behovinnrr or deployment
F
Fixed modde!
jut of faiturrs, F
Figure I: (a) Real-world datascts are often fraught with issues such as annotation noise, low-quality inputs, anomalies, and acquisition biases (e.g. demographic imbalances). Such issues may lead to undesirable performance of the trained models in deployment. Our approach aims at repairing such models by (b) identifying detrimental training examples which have caused the target failures, and then (c) erasing efficiently the memories of those examples from the models.
Our contributions: We develop a framework for repairing machine learning models by erasing memories of detrimental datapoints. The framework connects both identification and removal of detrimental data under (Bayesian) continual learning perspective, which brings forth practical benefits. Firstly, the framework subsumes works on influence function [8] and data deletion [ as specific examples, which are developed independently, and our work reveals their close connections and limitations. Secondly, the generality of our formulation allows translating any continual learning method into this model repairment setting, and opens doors to further research. In particular, extend Elastic Weight Consolidation [10 specific continual learnigg algorithm - to cause identification and data removal, and demonstrate improvements over the prior works in variety of settings where training data are contaminated with annotation and/or input noise.
2 Model Repairment by Data Deletion
Let us consider a prediction model p(y|æ,,) that returns a probability distribution of the output y given an input We make the i.i.d. modelling assumption and denote p(0)D) the posterior distribution over the model parameters 0 given training data D= ((z(()))))))) """"")))))))))))) ( At test time, the posterior predictive distribution is used to infer label y" given a new sample xt:
f plu |a= f)
ply |x" p)=
D)
where p(y" |x* .)) is the likelihood term for sample (x".y") .y"). While we use approximate posteriors in practice, we focus on the exact case for now to formalise the problem.
Imagine that, in deployment, this model makes incorrect predictions in certain situations. After collecting a "failure set" F
(n) ; 3 of examples from such failure mode in the test se we would like to repair the model, such that it improves performance on the failure set and similar future cases. We also argue that successful repairment should also maintain a similar level of performance on the rest of test ex- amples. These two objectives of model repairment are analogous to those of a medical treatment, in that both aim to fix specific problem while leaving the "healthy" part intact as possible. A further discussion of different aspects, including generalisatin,, efficiency and specificity properties is provided in appendix(A.I
In this work, we assume that the main reason for such failures F is due to the existence of detrimental examples in the training data D, for example, noisy labels, low-quality inputs and/or group imbalances. Our hypothesis is that, by removing these harmful datapoints and adapting the model accordingly, the model can be repaired to return correct predictions for datapoints in as well as similar future cases. We acknowledge that there might be other reasons for a model to make wrong predictions on F. such as bad local optima and model biases, which are outside the scope of this work. Addressing data-based failures is complementary to accounting for other problems and is of relevance regardless, as datasets typically come with unexpected issues no matter how much we curate them beforehand.
We formulate the process of model repairment as the following two steps (see Fig. for an illustration):
1. Cause identification: Identify set of detrimental datapoints, i.c. "failure causes" C in the training data D that contributed the most to the failure set
2. Treatment: Given the of failure causes C. adapt the model to predict correctly on the failure set F, while maintaining performance on remaining test examples.
"There may be multiple different ways in which the model fails [1]. and the failure cases may consist of several groups. One type of mistake might incur more costs than others (e.g. in certain medical applications, false negative is more costly than folse positive). Here assume that we have identified at least one failure type that would like to fix.

---

Algorithm Model Repairment
Input: training data D) failure cases F; approximate posterior (()))= likelihood p(=
Output: failure causes "repaired" posterior q-c (0)
Step I: Cause Identification
Update posterior: Apply continual learning method to ohtain q+ R3 p D,F) by fitring the failure set JF Compute influences of training examples on F; Calculate F(=)Vz E (Eqq. (9)
Find failure causes C: Return the examples with positive influence, C (= E F(z)>0) Step II: Treatment
Delete information of C: Apply a continua (unjlearning method to the original posterior q(0), and obtain the posterior on the corrected data =()
Below we describe a formal framework for performing these two steps. In the first phase of cause identifi- cation, we need to define a measure of how much a subset of training examples C= [(ai") ./c ) C is responsible for the failure cases F. To this end, we propose to check how much the posterior predictive distribution on J changes as a result of deleting from the training data:
where P(F(D) and p(F(D)(C) are the posterior predictive distributions before and after removing subset of training examples C' defined as follows:
P(F(D)= f
p(FDD)C) (at--/(((0(00////C p(F())=
II p(yx,.0) (3) (=.
If given C' leads to a large positive value of (CC) in Eq. ( it means that removing C. from D would have im- proved the performance of the Bayesian predictive inference on the failure set F. The first cause identifica- tion step thus entails finding subset C with the maximal log- log-density ratio rCC). In the second treatment step, we can directly adopt p(FlD C)) for such identified as the updated predictive distribution, as it confers the largest improvement over the failure cases. In other words, the posterior predictive distribution p( ((F (C) after data removal is key to both the search of detrimental datapoints and repairment of the model.
The central computational question is, therefore, concemed with the efficient calculation of p(FID) C) without retraining the model from scratch. The reasons for avoiding retraining are: (1) in the cause identification step, it is computationally prohibitive retraining needs to be done for every subset C D; and (2) in the treatment step, the resulting retrained model may be drastically different from the original model one would like to fix (due to, e.g. noises in the SGD-based optimisation and parameter re-initialisation), and may lose other desirable properties that one may wish to retain.
In this work, we present continual learning [12 framework to simultaneously address the aforementioned computational challenges cause identification and treatment. The framework is summarised in Algorithm i with the following key developments by leveraging (Bayesian) continual learning:
For cause identification (Section 2.1). we present a fast approximation to r(C). which requires a one-off approximation of ((0)D. (D,FF F.F) only via continual learning, and enables linear-time search for the detrimental datapoints in C.
2. For treatment (Section 2.2). we show that the approximation of p(0) (C) for C' identified in the first step can be achieved by performing new "continual learning task" using logp(Cl() as its "loss function".
Our framework is generic and flexible in the sense that any continual learning approach can be applied to both steps; we demonstrate, in particular, a concrete instantiation by leveraging Elastic Weight Consolidation 10 as the base continual learning approach. We now elaborate on the mathematical details of the two steps.
2.1 Step I: Cause Identification
Identifying the set of detrimental examples C requires solving the following optimisation problem
(4)
where IP) denotes the power set of D. Solving this comes with multiple computational challenges.
3

---

Firstly, naive approach would require computing the predictive distribution p(F(D),C and thus, the posterior p(0)D' C) - for every subset C of D, which prohibitively expensive. To address this, we present a "predictive" approach that removes this computational burden. The key idea is to notice that
(PPP0)/cC)))))) VCCD,
due to the i.i.d. modelling assumption. Inserting this into the Bayes' rule for computing p(O)D)(C) yields:
p(F10) p(D(O)p(() d0 -log (())C)
logp(FiD(C)=loo p ((F10))(01D)C))00
plC 0)
P(D)
((D)
Notice that again due to the i.idd modelling assumption,
P(D(C)
1 O(D
do.
(DD)
p(c10) ((D)
Therefore, we can compute the log density ratio r(C) = log [ D C) logp(F(D) in the following
"predictive" form, without computing p ( (D ()) (C) Appendix A.2 for derivations:
((OODDF) do d9-log log p(C|0)
O(O d0= log (xCC))) 1 p(C
((C)=log]
(5)
In this form, we only need compute the posterior p DDF) once, and can side-step the requirement of computing p(0 C) for inspection of every new C D. thereby removing one of the key computational bottlenecks. Intuitively, as the "predictive" formulation (Eq. (5) of r(C) computes the log expectation difference of p(c higher r(C) means that datapoints in C are less likely to be predicted correctly when the posterior updated by the information from the failure set F. In other words, this indicates conflicting information exists between F and C, and as we would like to repair the model to produce correct predictions on F and similar examples, the information of C should be removed from the model.
Secondly, we are still left with the combinatorial search for the best subset C, which is also prohibitive when the size of training data D is large, even when the "predictive" formulation Eq. (5) is used. We address this issue by a first-order Taylor series approximation of r(C). Let us re-write the log density ratio as
-
where F(c.(()) =log) clgp(cl0
(6)
Note that F(0.g(0))= 0 for any well-defined distribution g0)). We now perform a Taylor expansion of F(), around = 0: ((),g(0)) = -elyoto) [log((C 0)] + OEe)). Finally we obtain approximate log den-ityity ratio ((C)=r(() by plugging the first term of the Taylor expansion into the RHS of Eq. (
F(C) = Fp))=)p [Cg) "
Assuming that data are i.i.d, and defining z= (x.y). the above approximation can be expressed as the sum of individual log density ratios, #(C) - > (z),, where each term is given by
f(z) E p [log(( [)] E D.F) [oop(=]0)), p(z
Critically, with this approximation, in order to find a subset C of cardinality K that leads to the maximal F(C), it suffices to compute f(z) for every training example D and find the top K examples with largest F(z) values, thereby reducing the search space from O(D!!) choices to only O(D)) choices.
The per-instance formulation Eq. (8) has similar interpretation as the "predictive" formula Eq. (5), in that f(*) measures how much the predictive moments of 2 changes when the model is further trained on F. i... computation of p(0 D.F). If the difference is positive, i.e., f(z) >> 0, it means the example a conflicting evidence against the test examples in F: conversely, if f(z) <0, In practice, the failure causes correspond to examples with f(2)>0.
D O, then z and F are aligned.
Lastly, for non-linear models (e.g. neural networks). approximate posteriors (0) s p(0 D) and (+=(0) Rs p(0 D)F) are needed due intractability of the exact posteriors. We assume that ( is available after training, and suffers from the prediction failures F. As recomputing the posterior (+) from scratch can be expensive, we propose to use a continual learning technique [12] and obtain this quantity by updating the original posterior (0). Finally, we use the following metric f(z) in practice to calculate the detrimental impact of each training datapoint on the failure set, F. by replacing the exact posteriors in Eq. (8) with their corresponding approximations:
F(z) : E [log( (z 0)) -Bg. [logp( ())]
(9)

---

and the top K entries according to f(z) are selected to approximate the failure causes C. This metric is generic, and its implementation depends on the specifics in which both q(0) and q+ =()) are computed, e.g..
15 etc. Two concrete MLF/MAP point estimates, Laplace approximation 13] variational inference [1
g is specífic instance of Eq. ( and the second is derived by extending a continual learning method, known as Elastic Weight Consolidation (EWC) [0] to cause identification, and is a key methodological development in our work.
examples are provided: the first shows that the well-known linear influence function funttioon
Example (Linear Influence Function): Our proposed metric in Eq. (9) recovers the linear influence function from Koh & Liang []] when point estimates are used for 0. Assume that the model is trained on data 'D with parameters 0, which corresponds to an approximation of MI.E/MAP estimates, i.e., ((0)=5(0-0)) R (()))) After observing the set of failures F, point estimate of (() is obtained by performing single update of natural gradient ascent [16] on the log likelihood of JF with step size >>>>>>>>>> 0:
q+ (+ (0-
0 0+5 'V Vologg(Fi0).
(10)
where Fg is the empirical Fisher information matrix. This means
F(z) - golo---
F V glogg(z((0)
(11)
if defining the updated posterior as 9+ 500
The negation of the above equation coincides with the linear influence function (Eq. (2) in S when the failure set is assumed to be a singleton F= (=test) ) and y= 1. Note that the sign difference arises since our work aims to quantify the "negative"
0+F). (0)50(0-0-
influence rather than the "positive" one in contrast with the work of Koh and Liang (8).
Example 2 (Elastic Weight Consolidation): The generality of Eq. permits any continual learning method of one's 's choice for estimating the updated posterior q+ (+F(() pep F)F) after observing failure samples F. Here we illustrate how EWC [10 as a continual learning method can be adopted in the context of cause identification. EWC approximates p(F) by first performing Laplace approximation of the original posterior p(0 D) around the point estimate Ô, and subsequently finding the MAP solution O. For- mally, 0+ is obtained by maximising the objective below w.r.t. O via SGD (see Appendix A.3 for details):
N
logp( [F 1)
10 0
(12)
2
where the off- dia-onia elements of F are dropped for memory reason in practice. Intuitively, the first term encourages high accuracy on the failure set F whilst the second/third terms ensures the model parameters do not deviate too much in distribution. Defining ((0)= 5 θ) and
we have that
=lx)=lo))(=(0) logp (=)
(13)
To compute the above for each datapoint E D. we only need to solve the optimisation problem of Eq. [2)
by SGD once. We refer to this version of F(z) as EWC-influence function.
Comparison: The EWC-influence function generalises the linear influence approach. To see this, we derive the fixed point of the EWC objective Eq. 12) w.r.t. O (where we set =0)):
0 0-0-N 9 Vologg(F(0).
(14)
Then the update in Eq. 10) that is implicitly used by linear influence function can be viewed as (damped) one-step fixed-point iteration update initialised at for solving the fixed-point equation. As EWC-influence update (Eqq. (13) is obtained by using the optimum of Eq. (12). it is arguably more accurate than linear
influence function (Eq. (TT)) for measuring the (detrimental) effect of datum z to the model failures F.
2.2 Step II: Treatment
Once the causes C of the failures F are identified among the training data D, we seek to repair the model (() by erasing the memories of C. We formalise this problem as the computation of the posterior (C), i... C is absent from the training data D. A naive approach would re-run approximate inference on the whole "corrected" dataset C to obtain an approximate posterior q-c(0) (0)))((()))))))))) s0(() (C) which can be time consuming. But more fundamentally, by doing so, the obtained c-c(O) may be unrelated to the original g()) based on which were identified, due to, e.g. non-convex SGD optimisation issues. Moreover, this "model replacement" approach may not maintain other good properties of the original model q()).
5

---

Analogous to cause identification (Sec. 2.1 we propose to employ the continual learning approach to esti- mate efficiently the modified posterior p C). Applying Bayes' rule and some algebraic manipulations yield p(0 DD C') px (p(C))) (see Appendix A. Therefore the information about C can be removed by scaling the current posterior p (D) by the inverse of p(C)0) and re-normalising. In other words, we can treat the approximation of p (()) as a continual learning task, where the task is to "unlearn" the datapoints while using the posterior distribution pD) as the prior. In practice, the target model to be fixed corresponds to the approximate posterior q(0) (D)D) Therefore continual (un)learning is done by
(15)
The above approximation can be carried out with different approximate inference techniques such as MLE/MAP point estimate, Laplace approximation [10] and variational inference 77. Again provide a few examples to concretise this process as follows.
Example 1 (Fine-tuning on Corrected Data): Given a point estimate of model parameters Ô, i.e., = - 0). a simple way to approximate p(0)D' ()) is to fine-tune on the corrected dataset D D), and update the point estimate. The new 0-c of the repaired model are obtained by maximising the log-likelihood logp(D)(Cl0) via SGD, starting from 0.
Example 2 (Newton Update Removal): Guo al. ( proposed Newton update based method for data deletion. This method reduces to a specific form of Eq. when using log- ligliikli as its loss:
θ 2 FF, logo(CI0).
(16)
Here information about gets deleted by performing a single-step natural gradient descent on their log likelihood [1).. Also, notice the similarity with the way linear influence [8] is computed in Eq. 100 illuminating the relation between cause identification and treatment steps.
Example 3 (EWC for data deletion):
The update rule in Eq. (15) for data deletion is amenable to any continual learning approaches. For example, given model parameters 0, EWC-based deletion obtains new parameters 0 by maximising the following objective (see Appendix A.4:
N (0 (0-0) (0- Fg ( 1)00)))))000000000 01
-logp(C(0)-
(17)
where the first term seeks to remove information about C while the remaining terms discourage parameters from deviating from the original values. Contrasting this with Eq. (12 again reveals the connection
between EWC methods for cause identification and treatment steps.
Comparison: Similar to the comparison made in the cause identification part, EWC for data deletion also generalises the Newton update removal (Eq. (I6). ). This can again be shown by deriving (damped) one-step fixed point iterative update starting from Ô to approximate the fixed point of Eq. (7) when = 0. As EWC for deletion uses SGD approximate optimum of Eq. (17), it is arguably better than Newton update removal for erasing the effects of detrimental examples C, while better maintaining performance on other cases.
Connecting the two steps: We highlight the deep connection between the "predictive approach" for cause identification (Sec. 2..). and the continual (un)learning for data deletion in the treatment step. They share =
the key idea of editing the approximate posterior ) ( D)) via continual leamning which corresponds
to editing the factor graph [17 of p(0 (D), e.g., insertion of p(F((0) in cause identification, and deletion
of pCC))) in treatment. This unified view enables ones to take any continual learning method and extend it to both steps of model repairment. Indeed in our experiments, EWC as a better continual learning approach than e.g., one-step Newton update leads to improvement in both tasks over the prior works.
3 Related work
Model Editing. There is a recent surge of interest in developing targeted updates to correct model's unde- sirable behaviours, while leaving other desired properties intact. As naive fine-tuning methods often lead to overfitting to the failure examples and accuracy degradation on others, various strategies have been proposed. For example, Zhu et al. [18) employ a simple regularization technique to minimize parameter changes during the fine-tuning phase. Subsequent works [9 20 advocate for functional regularisation instead, e.g. KL. divergence in the output space, to achieve better regularisation. These lines of work, additionally, propose to meta-leaming 21 to learn to edit the target model, where the latest meta-leaming approach is proposed

---

by Mitchell et al. (22). Another promising approach 23 performs weight editing so that features of spe- cific concept (e.g, snow) map to the features of another (e.g. road). A commonality among these approaches is the focus on direct model edits for correction. Our work takes an orthogonal and under-explored angle where the aim is to "edit" the data instead, by identifying and removing harmful examples which cause failures - in turn, this difference makes our framework complementary to these model-editing approaches.
Continual learning. Continual learning is an active research area with related but broader scope than model repairment, which aims to develop methods that adapt the model for future tasks while maintaining model performance on previously learned tasks [2]. We focus on a more targeted problem in this work, yet introduce a framework that allows the use of any continual learning approach for model repairment. Our experiments presents EWC [10] as a practical instantiation of the framework. One can also leverage improvements over EWC such as online EWC (24), or other regularisation-bate
methods that are
motivated by Bayesian learning principles, such as variational continual learning
25 26], synaptic
intelligence [27], and orthogonal gradient descent (28). As approximations to r(C) rely on accurate
posterior approximations, advances in Bayesian continual learning methods are expected to improve the practical effectiveness of model repairment under our framework.
Data Selection and Valuation. Multiple techniques have been introduced for selecting "influential" training
examples on chosen metric (e.g. test accuracy), such as influence functions [8 30
31
32
33
34] Shap-
ley value-based approaches 35 36 37 and probability of sufficiency 38 Within the category influence
functions, two representative approaches include linear influence function g and ieD-influence (32.. The
former approach performs one-step update only, thus, while efficient, may be less accurate in reflecting the influence of datum z. The latter approach computes a projected difference between 0 and 0+ but with 0., obtained by running SGD fine- tine-tuning on training data without z. Thus SGD-influence is computation- ally inefficient. Compared to both baselines, it produces more accurate influence estimates than linear influence due to better optimisation, while it is more efficient than SGDD-iflluence as it requires only one optimisation procedure the given failure set F.
EWCC-inflencce approach achieve the best in both worlds:
Data Deletion. The detrimental data removal in the treatment step is related to data deletion, a rapidly devel- oping field of machine learning research [39 40 41 42 43. Closest to our work is variational Bayesian unlearning [4] which extends variational Bayes to data deletion settings. But the connection to continual learning is not explicitly made, and it is limited to applications in logistic regression and sparse Gaussian processes. In general, the main focus of existing data deletion research is to preserve data privacy, and datapoints to be removed are assumed provided. On the contrary, in this work. we focus the repainment of models and propose a unified procedure not only to remove data but also to identify which ones to do so.
4 Experiments
We evaluate the efficacy of the proposed framework in a) identifying the causes of target prediction failures in Sec. [. and b) repairing the original model by erasing the memories of such causes in Sec. 4.2 We augmented versions of MNIST and CIFAR- 10 datasets with simulated annotation and input noise. Such controlled experiments are performed for creating "ground truths" of failure causes - necessary for validating the quality of identification methods and for testing the method in a variety of settings.
Baselines. For the cause identification task, we compare our approach (EWC-influence) against the linear
influence function g and SGD- influence 32]. To avoid expensive computation of F Koh & Liang
g introduced two efficient approximations to the Hessian-vector product F. (Flog)FF))) the first solves arg min.(t" i. -VVu-loog(F(0)' with gradient descent (GD), while the second uses an iterative algorithm for stochastic approximation (SA) from [45). We implement these two variants (GD & SA) of linear influence in Pytorch, and use the original implementation for SGD-influence. For the model treatment task, we compare our method (EWC-deletion) against Newton update removal [9]. This method again requires computing Hessian- vector product for which we employ the same stochastic approximation technique (45). To isolate the evaluation of cause identification and treatment, we further consider in Section 4.1 fine-tuming
D C another repairment strategy, which would return the best repairment result if the set C correctly captures the detrimental datapoints. Lastly, we set the prior term e2 zero in eq. (12 and eq. (17 ensure fair comparison with linear influence and Newton update removal.
Common Set-up. We train the base classification models on the training split of the "augmented" MNIST and CIFAR-10 10 datasets. For MNIST, we use 6% (3000 samples) of the original training set to make the task
7

---

(a) Label Noise Models
(b) Class Distribution Missllssiifcation
(c) Label Noise Identification
(d) Top Failure Causes
Fraction of incoment Labels Fined Praction of trcomeet Labels Rooed
MNIST
7
9
9
7
True label
Count
9 7
7
9
2. 5
7
7
7
7
whbare 1501
LOCEL
7
9
7
9
Classes
CIFAR 10
True label
Count
Predicted label
Classes
Fraction of Data Inspected
Figure 2: Results on cause identification in the presence of annotation noise. (a) shows confusion matrices used
to simulate class- dass-dependent label noise on MNIST and CIFAR- 10. (b) shows the class distribution of the misclassified examples for single run. (c) plots how much of the identified causes match the samples with incorrect labels for different approaches. The shade represents the standard deviation computed from 5 different runs. (d) shows the
top 16 causes of the failures as ranked by EWC-influence. An enlarged version can be found in Appendix D
(a) Fixing Failures
Accuracy on Remaining Test Set
(b) Maintaining Accuracy
MNIST
MNIS
CIFAR-20
1.00
Suee
Accuracy Fallure
firtees
Cully
04
O mocial
Seuus
BRC
CN de awour.
T
14"
30th
Number of Instances Removed
Number of Instances Removed
Number of instances Removed Number of Instances Removed
Figure 3: Comparison of the quality of identified causes in the presence of annotation noise, The impact of gradually removing samples in MNIST and CIFAR- 10 datasets in the order of influence values r(z) measured on the failure (holdout and query) in (a), and on the remaining they are calculated on the set of test samples on which the original model makes correct predictions. We also plot the performance of another reference ("semi-oracle") that is the original model fine-tuned on the training data without the la- bel noise instances. The means/stds of all quantities are calculated over 5 runs. See (Dfendi(Dfforrrr an enlarged version.
in (b). We note that in (b), accuracy values start 1.0 as
more challenging. We use instances of CNNs throughout and train them using the Adam optimiser [46]. The architecture and training details can be found in Appendix C For evaluation, we separate the test set T into the set of misclassified examples, FF ("faiture set")) and the others, T which correctly classified ("re- maining set"). We further split the failure set into query, F₂ and holdout, F,, sets, where we only use the for- mer to identify failure causes C, and use the latter quantify how generalisably the removal of C can amend the failure cases. We stress that Fg is used for cause identification only, but not for further model adaptation.
4.1 Identifying Failure Causes
To induce test prediction failures, we randomly flip labels in the training set between semantically similar classes (c.g. and 7 for MNIST, and cats and dogs for CIFAR-IO) 10) according to the confusion matrices in Fig. a). As a result, the classes of miss-classifie test examples are concentrated on those classes with label noise as depicted in Fig. 2bb.
Annotation Noise.
To measure the accuracy of identifying incorrectly labelled examples, we inspect the training examples D the descending order of 7(z) computed with F, which contains 50% of the miss- classclasssfied test cases, and calculate the fraction of incorrectly labelled datapoints in inspected examples. Fig. (]() shows that EWCCiifluence identifies more failure causes earlier on compare to other methods, and is the closest match to the "Oracle" baseline which has full knowledge of samples with wrong labels. Fig. d) shows that the top few causes according to EWC-influence are the samples with incorrect labels, while the least harmful ones are the images of the same classes but with the correct labels as shown in Fig. 7 in Appendix B
8

---

(a) MNIST
(b) CIFAR-10
(c) MNIST
(d) CIFAR-I0
Random Noise
Random Naise
Adversarial Attacks
Adversorial Attacks
Label
Number of Commuted majes Dercettd
Number Co Cormuptedd imgges Dereccees
Accurscy on Faluue Numte of Soseono Cooe WIAA Soriii
Namber ppreet
Number Data tupecied
Nurborr of Inspected
of inspected
humber Data trapacted
Aeurary S4t
Accuracy on faltrre Se:
Accurecy Hemariing Text Accurscy
SrAe D40
ANCUrnty on Remainnngg
PECUPacy on Remainig
Nenanin
04
Figure 4: Results cause identification in the presence of different input noises. From top to bottom, we show examples of corrutted samples (synthetic proxy for potential causes of failure), ii) how many of the identified causes corresoond to samples corrupted with input noise, iii) and iv) performance in failure holdout set Fi, and remaining test set when removing the top 1000/20000 identified causes in MNIST/CIFAR 10. The influence values are calculated with respect to 50% of test- time failure cases that belong to the classes that suffer from input noise. EWC -Wf-lnnleence identifies "harmful" (adversarial) input noise better than random while avoiding "harmless" (random) input noise. An enlarged version is available in Appendix [D
As stipulated in Sec. 2 set of identified causes C is of higher quality if removing them leads to a larger gain in accuracy on the failure set while maintaining performance on others. To measure such quality of causes, fine-tune the base model on D C and report the accuracy the failure query set Fy.. the holdout failure set Fú, as well as the remaining test set, T.F. Results in Fig. 3 suggest that removing failure causes according EWC-influence yields the highest increase in accuracy on the failure set F without hurting performance in the remaining test set T.F We also note that all of the methods are able to fix the failures better than randomly removing datapoints, and more interestingly, for MNIST, when enough causes are crased (* 109), all methods even surpass the case in which all label noise instances are removed. This result implies that, while annotation noise a major detrimental factor, the prediction failures also arise from other types of harmful examples.
Lastly, we evaluate the sample efficiency of cause identification by reducing the size of the query set Fg- J6 Appendix[B shows that all approaches degrade gracefully in repairment performance as the query size gets smaller, but overall EWC-influence still remains the best in terms of label noise detection and repairment accuracy on the failure set and the remaining set.
Random Input Noise. In this experiment, we inject synthetic outliers into MNIST and CIFAR-10 10 and test the quality of cause identification. We select a set of target classes - 1, 7, 6. 9 for MNIST and plane, bird, cat, dog for CIFAR-
and randomly corrupt 30% of the images in those classes by adding salt-and-pe and-pepper noise (i.e. replacing pixels with extreme values and 255) in MNIST and Gaussian noise in CIFAR-I0 10. The top row in Fig, 4 shows examples, and those corrupted images constitute roughly 12% of the whole training set. Sec. C in the appendix provides details.
We use subgroup of failures in the target classes as the query Fg to compute influence values. Surprisingly, the second rows in Fig. 4 a) and (b) show that EWC-influence largely avoids selecting the corrupted images as the top 1000 causes for MNIST and the top 20000 causes for CI. However, the third and the fourth rows show that removing those causes results in the best treatment performance on failures while maintaining the performance a level similar to other basclines. In fact, removing all the input noise and

---

135
Accuracy Remaining
6/72
Reaten Ubeeeeeton
Kedde
Cerette Duta
9.55
DwCoclante
urauungge Camossuuu
Accuracy on Fallure Set (e)
Accuracy Failure Set
(b)
Figure 5: Comparison of deletion-basel treatment methods on (a) MNIST and (b) CIFAR-IO. For Newton-update-
deletion and EWC-deletion, plot multiple results for varying hyper-parameters visualise the trade-off between the accuracy on the failure set and the remaining set. The closer to the top right comer, the more desirable.
retraining is not able to fix the failures by much, indicating that EWC-influence is able to correctly avoid these relatively harmless outliers and detect other more harmful causes. Fig. 8 in Appendix visualises the most harmful examples identified by EWC-influence. Many of them appear be ambiguous instances in non-target classes, e.g. wonky digits, close-up views of vehicles, a real instance with incorrect label (4), etc.
Adversarial Poisoning. To simulate input noise that can induce test-time failures, we introduce contaminated data by randomly corrutting 30% of the training images in those previously mentioned target classes. These poisoned datapoints are adversarial images crafted by the fast gradient sign method (FGSM) 48 on separate set of victim models trained on the original clean datasets, and they are labelled by the classes predicted by the victim models. The poisoned datasets are then used to train the base models that are used for evaluation of cause identification. Fig. (cc) and (d) show that most of the influence functions detect the corrupted samples better than the "random" baseline. The dashed lines in the third row show that removing all of the corrupted inputs lead to a significant gain in accuracy on the holdout failure set in comparison with the random noise setting, illustrating the larger extent of harms caused by data poisoning. However, most of the identification methods still outperform this reference by a large margin. This suggests again for the presence of other influential samples, and EWC-influene is able to pick up the most important ones, judging by the accuracy on the failure set.
Speed Comparison. Table 1 in Appendix B shows the total run-time of cause identification methods on a single GPU for their best sets of hyper-parameters selected based on the treatment accuracy on the failure set. For both datasets, EWC-influence achieves comparable or shorter run time than the baselines.
4.2 Comparison of Treatment Methods
We evaluate the performance of different deletion-based methods for treatment introduced in Sec. 2.2 on MNIST and CIFARIO datasets with simulated annotation noise, used in the previous section. We run both EWC-deletion (ours) and Newton update removal [9] methods with early stopping based on the query set accuracy, and experiment with different hyper-paramest settings (see Sec. Gec. Appendix) to achieve dif- ferent trade-offss between failure set accuracy and remaining set performance. Here EWCCinfluence is used identify the causes, and the top 15% examples were removed by the respective deletion methods. Such trade-off is shown in Fig. where fine-tuning on D',C is included as an "upper-bound" reference for data deletion performance. On MNIST, EWC-deletion attains considerably better trade-off between treatment and maintenance compared to Newton-update -eelttt-,eeeion, and is much closer to the fine-tuning reference. For CIFAR- 10, EWC-deletion beats the Newton-update deletion by 5% in the best failure accuracy while the order reverses for the best accuracy on the remaining test set but with less than 1 % difference.
5 Conclusions
In this work, we develop generic framework for repairing machine learning models by erasing memories of detrimental datapoints. The framework consists of two key components, that are, the mechanism for iden- tifying the "causes" in training data which are responsible for the given failures, and the adaptation method for fixing the model by removing information about them. The two components are connected under the Bayesian view of continual (un)learning, which brings forth several practical benefits. Firstly, the framework subsumes some recent works on influence function and data deletion as specific examples, and elucidate their limitations. Secondly, the generality of our approach allows leveraging recent advances in continual leaming in this new problem of model repairment. In particular, we extend Flastic Weight Consolidation to cause identification and data deletion, and demonstrate empirically its competitive performance in both tasks.
10

---

Acknowledgements
We would like to thank Ozan Oktay, Stephanie Hyland, and Ted Meeds Microsoft Research Cambridge and Jin Chen at University College London for their valuable feedback on an early version of this work.
References
[1] Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: a survey, IEEE transactions neural networks and learning systems, (5(5):8458899 2013.
[2] Han Xu, Yao Ma, Hao-Chen Liu, Debayan Deb, Hui Liu, Ji-Liang Tang, and Anil K. Jain, Adversarial attacks and defenses in images, graphs and text: A review. International Journal of Automation and Computing, 17(2):151-778, 2020.
[3] Pang Wei Koh, Shiori Sagawa, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference Machine Learning, pages 5637-5664. PMLR, 2021.
[4] Antonio Torralba and Alexei Efros. Unbiased look dataset bias. In CVPR 2011, pages 1521-1528. IEEE, 2011.
[5] Irene Chen, Fredrik D Johansson, and David Sontag, Why is my classifier discriminatory? In Advances Neural Information Processing Systems, pages 3539-3533 2018.
[6] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:I911.088 08731, 2019.
[7] Cuong V Nguyen, Yingzhen Li, Thang Bui, and Richard E Turner. Variational continual learning. arXiv preprint 1xXi::7110.10288 2017.
[8] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In ICML, 2017.
[9] Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens van der Maaten. Certified data removal from machine learning models. arXiv preprint 191 2019.
[10] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Bann Barwinska, al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):521-3326, 2017.
[11] Thomas Henn, Yasukazu Sakamoto, Clément Jacquet, Shunsuke Yoshizawa, Masamichi Andou, Stephen Tchen, Ryosuke Saga, Hiroyuki Ishihara, Katsuhiko Shimizu, Yingzhen Li, et al. A principled approach to failure analysis and model repairment: Demonstration in medical imaging. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 509-518. Springer, 2021.
[12] German Parisi, Ronald Kemker, Jose I. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: review. Neural Networks, 1 11::54-71, 2019.
[13] David JC MacKay. Bayesian methods for adaptive models. PhD thesis, Califomia Institute of Technology, 1992.
[14] Michael Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. In Learning in graphical models, pages 105-161. Springer, 1998.
[15] Charles Blundell, Julien Coreeiie,, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In International conference on machine learning, pages 1613-1622. PMLR, 2015.
[16] Shun-Ichi Amari, Natural gradient works efficiently in learning. Neural computation, 10(2):251-276
1998.
11

---

[17] Frank R Kschischang, Brendan J Frey, and H-A Loeliger. Factor graphs and the sum-product algorithm. IEEE Transactions on information theory, 47(2):98-513 -194 2001.
[18] Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:20/2.00366 2020.
[19] Anton Sinitsin, Vsevolod Plokhotnyuk, Sergei Popov, and Artem Babenko. Editable neural networks. arXiv preprint arXlv:2004.00345, 2020.
[20] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. 2021.
[21] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1 1126-1135 135. PMLR, 2017.
[22] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing scale. CoRR, 2021.
[23] Shibani Santurkar, Dimitris Tsipras, Mahalaxmi Elango, David Bau, Antonio Torrabaa, and Aleksander Madry. Editing a classifier by rewriting its prediction rules. In Neural Information Processing Systems (NeurlPS), 2021.
[24] Jonathan Schwarz, Wojeiech Czamecki, Jelena Luketina, Agnieszka
Yee Whye
Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International Conference on Machine Learning, pages 1428-45337 PMLR, 2018.
[25] Pingbo Pan, Siddharth Swaroop, Alexander Immer, Runa Eschenhagen, Richard E Turner, and Mohammad Emtiyaz Khan. Continual deep learning by functional regularisation of memorable past. In Advances in Neural Processing Information Systems, 2020.
[26] Noel Loo, Siddharth Swaroop, and Richard E Turner. Generalized variational continual learning. In International Conference Learning Representations, 2021.
[27] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence.
In International Conference on Machine Learning, pages 3987-3995, PMLR, 2017.
[28] Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pages 3762-377 -3733. PMLR, 2020.
[29] Pang Wei W Koh, Kai-Siang Ang, Hubert Teo, and Percy S Liang. On the accuracy of influence functions for measuring group effects. In Advances in Neural Information Processing Systems, pages 5254-5264, 2019.
[30] Elnaz Barshan, Mare-Etienne Brunet, and Gintare Karolina Dziugaite. Relatif: Identifying explanatory training examples via relative influence. arXiv preprint arXiv:2003.11630, 2020.
[31] Ryan Giordano, William Stephenson, Runjing Liu, Michael Jordan, and Tamara Broderick. A swiss ammy infinitesimal jackknife. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1139- 1147,117,222211 2019.
[32] Satoshi Hara, Atsushi Nitanda, and Takanori Maehara. Data cleansing for models trained with sgd. In Advances in Neural Information Processing Systems, pages 4213-4222, 2019.
[33] Rajiv Khanna, Been Kim, Joydeep Ghosh, and Sanmi Koyejo. Interpreting black box predictions using fisher kernels. In The 22nd International Conference on Artificial Intelligence and Statistics. pages 3382-3390, 2019.
[34] Alexander Warnecke, I Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. arXiv preprint arXiv:2 108. 1577, 2021.
[35] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In ICML, 2019.
12

---

[36] Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with polynomial time algorithm for shapley value approximation. In Interaatoonal Conference on Machine Learning, pages 272-281. PMLR, 2019.
[37] Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. Towards efficient data valuation based on the shapley value. In The 22nd Interattinnal Conference Artificial Intelligence and Statistics, pages 1167-1176. PMLR, 2019.
[38] Aleksandar Chakarov, Aditya Nori, Sriram Rajamani, Shayak Sen, and Deepak Vijaykeerthy, Debugging machine learning tasks. arXiv preprint 1rXiv:1603.07292, 2016.
[39] Lucas Bourtoule, Varun Chandrasekaran, Christopher Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. arXiv preprint a7X1v:191..03817, 2019.
[40] Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. In Advances in Neural Information Processing Systems, pages 3518-3531, 2019.
[41] Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine leaming models: Algorithms and evaluations. arXiv preprint arXiv:2002 /0077, 2020.
[42] Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajerdi Descent-to-delete: Gradient-based methods for machine unlearning, arXiv preprint 2xXx:20000000223 2020.
[43] Varun Gupta, Christopher Jung, Seth Neel, Aaron Roth, Saeed Sharifi- Maaiii-Malvajerdd. and Chris Waites. Adaptive machine unlearning. arXiv preprint arXiv:2106.04333 043788 2021.
[44] Quoc Phong Nguyen, Bryan Kian Hsiang Low, and Patrick Jaillet. Variational bayesian unlearning. arXiv preprint arXiv:2010. 12883, 2020.
[45] Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization in linear time. stat, 1050:15, 2016.
[46] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1411v:1412.6988/ 2014.
[47] Curtis G. Northeutt, Lu Jiang, and Isaac L. Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research (JAIR), 70:1373-1411 2021.
[48] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 11x:1141..6572. 2014.
[49] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
13

---

Checklist
The checklist follows the references, Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], , [No] or [N/A] You are strongly encouraged to include justification your answer: either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [No] The code is proprietary at present. It will, however, be possible to open-source the code by September this year.
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "znSxSnQAuEL", "reviewer": "Reviewer_7Beq", "summary": "The authors propose a Bayesian continual learning approach which aims to remove memories from a neural networks weights if those memories stem from data which ultimately hurts the performance of the model on some defined failure set.", "questions": "- What are the concrete improvements over [2] which has a similar deletion approach?\n\n- How does this approach compare to dataset condensation? It seems like a method of dataset condensation for a specific purpose.\n\n- What is the underlying model which is used for CIFAR10? It looks like the 'original model' in figure 3 gets 100% accuracy, but the appendix quotes using ResNet50 from [1] where the a 50 layer network should give about 7% error rate.\n\n- Were any of these results actually performed on a BNN with a distribution over the weights, or was the BNN interpretation only used for purposes of derivation?\n\n- The legends in all charts in the main text are quite hard to read because of their small size, could the size of the legends and axes be increased?\n\n- I find the notation and derivation of equations 5 and 6 to be too terse, and it took some effort to work out why they are true. Could a full derivation be included for the reader?\n\n# References\n\n[1] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).\n\n[2] Guo, C., Goldstein, T., Hannun, A., & Van Der Maaten, L. (2019). Certified data removal from machine learning models. arXiv preprint arXiv:1911.03030.\n", "limitations": "Yes, the authors have offered some limitations in Section 2. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- The method is well derived\n- The experimental results show improvements over continual learning baselines.\n\n# Weaknesses\n\n- The datasets evaluated are quite small (MNIST, CIFAR-10). Given the method, it should be possible to use a pretrained model and apply it to a larger dataset such as ImageNet, correct? If that is the case, then couldn't a larger scale evaluation be done by taking the largest errors on the ImageNet validation set and applying the method and then measuring the difference between the Failure set and the remaining images in the ImageNet test set?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_9Hpv8G5IHW", "reviewer": "Reviewer_e7Kn", "summary": "This paper proposes a bayesian technique for determining the set of points that contribute to model errors and updating models to fix performance on the failure set while retaining performance. The key issue they overcome is determining the set of points that lead to errors is computationally hard, because you'd need to check every subset. Instead, they introduce a method that determines the set of points causing the errors in linear time, wrt dataset size. After they provide a technique to unlearn the datapoints in the failure set. Finally, they show their framework enables more efficient identification of the corrupt set in empirical evaluations over baselines such as random.\n\n", "questions": "- I'm a bit confused about the notation used with $p(F|D)$... This looks like the predictive distributon on the failure set given eq. 1, but its the predictive distribution of the model itself with the failure set. Could you clarify the usage here?\n- I'm having a hard time following the Random Input Noise paragraph experiments. What is \"salt and pepper\"? It seems like ewc isn't detecting the corrupted images as efficiently as the baselines, what's going on here? \n- Also, the bar plots for accuracy in fig. 4 have large ranges making it look like the accuracies are quite close, but for ewc for mnist for example, there's a 3 point different from random, which is very large for mnist. Is this method having sigificant effects on performance?\n- Why does the test set accuracy for fig 3 start at 100%? It seems surprising these models getting 100% test set accuracy on these datasets?\n", "limitations": "The authors have done a sufficient job here.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Interesting problem & quite relevant. The connections between influence functions are quite nice.\n- Empirical evaluation is pretty comprehensive \n\nWeaknesses:\n- The notation usage is a bit dense and hard to follow in places. Often found myself having difficulty recalling what terms mean. Also, some terms seem like they might be a bit overloaded. \n- The writing is hard to follow in places, and I found myself confused about what was going on several times while reading. This was particularly true in the empirical evaluations.\n\nThis paper is interesting overall and presents a seemingly useful method, but I'm having difficulty understanding the empirical evaluations in particular in several different places and found it quite hard to follow.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RPN0yTgXyMi", "reviewer": "Reviewer_4Kxq", "summary": "The authors seek to find training points that most contribute to a model’s failures. They do so by using continual learning techniques to identify the training examples with the highest influence for model failures. They then again use continual learning to try to “forget” features learned by these training examples. ", "questions": "I had some questions on the proof/theoretical motivation in Section 2.1:\n- The i.i.d modeling assumption you make seems to be (for two disjoint portions $D, \\mathcal{C}$ of the training dataset) that $P(\\mathcal{D}, \\mathcal{C} \\mid \\theta) = P(\\mathcal{D} \\mid \\theta) P(\\mathcal{C} \\mid \\theta)$. Can you walk through why this assumption is justified, and when it might not be? ($\\mathcal{C}$ and $\\mathcal{D}$ are not from the same distribution, since $\\mathcal{C}$ are the examples that cause failures). \n- In Example 1, you walk through how your method recovers linear influences. However, linear influences require taking a gradient step on the training example (so here z) and then measuring the change in prediction of the test example (so here $\\mathcal{F}$). You seem to be doing the reverse. Are you recovering the transpose of the linear influence matrix?\n- What is $z^*$ in line 140?\n", "limitations": "The authors did not address limitations of their work, or discuss negative societal impact. (If I missed it, please point it out in the author response)", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n*Quality:* I was relatively impressed by the experimental results of the method: particularly the results on identifying mislabeled examples and data poisoning. This certainly seems like a cheaper method for isolating influential points than retraining. \n\n*Originality/Significance:* The problem is relevant to the community, and the method seems to be sufficiently novel. Additionally, the framework for \"forgetting\" the features learned by particular datapoints is interesting. \n\n## Weaknesses\n*Clarity*: Section 2.1 was not clearly written (see questions below). It may be helpful to start with why, even though $\\mathcal{F}$ is in the test set, this method requires taking gradient steps on $\\mathcal{F}$.  Providing additional intuition on EWC in the main paper would be helpful as well (this information is currently in the Appendix, but could be moved up in a compressed form).  \n\n*Quality:* What happens when you apply this to an untouched CIFAR-10 dataset? Can you identify mislabeled examples within the dataset? Does your accuracy improve, or does performing continual learning in this setting reduce accuracy if there is no planted pathology?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept