Representer Point Selection via Local Jacobian Expansion for Post-hoc Classifier Explanation of Deep Neural Networks and Ensemble Models
Authors: Yi Sui
Ga Wu
Scott Sanner
Venue: NeurIPS 2021 Poster (2021)
Keywords: Model Explanation
Sample-based Explanation
Representer Point Selection
Local Jacobian Taylor expansion
Deep Neural Network
Ensemble Models

Abstract: Explaining the influence of training data on deep neural network predictions is a critical tool for debugging models through data curation.  A recent tractable and appealing approach for this task was provided via the concept of Representer Point Selection (RPS), i.e. a method the leverages the dual form of $l_2$ regularized optimization in the last layer of the neural network to identify the contribution of training points to the prediction.  However, two key drawbacks of RPS are that they (i) lead to disagreement between the originally trained network and the RP regularized network modification and (ii) often yield a static ranking of training data for the same class, independent of the data being classified.  Inspired by the RPS approach, we propose an alternative method based on a local Jacobian Taylor expansion (LJE) of the Jacobian.
We empirically compared RPS-LJE with the original RPS-$l_2$ on image classification (with ResNet), text classification recurrent neural networks (with Bi-LSTM), and tabular classification (with XGBoost) tasks.
Quantitatively, we show that RPS-LJE slightly outperforms RPS-$l_2$ and other state-of-the-art data explanation methods by up to 3\% on a data debugging task.  Qualitatively, we observe that RPS-LJE provides individualized explanations for each test data point rather than the class-specific static ranking of points in the original approach.  Overall, RPS-LJE represents a novel approach to RPS that provides a  powerful tool for data-oriented explanation and debugging.


Representer Point Selection via Local Jacobian Expansion for Post-hoc Classifier Explanation of Deep Neural Networks and Ensemble Models
Yi Sui University of Toronto
Ga Wu Borealis AI ga. wwwborealisai.ccom
Scott Sanner University of Toronto ssannerêmie utoronto.ca
yi. pr.suiine miaiimail... wtw.unto.o..... ca
Abstract
Explaining the influence of training data machine learning model predictions is a critical tool for debugging models through data curation. recent appealing and efficient approach for this task was provided via the concept of Representer Point Selection (RPS), i.c. a method the leverages the dual form of l2 regularized optimization in the last layer of the neural network to identify the contribution training points to the prediction. However, two key drawbacks of RPS-Iz are that they (i) lead disagreement between the originally trained network and the RPS-12 regularized network modification and (ii) often yield a static ranking training data for test points in the same class, independent of the test point being classified. Inspired by the RPS-I2 approach, we propose an alternative method based a local Jacobian Taylor expansion (LJE). We empirically compared RPS-LJE with the original RPS-I2 on image classification (with ResNet), text classification recurrent neural networks (with Bi-LSTM), and tabular classification (with XGBoost) tasks. Quantitatively, we show that RPS-LJE slightly outperforms RPS-(2 and other state-of-the-art data explanation methods by up to 3% on data debugging task. More critically, we qualitatively observe that RPS-LJE provides stable and individualized explanations that are more coherent each test data point. Overall, RPS-LJE represents novel approach to RPS-I, that provides powerful tool for sample-based model explanation and debugging.
Introduction
Deep learning as well as ensemble methods such as XGBoost 22 have revolutionized the field of machine learning and led to unprecedented levels of accuracy in variety of data-driven prediction applications. However, can be extremely challenging to debug these complex methodologies when they make incorrect predictions. The first step in this debugging process is explaining the prediction, which can stem from variety of interpretive processes, For example, we may seek to understand the salient regions [18 20 that led to prediction, in more recent line of
key features 13 16
work, to understand the influence of the training data on test data predictions I I 15 23)
In this paper focus particularly appealing (and efficient) method for understanding the impact of training data on test predictions, namely Representer Point Selection (RPS) [23), which leverages an application of the representer theorem [17] to deep neural networks. Specifically, it uses the dual form of la regularized optimization in the last layer of the neural network to identify the contribution of training data the test prediction. Compared to model-agnostic approaches [ that estimate
"Contributions were made while author was the University Toronto.
Affiliate to Vector Institute of Artificial Intelligence, Toronto.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

test
tex:
Varn'e 4522
Snan 3319
1i 4275
Viain 1403
Sran 10/2899
li He 3499
Tran 1455
Expla liptaiatoon
Explination
Srwn
TraIn
126 11
Explnaation
Explanation
Exp Explanation3 3
Explnation
(a) Same test image, different /2 weight.
(b) Different test image, same 12 weight.
Figure 1: Drawbacks of Current Representer Point Selection Explanation (RPS-I2) on CIFAR- 10 Dataset. The target model is ResNet [6. (a) Explanations vary when by regularized fine-tuning of RPS-12 conducted with different hyperparameter weightings. l2 coefficient value of each column from left to right: 1e-5, le-4, le-2, le-1] (b) RPS-I2 produces identical aecricallxxppaan explanations (and rankings) for different test samples that belong to the same predicted class.
data influence after optimization, the RPS approach directly integrates with the prediction model to provide high-fidelity white-box interpretation of the prediction. Compared to the influence function-basee approach [T] the RPS is more computationally efficient as it focuses on the last layer of neural network instead of all parameters.
While holding many advantages, we note the existing RPS approach faces two key drawbacks. First, as the RPS approach requires fine-tuning the last layer of the neural network with la regularization, leads to disagreement between the originally trained network and the 12 regularized network and is furthermore highly sensitive to l2 regularization strength (cf. Figure justify the explanation produced by RPS is for the original model or the modified model. Second, we observe that the explanations produced by RPS are nearly identical for all test samples that are classified into the same category, which provides more of class-level rather than instance-level explanation for RPS (cf. Figure [b)). While the RPS approach is appealing in principle, both of these drawbacks significantly harm the utility of the RPS explanation approach in practice.
Thus, it is hard to
To address these issues, this paper presents an alternative derivation for Representer Point Selec- tion (RPS) based on a Local Jacobian Taylor expansion (L.JE), which corrects for both aforementioned problems. We empirically compared RPS-LJE to the RPS-Iz on image classification (with ResNet [6).. text classification (with Bi-LSTM), and credit analysis (with XGBoost (2] tasks. Quantitatively, we show RPS-LJE outperforms RPS-/2 and other state-of-the-arr data explanation methods by up 3% on a data debugging task. Qualitatively - and perhaps most importantly - the RPS-LJE provides stable (i.e., no need for l2 tuning) and diverse explanations that are more coherent the test data.
2 Representer Points Selection for Explaining Deep Neural Networks
2.1 Preliminaries
In machine learning context. representer theorems 17 loosely state that under certain conditions, model prediction y. of test sample 2, can be expressed linear combination of kernel evaluations X(x,,xx) between each training point and the test sample x, such that
y Σ (,x((x,))).
(1)
where a; is the weight of the training data point i that is independent from the test point t.

---

164
1.0
1.0
0.8
Tenk of (ekk..
rank of (akk0...
0.6
0.4
0.2
0.2
0.0
0.000
0.001
0.002
0.503
0.004
0.005
0.006
0.0
0.2
0.4
0.6
0.a
vector lengthh of ad(())
vector length c.(0k))
(a) RPS-12
(b) RPS-LJE
Figure 2: Correlation between the Rank of Explanations (x, ((x))) and ferm (x)(x;)' We show results for a binary classification task (horses vs. cars), where we only look at a; of the positive class. (a) RPS-I2's explanation rank heavily depends on a;((x;), which shows strong dominance of (x x.) that suppresses information the test example x, being explained. (b) The proposed RPS-LJE explanation is not dominated by ax)(x,), which instead varies widely with
As the representer theorem linearly separates the contribution training samples to the prediction,
it has been introduced into the deep learning model interpretation research (23] for tracking the
training data influence on predictions. Concretely, the current Representer Point Selection (RPS) approach [23] introduces l2 regularization term (for the last layer's parameters O C ①) into model's 's objective function, which facilities its derivation such that fulfills Equation by setting
1 ac(x, Yc, 2) and K((;;, xx) (xx;)" Tx(())
2An 0xx)
where À denotes the hyper-parameter of l2 regularization term, denotes the number of total training samples, and o(x,) and o(x;) ) are the representations of input x; and x; layer L. As this explanation approach works on the last layer of a network and is efficient to compute, it demonstrates significant advantage over Influence nunction-based approaches [[]
terms of computational efficiency.
2.2 Caveats of Current Representer Points Selection
As mentioned, to facilitate its derivation, the current RPS approach introduces an l2 regularization term into the computation, which inevitably violates the saddle point assumption its derivation relies on. To address this, the RPS conducts fine-tuning on the trained model with new objective function
+ arg min 8
C ( θ), 4 P giveen)) A||O
(3)
where ( + given) and ( 3) represents the post-activation prediction of 2; of the given model and fine-tuned model, respectively. Correspondingly. the kernel weights in the expression of Equation] are updated
1 aL(x;, Vo. θ³)
(4)
2An o(x)
As the data importance factor a; is computed with respect to the l2-modified parameter set O', the explanations correspond the modified model instead of the original model. Indeed, the gap between the two sets of parameters is sensitive to the hyper-parameter setting of the lo regularizer. This reflects our previous observation in Figure [ a) showing that prediction explanations can vary significantly based on different la regularizer weightings.
Also, by inspecting the expression of training data contribution
dL(x,, Vi, O*)
0xx()) = axK((x, = 2An 00
((x,)) ((xx,)T((xxx) (x(x))
OC(x,, ys, 0-) 000 ((x)
ac(x,, U/. ①*)
(5)
((x)) = 2An
o(x:),
2An (xx))
00;
J0;
dominamt term
3

---

we note the first-order derivative of the loss of single training example is the dominant term that suppresses a small difference of test example representation o(x,) among test samples in the same class. Figure P a) shows the strong positive correlation between the dominant term and the relevant prediction errors!
order of the training data contribution. This supports our observation in Figure Tb) that explanations tend to be identical for the test samples being classified into the same category making RPS more of a class-level than instance-level explanation that defeats the intent of uncovering instance-level
3 Representer Point Selection via Local Jacobian Expansion
In an effort to preserve the conceptual and computational advantages of the previously discussed RPS- l2 methodology while improving fidelity of the explanations to the original model and encouraging test instance-level explanation, we now present novel derivation of Representer Point Selection through a Local Jacobian Expansion (RPS-LJE). Similar to RPS-12, RPS-LJE also expresses the pre-activation prediction outcome of a well-trained classification model (near the saddle point) as a linear combination of kernel evaluations between the test sample and the training points. Specifically, we use first-order Taylor expansion on the Jacobian matrix in our derivation, which avoids the problems introduced by imposing the additional l2 regularization term in RPS-I2. First, we begin with the formal problem definition.
Consider a classifier Me. (the target model in the following context) that has learned to map input observation x, E IR to an output space y/+ E {1 ... kj, whose pre-activation predictior Me could be formulated
y. - Mei (x, = (xx))
(6)
where bold y; E IR represents the prediction vector (one element for a class). Our goal of prediction explanation is reformulate Equation 6 into the format of Equation I such that the contribution of each training data point on the prediction is linearly separable.
While the above setting appears restricted, it represents large group of machine learning models commonly used in practice; many well-known models, such as ResNet [6] for images, Transform- ers [22] for text, and even XGBoost [2] for tabular classification tasks, can be expressed in this simple formulation leveraging a feature embedding stage followed by pre-activation linear prediction stage.
3.1 First-order Taylor Expansion on Jacobian Matrix
We begin by presuming we are given a well-trained target model MMe., whose loss derivative with respect the decision making parameter O, is close to saddle point such that
0 OC(x,, Ui. +)
ae,
0.=0
Here, we assume that the loss term C is twice- difteeddfferentialle and strictly convex with respect to the last linear layer parameters O In the case of non-convexity, we adopt the approach proposed in the Influence Function 11 to form a convex quadratic approximation of the loss by introducing a damping term
With any parameter assignment 02 close to the model parameter @ further rewrite Equa- tion 7 via a first-order Taylor expansion such that
aL(xi, Vi, +)
OL X Vi.
+)
0 ~
+(0), - 01)
(8)
d0,
d0? 1
0
0
Ve, (")
Hag Mm v3, (.)
This expansion yields the model parameter O as linear combination of functions of each training data point
0 = - Σ Nop Ve, L(x,, Ui. 0²) +
(9)
For the trained classifier, the predictions before and after activation consistent.

---

where Ç is a negligible error term. Intuitively, this equation shows the model parameter O could be reconstructed through a one-step gradient descent from the nearby parameter assignment O4 with dynamic learning rate Ho (as matrix).
We remark that the expression in Equation 9 represents the contribution of each data point as linearly separable function, which is fundamentally different from recording gradients at training time as data importance score [15]. That is, the gradients during model training are sensitive to both training order and optimizer settings.
For Equation 9 to hold after the Taylor expansion, 07, has to be close to the model parameter θ Therefore, we propose to estimate o through one-step stochastic gradient ascent from the trained model using any optimizer (Adam, RMSProp, etc), such that
C(x, Vi, θ7)
Σ C(x,, θ7)
(10)
and 0% is maintained close to the original model parameter 6% with small loss shift.
3.2 Representer Point Selection with Local Jacobian Expansion (RPS-LJE)
With the derivation of Equation we now reformulate the pre-activation prediction of test point X₁ as a weighted linear combination of kernels leading to our key result of the final form of RPS-LJE (see Appendix B for more detailed derivation):
1
1 Ho: Vo, L(x,, Vi, 0*) o(xx)
2
n
OC(x; Mii θ¹)
(11)
0
(xx)) -He 0(x))())
o(x,)* (xx)))
k(x.xx))
Similar to the claim of the original RPS using an norm (RPS-12), when a training data x; is close to the test point in the representation space with a large positive value agk the prediction score for class is increased. On the other hand, when the agk is a large negative value, the prediction score for class k is then decreased.
The first notable difference between derivation of RPS-IJE and the original RPS-1z is that the Or term now contains an inverse of second-order derivative that estimates the correlation among the parameter entries in O. This modification mitigates the risk of over-weigtting training data points with large predictive error that causes small number of data points to dominate the explanations as discussed in Section 2.2 and Figure [b). Figure b) illustrates the effects of correction.
The second difference is that the prediction explanation of our derivation of RPS-LJE is faithful w.r.t. the original model MM instead of the 1 Iz-regularized model Me-. To clarify, the left hand side of of Equation| [1 is different with that RPS-(2 in Equation 5 In addition, data importance factor a; in the new derivation no longer depends on the le regularization hyper-parameter, Hence, the changes in the RPS-LJE framework directly address the problem described in Figure aa).
3.3 Relation to Influence Function-based Interpretation
The Influence Function method [T]
estimates the prediction importance of each training data point by up-weighting the data points with small perturbation that results in the following final expression
Loss (x,, xx) -e C(x, Vt, @ttT Hot L(x,, Vis θ
(12)
Ty (x.) ac
we rewrite the above equation by expanding the first-order derivatives with chain rule such that
Dup don (x; X,) = aL(x,, Dr. θ
OC(xi, Vi, O1) o(x,)* (xxx),
H-1 oo ox,,)
(13)
00 (xx))
5

---

Tedt
Ted
tact
Teci 2
paRs
Exskenation
Explanatin
Explanatiom
2
(a)
(b)
Figure Sanity Check of Representer Point Selection with Local Jacobian Expansion (RPS- IJJ) on CIFAR-10 Dataset. The target model is ResNet [6] (a) RPS-LJE can correctly produce individualized explanations. (b) Explanation maintains consistency as the gradient hyperparameter
setting changes. Learning rate of each column from left to right: [5e 4, le - 3,5e - 3, le 2].
we note the expression is very similar to RPS-LJE proposed in this work (as described in Equation 11 except two subtle differences:
The Influence Function includes multiplicative factor ((.))) that is relevant to the test ae ((x,)
point, whereas the RPS-LJE has an additive factor " 10x0(xx)) that also only relevant the test point.
The derivative in the Influence Function respects the original model parameters Ot, whereas RPS-LJE's s derivatives apply to +*
Ultimately, these differences help explain the comparative performance of these methods in experiments (cf. Section 4.3). but - because they are also subtle they help explain the similarity of explanations produced by RPS-LJE and Influence Functions for image classification explanation.
Experimental Evaluation
We perform range of experiments with multiple datasets (and corresponding model architectures) and evaluate the performance of RPS-LJE against the original RPS-I2 as well as Influence Function- based approaches. Note that all three methods do not specifically require access the training history, which is rarely available in deployment settings. In contrast, TracIn [15], which requires access to training checkpoints, is not in the scope of comparison. The goal of these experiments is to demonstrate that the alternative derivation of RPS described in this work, RPS-LJE, successfully addresses the critical drawbacks of RPS-Iz and leads substantial performance improvement multiple use cases, including data debugging and model behavior explanation. All code to reproduce these results is publicly available Github
4.1 Sanity Check of Representer Point Selection with Local Jacobian Expansion
Before descríbing our quantitative analysis, we first start with a sanity check to show the proposed RPS-LJE approach indeed addresses the problem we highlighted earlier in Section 222 and Figure] [ Concretely, we repeat our showcase in Figure 3 Jusing RPS-LJE with the same test examples and target model. Here, we highlight the following observations:
shown in Figure 3 a), the RPS-LJE can produce an individualized explanation for test samples in the same category. This observation reflects our previous description in Figure 2b), where the term (x)(x,) in RPS-LJE no longer dominates the explanation ranking.
http::///gttabb..cc/cchhy////

---

6.6
RPS-
infuurce fanck
6.91
Iniserce famction
RPS-l
36
Irflurcee
APS-S Rantom
-8-
Raedom
faniom
Randon
0.52
0.00
0.00
02
60
05
6.3
05
(a) CIFAR-I0 Image Classification with ResNet
(b) IMDB Sentiment Analysis with LSTM
Infuence function Fondon
0.70
Int urnce fanction
Rovdoo
0.75
20.16
0.7
7140201 Teaiingg dala Shockve)
Prasions Ueirog Mate shocked
(c) German Credit Risk Analysis with XGBoost
Figure 4: Performance Comparison on Dataset Debugging Tasks. For each sub-figure group, we show data correction rate vs. fraction of data checked (left) and how such correction would impact performance (right). RPS-LJE shows slightly better performance than Influence Functions and marginally better than the original RPS with l2 norm injection.
To test the sensitivity of one-step gradient descent in RPS-LJE (used to search θ as described in Section 3.1, we experimented with multiple learning rates 7 in large range 7 (5e-4 le-3, 5e-3 ... 1e-2}. As shown in Figure 3(b) b), the explanations vary little with the learning rate.
4.2
Fixing Mislabeled Training Examples (Data Debugging)
In this experiment, we simulate an application scenario where human experts need inspect the data annotation quality of the training set that directly impacts the model performance on the test samples.
We simulate data debugging on three classification tasks, including 1) binary image classification with ResNet [6] (ResNet-20) on CIFAR-10 10 12 dataset (horses vs. cars) presented the RPS- paper 223 2) sentiment analysis with Bi-LSTM (7) IMDB [14] dataset, and 3) credit risk identification with XGBoost on German Credit dataset [4] The datasets are intentionally corrupted by randomly flipping 20-30 percent of the data points' labels, naturally resulting low test accuracy. Our goal is to identify which data points' label corruption are more harmful and correct them as early as possible. With the partially corrected dataset (after each 5% of checking), we retrain the models and record the test accuracies for each task. Experiments are repeated for ten random split and corruptions on CNN and XGBoost and five random split and corruptions on RNN; we report the average result. For RPS approaches (LJE and 12),, we pick data points that have the largest self- prediction contribution Jo,,C(x,, x; as suspicious corrupted data points. For Influence Functions, we use self-influence (see Equation 12) as the score of ranking. For the random baseline, we picked the data to check randomly.
Figure 4 shows the experimental results. Here, we highlight the following observations:
RPS-LJE either slightly dominates RPS-Iz or performs comparably to it. This demonstrates that is an effective data debugging tool comparable to as RPS-12, since both show a significant performance gap the random bascline on all three tasks.
RPS-LJE shows slightly better performance than both Influence Function and RPS-I2 on all tests. For the image classification task (Figure aa), the RPS-LJE shows 3% better performance (around 60 more mislabelled data identified) than original RPS-Iz after searching through 40% of all training data. For the sentiment analysis task (Figure {b)) the performance improvement about 1% (around 35 more mislabelled data found than the others).
We note that, for RPS-12, there is difference using |as| (xx((x;;, x:) to select corrupt data points.
7

---

infuuence function
RPS-6
RPS-LJE
Insteatt
2013
Figure 5: Comparison of Top-4 Influential Training Images for Three Test Samples. For each test sample, the upper shows positive influential points, whereas the bottom shows negative influential points. Examples are randomly selected from 2000 test samples in our experiment setting.
4.3 Understand Model Behavior through Prediction Explanation
In this section, we qualitatively analyze the prediction explanation ability of candidate sample-base bassed explanation models on the three tasks mentioned in [.22 While we compare the explanation results in this experiment, only intend compare and contrast the three explanation approaches.
CIFAR-10 Image Classification with ResNet
In Figure 5 we visualize the top-4 training points (both positively and negatively) that have the strongest influence on the prediction of test samples from Influence Function, RPS-12, and the proposed RPS-LJE approaches on the image classification model. Here, we make the following key observations:
The explanation produced by RPS-LJE is more similar to that of Influence Function than RPS-Iz but with some differences in explanation order. This observation aligns with our previous conjecture in Section 3.3 that the RPS-LJE's formula of computing training data importance appears similar to that of Influence Functions.
Explanations produced by RPS-(2 contain lot of repetition. Training example ID-3291 and ID-7655 are in all three test samples' explanation lists.
IMDB Sentiment Analysis with LSTM
Table J shows the top-3 explanations produced by the candidate explanation approaches on the IMDB sentiment analysis model (Bi-LSTM). Here, we make the following key observations:
Among the three explanation approaches, RPS-LJE's explanation more coherent to the properties of the points. For instance, the explanations of RPS-LJE for Sample all start with "This", which hints to the auditor that the model has made a generalization (whether correct or not) that positive sentiment sentences start with "This" based on the identified training data. Similarly, for Sample 1 and 3, we see the explanations follow the same narrative style by starting with "T". Explanations provided by Influence Functions generate similar explanations out of the top 3) with RPS. But Influence Functions sometimes produce confusing explanations. For example, Sample 2 has a positive comment, but Explanation provided by the Influence Function has negative sentiment. Similarly, Sample 3 and its Explanation 3 have the opposite sentiment.
German Credit Risk Analysis with XGBoost
Table lists the predictions of three test samples the German Credit dataset through the XGBoost model with corresponding explanations produced by RPS-LJE and Influence Function. Here,

---

Table 1: Explanation Comparison among RPS-LJE, RPS-12 and Influence Function on LMDB sentiment analysis data, Column "Sentiment" is the target (label) column, and raw review is the input of Bi-LSTM network. Examples are randomly selected from the test set.
Senriment
Raw Reviews
Sample
Test point
41
posiive
This movie is good for entertainment purposes, but it is not historically reliable. If you are look...
Explanation
741
positive
This movie about sexual obsession Bette Davis plays Mildred, This woman who men
RPS-LJE
Explanation
14701
posirive
This is very memorable spagheni western It grent storyline, interesting characters,and some-
Explanztion
3159
positive
This movie was featured on a very early episode of Mystery Science Thester 3000) when I see-
Explanation
9112
positive
Tim Krabbe the praised suthor of Het Gouden Ei" novel that was put the twice-.
RPS-12
Explanation
3704
positive
THE DEVIL'S PLAYHHNG my second attempt a Joseph Samo production although
Explanation
4000
positive
So, Todd Sheets once stated that he considers his 1993, shot on vidoo 2 epic. Zombie Bloodbath-
Influence function
Explanation
14701
posirive
This is very memorable spaghetti western It has grent storyline, interesting characters and some-
Explanation
741
posirive
This movie shout sexual oheession Rete Davis plays Mildred. This woman who
Explanation
669
positive
Did you ever wonder how far one movie could go ""Schizoohrennia relenlessly explores the world-
Sample
Text point
525
positive
can think of movie that better captures the concept of grace, in thoological sense The well
Explanation
14109
posirive
tell you just how bad movic was the movic and I haven's It sean it yet.hu cringe at
RPS-LIE
Explanation 2
2300
positive
am one of Jehowah's Witnesses and I also week in an acste care medical facility:Over the years
Explanation 3
6372
posirive
think the rilm is ediacitional However fails to treat the issue which sparked so much controvercy...
Explanation
9112
positive
Tim Krabbe the praised author of 'et Gooden
novel was put on the screen twice...
RPS-12
Explanation
3704
positive
THE DEVIL'S ' PLAYTHING is my second attempt a Joseph Samo production and although I
Explanation 3
4000
posirive
So, Todkl Sheets once stated that he considers his 1993, shoo--nn Z epic, Zombic Bloodbath-
Influence function
Explanation
negative
A memorable line from shoet lived After viewing the episode where that line introduced
Explanation
14109
positive
can tell you just bow bad this movie is was the movie and I haven't seen it yet.but I cringe al-
Explanation 3
2300
positive
am one of Jehovah's Winnesses and I also work in an acuse care medical facility:Over the years I ...
Sample
Test point
13087
negative
this film about doctor who the possibility new I'm sorry but dicn't like this doe very much I can think of a million ways it could have heen hetter-
really
a young...
Explanation
3064
negative
RPS-IJE
Explanation 2
4622
negarive
have be completely honest in saying first that I fell ssleep somewhere in the middle, so can-
Explanation
9777
negative
roceedl viewed Manufactured Landscapes the Scatle International Film Festvaal was drawn-
Explanation
4801
negative
A so common horror story about a luxury building at Brooklyn which hides the gates to hell. It is
RPS-1
Explanation 2
11015
negarive
The thing that stands out in mind in this nim ( sadly is the introductiion where John Berlin-
Explanation
12446
negative
grade 11, watched fast forward. I sugest you I'm sorry but didn't it like this doe very much I can think of a million ways could have been better
Taped this night movic when
Influence Function
Explanation
3064
negative
Explanation 2
4622
negative
have to be completely honest in saying first that fell asleop somewhere in the middle, so I can
Explanation
16805
posicive
Afler viewing several episodes of this series,] have come the conclusion that television producers-
removed RPS-12 from the candidate list as it requires fine-tuning the model with 12 normalization, which is incompatible with tree ensemble models. Here, we highlight the following observations:
The explanation produced by RPS-LJE is more similar to the test point than that of Influence Funetion the sense of sharing similar feature values. For instance, the "Checking Account" values of the RPS-LJE explanation always align with the test samples, but that of the Influence Function does not.
Influence Function tends to provide diverse explanations for each sample case, where the explana- tions produced by it show more or fewer differences.
5 Conclusion and Discussion
We presented an approach for explaining the impact of training data on a test prediction, called Representer Point Selection via Local Jacobian Expansion (RPS-LJE). Our approach aimed to correct two drawbacks of the existing Representer Point approach (RPS-12) 123,, namely that it often I) produces identical explanations for different instances in the same class and 2) produces highly varying explanations based on the strength of an le regularization modification to the original model. We began by observing these problems empirically and then analyzing the RPS-I2 derivation to reveal the technical source of these problems. We then proposed corrections to derive a novel form of RPS based on a local Jacobian Taylor expansion that addresses the technical limitations of the RPS-12.
We conducted multiple experiments that quantitatively and qualitatively analyzed the proposed RPS- LJE against existing state-of-the- approaches, RPS-12 and Influence Function. Our experiments empirically show that the proposed RPS-LJE fulfilled our expectation in terms of correcting RPS-
9

---

Table 2: Explanation Comparison between RPS-LJE and Influence Function on German Credit Data. Column "Risk" " is the target (label) column, and all columns after it are feature columns. Examples are randomly selected from the test set.
TD
Risk
Cheoking Account
Crudit History
Savings Account
Other Debiers
Fmpployment
Nample
point
Jow
critual secours! credits existing
linle
to
Explanation
210
Jow
eritcal sccouns other crediis existing
unknown/none
I 4
RPS-LIE
Explanation
526
low
ericcal sccours other credis existing
moderate
to-
Explanation
204
low
crincal sccours/ other cradis existing
unknewn/none
than
Explanation
668
high
eritcal sccouns other credits existngg
little
co-pplicaant
mure than'
Inflsence Function
Explanation
747
high
credit paid back daiy fill now
litll
loss than year
Explanation
611
high
modienate
credits poid back duly
mederats
mure than
Sample
point
hiph
credits
Explanation
high
credits paid
wiknomnfnoae
than than unzmployed
RPS-LIE
Explanation
credits
uiknonn/none
Explanation
credits
Explanation
828
high
exisiing credit paid back dely fill now
unknewm/none
mure than In than
Inflsence Function
Feplanation
152
high
little
existing credis poid back duly till
littl
Explanation
796
high
existing credits back duly
unknown/none
Sample
point
high
poor
paying
past
linte:
to years
Explanation
174
high
paylag past
littl
than your
RPS LIE
Explanation
high
paysng past
littl
than your
Explanation
high
exising
paid
litll
than year
Esplanation Explanation
high
paying past
year
Inflccnce Function
high
paying past
linle
unenployed
Explanation
low
sone
exisiing credit paid duly unknown none
nome
than year
1z's 's problems. produces individualized explanations instead of a class-level explanation and quantitatively performs comparably to, outperforms, existing data-explanation approaches.
Scope of Application
This paper presents novel method of sample-based model explanation for classifiers including deep neural networks and ensemble models. As stated in Section RPS-LJE is applicable to classification models with linear last layer before the activation function. While this requirement may seem restrictive, many widely-adopted classifiers satisfy them.
we state our assumption that the given model is well-trained (near the saddle point), and thus the gradient of loss with respect to the parameters is close to zero. RPS-12 (23 and Influence Function [1I] also makes the same assumption. The assumption often holds in practice, as many
In Section 3.1
classification models will be trained to covverennee before deployment. However, one exception to this assumption would be when early stopping is used as a form of regularization.
7 Broader Impacts
Deep learning well as ensemble methods such as XGBoost have shown exceptional performance in a variety of data-driven prediction applications including image classification, sentiment classification, and risk classification (cf. Section 4 This paper proposes methodology to explain these complex classifiers via estimating the influence from each training datum on model predictions.
Sample-based explanation methods are critically important for validating and improving classifiers. For example, explanations can increase the interpretability and transparency of a model's 's decision making process, and thus help to assess the faimess of the model. Furthermore, monitoring training data quality can facilitate the debugging process and thus improve model performance. One potential negative impact is that the use of training data for explanations may raise privacy concerns in some situations; however, measures for data anonymization may (partially) help mitigate such issucs.
Acknowledgments and Disclosure of Funding
Yi Sui was funded by a University of Toronto Dean's 's Spark award Scott Sanner. Ga Wu was funded by a Canadian NSERC Discovery Grant award to Scott Sanner.
10

---

References
[1] Bien and Tibshirani. Prototype selection for interpretable classification. The Annals of Applied Statistics, pages 2403-2424, 2011.
[2] T. Chen, T. He, M. Benesty, V. Khotilovich, Y. Tang, H. Cho, et al. Xgboost: extreme gradient boosting. R package version 0.4-2, 1(4), 2015.
[3] P. Dabkowski and Y. Gal. Real time image saliency for black box classifiers. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 18, 2017.
[4] D. Dua and C. Graff. UCI machine learning repository. 2017.
[5] R. C. Fong and A. Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[6] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[7] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 1997.
[8] R. Khanna, B. Kim, J. Ghosh, and O. Koyejo. Interpreting black box predictions using fisher kernels. In AISTATS, 2019.
[9] B. Kim, O. Koyejo, R. Khanna, al. Examples are not enough, learn to criticize! criticism for interpretability. In NIPS, pages 2280-228888 2016.
[10] B. Kim, C. Rudin, and J. A. Shah. The bayesian case model: A generative approach for case-based reasoning and prototype classification. In Advances in neural information processing systems, pages 1952-1960, 2014.
[11] W. Koh and P. Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML'17, page 1894, JMLR.org, 2017.
[12] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[13] S. M. Lundberg and S.-I. Lee. A unified approach interpreting model predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Curran Associates,
Advances Neural Information Processing Systems 30, pages Inc., 2017.
[14] A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng. and C. Potts. Learning word vectors for sentiment analysis.
[15] G. Pruthi, F. Liu, M. Sundararajan, and S. Kale. Estimating training data influence by tracing gradient descent, 2020.
[16] M. T. Ribeiro, Singh, and Guestrin. "why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135-1144, 2016.
[17] B. Schölkopf, R. Herbrich, and A. J. Smola. A generalized representer theorem. COLT "01/EuroCOLT '01, page 416-426, Berlin, Heidelberg, 2001. Springer-Verlag.
[18] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. International Journal of Computer Vision, 128(2):336-32) 359, Oct 2019.
[19] B. Sharchilev, Y. Ustinovsky, P. Serdyukov, and M. de Rijke. Finding influential training samples for gradient boosted decision trees. In ICMI., 2018.
[20] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps, 2014.
[21] M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML'17, 2017.
[22] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, u. Kaiser, and
I. Polosukhin. Attention is all you need. In Proceedings of the 3 International Conference on Neural Information Processing Systems, NIPS' 17, page 6000-6010, 2017.
11

---

[23] C.-K. Yeh, J. S. Kim, I. E. Yen, and P. Ravikumar. Representer point selection for explaining
deep neural networks. NIPS' 18, 2018.
12


Reviews:
[{"review_id": "viEtBOOW4G", "reviewer": "Reviewer_Tvmz", "summary": "This paper addresses sample based explainable AI, where a model prediction at test time is explained by producing a ranked list of training set instances.  They build upon a method called Representer Point Selection, which develops a representer theorem wherein contribution to test loss is decomposed into terms involving the features of the last layer the trained network.  The authors address two shortcomings of the method by developing  a local Jacobean Taylor expansion that alleviates both.  They present experiments on data set cleaning (in line with the original RPS paper), as well as on three other tasks commonly evaluated with sample based explainability.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors take a narrow view of explainable AI for error analysis through dataset inspection.  However, they present their case for representer point selection through local Jacobian expansion (RPS-LJE) quite clearly.  They address the following two deficiencies of RPS.  First, RPS as originally proposed requires fine-tuning of the last layer network weights with $\\ell_2$ regularization optimized via L-BFGS to drive the magnitude of the gradient to vanishing, creating a discrepancy between the original model making predictions, and the resultant model providing a link between training points and test predictions.  Second, they provide evidence that most high ranked training set samples are identical for test samples sharing the same class, which limits its utility for explaining individual predictions.  \n\nTheir main technical contribution is in reformulating the model prediction $\\hat{\\mathbf{y}_t} = \\Theta\\_{L}^{m}\\phi{\\mathbf{x}_t}$ so that it is linear in the contribution of the training points but without introducing $\\lambda$ or leaving it dominated by the contribution of the gradient of the loss wrt the modified parameter set, laid out in sections 3.1, 3.2 and especially equation 11.\n\nA few comments:\n\n1. Equation 11 contains the inverse of a Hessian matrix evaluated at a nearby set of parameters $\\Theta^{*}_{L}$.  Even for small sets of parameters, the Hessian can be difficult to estimate and may require the addition of a damping term on the diagonal to ensure it is positive definite (the original Influence Functions paper by Koh and Liang encounter this).  Did the authors encounter any practical necessity to ensure the Hessian is positive definite?\n\n2.  The qualitative experiments in section 4.3 are somewhat underwhelming.  It is not clear by my reading why the highlighted examples in either Credit Risk analysis or sentiment analysis with LSTM convince me of the method's validity.  They do establish that the method can work on models beyond convolutional (or feed forward) networks, but perhaps a more convincing set of experiments with a more relatable outcome would help?  For example, in the case of sentiment analysis, this could take the form of generating adversarially perturbed training data to augment the training set, and plotting the rank distribution of the adversarially altered negative control training data against the top ranked bona fide training data.  \n\nFinally, this paper would be more immediately useful if it released code to implement RPS-LJE in either TensorFlow or PyTorch (as the original RPS paper did).\n\nOverall, I found this a very clearly written paper that presented a solid if narrow contribution, and should be a clear accept.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No code provided, but no other relevant comments here.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "maE2szfnep0", "reviewer": "Reviewer_M1Vv", "summary": "The paper proposes a new representer decomposition based on the approximate optimal condition. This address issues for RPS-L2 by 1. this does not rely on regularization strength 2. this provides an explanation that varies much more than RPS-L2 across different testing points. They verify this in their experiments.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper aims to address issues of RPS-l2, but the key point of RPS-l2 is that the prediction of the test point can be decomposed by training point contribution. The same cannot be said for RPS-LJE as (7) is often not realistic in practice. (7) not realistic will result in (11) being not accurate. In RPS-l2, the prediction of the test point and the sum of training point contribution has >0.95 correlation. I suspect RPS_LJE will get a correlation much lower. Therefore, I view this as a trade-off between the accuracy of the decomposition and the difference between original model and fine-tuned model. Moreover, the issues of RPS-l2 can be alleviated by changing the lambda parameter, as when lambda is large, alpha_i will not dominate. When lambda is small, the difference between original model and fine-tuned model will be small. When you don't fine-tune the model until exact convergence with large lambda, then both issues may go away (with the additional downside that the decomposition is not accurate). However, the proposed RPS_LJE is also not accurate in decomposition. Without a better empirical study of the trade-off for RPS_LJE and RPS_l2 with different strategies, the advantage of RPS_LJE is unclear.\n\nHowever, the main issue is that the method seems in spirit the same as the idea of influence function. I am confident that if you replace the  theta^* by original theta in (11), you will get almost the same explanation result (this should be a control experiment). After the replacement, (11) and (13) are almost identical. Since the inverse hessian term is the same, and thus the calculation cost of influence function carries over. Therefore, I cannot imagine any use cases where RPS_LJE will be chosen over the influence function. Based on this, I really feel the paper has limited novelty and contribution. (in fig 4, one image out of 16 images of proposed method and influence function is different).\n\nTo improve the quality of the paper, the authors should 1. evaluate how well F(x) correlates with \\sum_i ai K(x,x_i) 2. compare RPS_LJE with modification of RPS_L2 with different alpha and different optimization strategy 3. point out use cases where RPS_LJE may be more useful than the influence function.\n\nFrom a practical viewpoint, point 3 is especially important. If the authors cannot do the third point, the paper would be better as a decomposition viewpoint of the influence function (if point 1 is achieved), but not a new method. If the authors can provide more (convincing) use cases in practice when RPS-LJE is better than the influence function, I will consider raising my score. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "lnNDzwjA_AW", "reviewer": "Reviewer_3a5c", "summary": "The aim of this paper is to identify the contribution of the training data to the prediction. This work highlights the short comings of the existing Represented Point Selection (RPS-$l_2$) both empirically and by analyzing the derivation of RPS$-l_2$, and proposes an improved method called RPS-LJE. RPS-LJE is a clear improvement over RPS-$-l_2$ both qualitatively and quantitatively. Qualitatively, it is observed that the explanations produced by RPS-LJE isn't repetitive like RPS$-l_2$. Quantitatively, it performs comparably to, or better that the existing data-explanation approaches.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The approach presented in this work to address the shortcomings of RPS-$l_2$ is definitely novel\n\nPlease consider citing the following saliency methods for model explanation  -\nhttps://arxiv.org/pdf/2006.16322.pdf\nhttps://arxiv.org/abs/1703.01365\n\nQuality:\nThe work is technically sound. The derivation of the RPS-LJE is pretty neat and intuitive. The effectiveness of the algorithm is further demonstrated empirically by the experiments where we see both qualitative and quantitative improvements.\n\nClarity: This work is well written and well organized.\n\nSignificance:\nThe results are significant.\n\n----------------------\nPost Rebuttal Comments:\nAfter going through the rebuttal I still believe that the paper is an accept, though not a *clear accept*, and hence, I'm updating my score to **7**. \n\nSome comments for a future version of the paper:\n\nI agree with DC2m that the (mean) test scores as reported aren't meaningful enough to make a quantitative argument that RPS-LJE is better than the baselines. However, the authors' response seems valid as well, i.e., the std is primarily influenced by the randomness of data corruptions across multiple runs. In such a case, the authors could report the fraction of examples for which RPS-LJE achieves the best score. Comparing the mean and std of this metric would be more meaningful.\n\nAlso, do consider citing: [Scaling Symbolic Methods using Gradients for Neural Model Explanation](https://arxiv.org/pdf/2006.16322.pdf)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Nil.", "review_text": "", "extra_scores": {"ethical_concerns": "Nil.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hour"}, "extra_text": {}}, {"review_id": "FswFW4Kix-L", "reviewer": "Reviewer_DC2m", "summary": "This paper addresses the problem of model explainability by looking at which training examples were responsible for the model’s predictions. The contributed model takes inspiration from RPS and changes its computation of the data importance factor such that it becomes less sensitive to the choice of hyper-parameters and leads to higher diversity in the explanations. The proposed RPS-LJE is compared against two baseline approaches from the state of the art.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** \n\nThe proposed approach seems of moderate originality. It builds on top of RPS and changes the way it computes data importance factors. In particular, the proposed computation of the data importance term is rewritten as a gradient step with a dynamic learning rate. The resulting formulation becomes, as acknowledged in the paper, similar to the influence function method. Moreover, there is no related work section. While this could be ok, the related work covered in the submission appears to be rather limited. Positioning the contribution wrt a larger body of literature would strengthen the contribution. A few works that the authors might want to consider:\n- https://arxiv.org/pdf/1810.10118.pdf\n- https://arxiv.org/abs/1802.06640 \n\nand with a different flavor: \n- https://arxiv.org/abs/1412.6806\n- https://arxiv.org/abs/1704.03296\n- https://arxiv.org/abs/1910.08485\n- https://proceedings.neurips.cc/paper/2017/file/0060ef47b12160b9198302ebdb144dcf-Paper.pdf \n\n**Quality and significance:**\n\nThe submission appears technically sound. The proposed approach is tested on three data debugging tasks (1 model per dataset) and compared against 2 baselines from the state of the art (RPS and influence function) and a random baseline. The reviewer's main concerns are wrt the experimental validation of the proposed approach and its motivation. See below.\n\n- The choice of debugging tasks is unclear. Why train on a binary CIFAR-10 version (cars vs horses) as opposed to the standard `multi-class setup?\n- The choice of methods to compare against is unclear and seems rather limited. Why limit the comparison to RPS and influence function? Why not consider [11] as well? Why is RPS removed from the German Credit Risk Analysis task? Why not use a model which is compatible with the RPS baseline?\n- It appears that the experimental setup used in the paper is different than the one in the RPS paper. The paper considers corrupting 20-30% of the data, whereas 40% is corrupted in RPS. Why not stick to 40%?\n- The paper claims that the proposed approach leads to more diverse explanations than RPS and shows this qualitatively. Would it be possible to quantify the diversity achieved by the proposed approach? e.g. by computing the % of explanations that are different from each other among the retrieved ones.\n- The paper claims that the proposed approach leads to explanations that are faithful with the original model; however, the statement is only vaguely backed up. Could the authors quantify the agreement between original model output and the updated one?  (e.g. correlation) and compare the agreement with the one of other methods?\n- Results are averaged across different runs but only the mean appears on the plots. How do the std look like? Are the observed improvements significant?\n- What is the effect of considering learning rates even bigger than 1e-2? or in other words, how big does the learning rate have to be to start observing differences in the selected explanations?\n- If we consider two images from the same class with similar appearances, do they lead to the same explanations?\n- It seems that a natural baseline to the data debugging task would be to use the entropy at the output of a model to decide which data points to correct.\n- Some experimental details are missing e.g.: which ResNet version is used? How is the RPS lambda chosen when making comparisons? How is the learning rate required by the proposed approach chosen?\n\n\n\n\n- The paper does not motivate why explainability approaches which rely on understanding the influence of the training data on the test predictions are important.\n- The paper does not motivate why explaining the predictions of a model is important.\n- It is not clear why diversity in the explanations would be a desirable feature to improve the trustworthiness of a system. Could the authors elaborate on this?\n- Could the authors comment on the use cases of the proposed approach? If the system fails, what happens? How can we mitigate this?\n- What conclusions can we draw about the trustworthiness of the models using the proposed approach? What do we learn from those models that we couldn't learn from previous art?\n\n**Clarity:**\n\nOverall the paper is well written and easy to follow. Please find below some minor suggestions to improve its presentation.\n- Figure 1: Include values of l2 weight. Does the observation about diversity hold for different l2 weights?\n- Figure 2: Include details such as model used.\n- Figure 3(b): Include choices of learning rates.\n- Figure 4 contains a lot of plots. When discussing the figure, it would be good to point the reader towards each specific subplot from which conclusions are being drawn.\n- Figure 5: It could be interesting to share associated importance values.\n- The model notation introduced in Section 3 seems unnecessary.\n- Introduce n (line 58).\n- What is CNN in line 189 (the ResNet model?), same with RNN (the previously introduced LSTM?), it would be better if the authors could stick to the introduced names\n- Although intuitive, random baseline should be explained.\n- Appendix is rather packed, it would be helpful to refer to different relevant sections of the appendix in the main body of the paper.\n\n**Comments about the checklist:**\n- The code is not shared. Please revisit 3(a) and 4(c) and indicate that the code *will be* shared.\n- Please review 4(b) and 4(d). Even if datasets are publicly available, they might come with their licenses, and unfortunately it is not always the case that consent was gathered from people.\n\n**Post-rebuttal comments:**\n\nThe reviewer would like to thank the authors for their rebuttal which addressed a number of the questions raised in the reviews. After careful consideration and discussion with other reviewers, the reviewer agrees that the goal of the paper is sounds and that the proposed approach has some edge of novelty. Bridging IF and RPS-LJE may be of interest, although as acknowledged in the rebuttal, the benefits of the diversity achieved by the proposed approach remain unclear. \n\n\nThe remaining main concern lies in the execution, and in particular the experimental evidence to support the claims and to ensure robustness of the conclusions of the paper. Unfortunately, the rebuttal did not take the opportunity to discuss the statistical significance of the results. The reviewer agrees that the variance of all methods might come from the randomness of data corruptions, but still thinks it would be worth understanding how robust each approach might be wrt this randomness. Moreover, the authors chose a different experimental setup than the one in previous art. While this might be ok, the decision still seems arbitrary. As suggested in the rebuttal, increasing the % of corruption might make differences between the random baseline and other approaches smaller, but considering different % could perhaps be useful to paint the full picture and potentially lead to more insightful discussion of the results. Beyond these points, unfortunately the rebuttal did not take the opportunity to discuss potential use cases/failure modes of the proposed approach, very briefly surfaced some of the limitations pointed by the reviewers, and did not comment on the conclusions that can be drawn from this research. \n\nThose are the main reasons why the reviewer remains on the fence and decides to keep their original score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper discusses the broader impact of their work in the appendix. Although the checklist indicates that the limitations are discussed in the appendix, the reviewer could not find this discussion. The reviewer would strongly encourage the authors to move the broader impact to the main body of the paper. The reviewers would strongly encourage the authors to include a limitations discussion, also in the main body of the paper. This is crucial information that should not be disregarded.\n\nWhat are the limitations of the proposed approach? It would seem that one of the limitations is that it would require storing all training data. Could the authors comment on that? Could we consider storing a subset of the training data that would be sufficient to provide explanations? How could we select the data points to consider?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a variant of RPS called RPS-LJE that is claimed to offer several advantages over the previously proposed RPS-L2. The “explanation” acts on the last layer of deep nets only, atributing predictions to training points while treating the neural network as fixed. As such, it seems unreasonable to call this (or any related works that act similarly) explanatinos of “deep neural networks” vs of linear models wiith some fixed basis. The conversation centered around several claimed advantages — efficiency (it appears to be resolved in discussion that there is no efficiency gain over influence functions), a faithfulness advantage over RPS-L2, and the striking similarity and high agreement with influence function, raising the question of whether the method is really adding anything or if the novelty is all in the new perspective/derivation. 

The rebuttal and discussion were thoughtful and all reviewers were engaged. However, while they converged on several points of fact, a few others were left unresolved and their were some genuine normative disagreements about whether the contributions that withstood the test of the debate were sufficient to warrant acceptance. 

I will not hide the fact that this is a difficult metareview to write and that I am not sure how to resolve the remaining impasse. I am still seeking additional opinions and hope to make a more confident assessment but believe this is a genuinely difficult case. My current best assessment now is that this paper is truly borderline. However, I believe that with reasonable adjustments to claims and presentation per promises made in the discussion period, this paper is honest work and we can leave it to posterity (vs the review process) to determine the work's eventual impact. 
Decision: Accept (Poster)