Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper addresses sample based explainable AI, where a model prediction at test time is explained by producing a ranked list of training set instances.  They build upon a method called Representer Point Selection, which develops a representer theorem wherein contribution to test loss is decomposed into terms involving the features of the last layer the trained network.  The authors address two shortcomings of the method by developing  a local Jacobean Taylor expansion that alleviates both.  They present experiments on data set cleaning (in line with the original RPS paper), as well as on three other tasks commonly evaluated with sample based explainability.
main_review: The authors take a narrow view of explainable AI for error analysis through dataset inspection.  However, they present their case for representer point selection through local Jacobian expansion (RPS-LJE) quite clearly.  They address the following two deficiencies of RPS.  First, RPS as originally proposed requires fine-tuning of the last layer network weights with $\ell_2$ regularization optimized via L-BFGS to drive the magnitude of the gradient to vanishing, creating a discrepancy between the original model making predictions, and the resultant model providing a link between training points and test predictions.  Second, they provide evidence that most high ranked training set samples are identical for test samples sharing the same class, which limits its utility for explaining individual predictions.  

Their main technical contribution is in reformulating the model prediction $\hat{\mathbf{y}_t} = \Theta\_{L}^{m}\phi{\mathbf{x}_t}$ so that it is linear in the contribution of the training points but without introducing $\lambda$ or leaving it dominated by the contribution of the gradient of the loss wrt the modified parameter set, laid out in sections 3.1, 3.2 and especially equation 11.

A few comments:

1. Equation 11 contains the inverse of a Hessian matrix evaluated at a nearby set of parameters $\Theta^{*}_{L}$.  Even for small sets of parameters, the Hessian can be difficult to estimate and may require the addition of a damping term on the diagonal to ensure it is positive definite (the original Influence Functions paper by Koh and Liang encounter this).  Did the authors encounter any practical necessity to ensure the Hessian is positive definite?

2.  The qualitative experiments in section 4.3 are somewhat underwhelming.  It is not clear by my reading why the highlighted examples in either Credit Risk analysis or sentiment analysis with LSTM convince me of the method's validity.  They do establish that the method can work on models beyond convolutional (or feed forward) networks, but perhaps a more convincing set of experiments with a more relatable outcome would help?  For example, in the case of sentiment analysis, this could take the form of generating adversarially perturbed training data to augment the training set, and plotting the rank distribution of the adversarially altered negative control training data against the top ranked bona fide training data.  

Finally, this paper would be more immediately useful if it released code to implement RPS-LJE in either TensorFlow or PyTorch (as the original RPS paper did).

Overall, I found this a very clearly written paper that presented a solid if narrow contribution, and should be a clear accept.


limitations_and_societal_impact: No code provided, but no other relevant comments here.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a new representer decomposition based on the approximate optimal condition. This address issues for RPS-L2 by 1. this does not rely on regularization strength 2. this provides an explanation that varies much more than RPS-L2 across different testing points. They verify this in their experiments.
main_review: The paper aims to address issues of RPS-l2, but the key point of RPS-l2 is that the prediction of the test point can be decomposed by training point contribution. The same cannot be said for RPS-LJE as (7) is often not realistic in practice. (7) not realistic will result in (11) being not accurate. In RPS-l2, the prediction of the test point and the sum of training point contribution has >0.95 correlation. I suspect RPS_LJE will get a correlation much lower. Therefore, I view this as a trade-off between the accuracy of the decomposition and the difference between original model and fine-tuned model. Moreover, the issues of RPS-l2 can be alleviated by changing the lambda parameter, as when lambda is large, alpha_i will not dominate. When lambda is small, the difference between original model and fine-tuned model will be small. When you don't fine-tune the model until exact convergence with large lambda, then both issues may go away (with the additional downside that the decomposition is not accurate). However, the proposed RPS_LJE is also not accurate in decomposition. Without a better empirical study of the trade-off for RPS_LJE and RPS_l2 with different strategies, the advantage of RPS_LJE is unclear.

However, the main issue is that the method seems in spirit the same as the idea of influence function. I am confident that if you replace the  theta^* by original theta in (11), you will get almost the same explanation result (this should be a control experiment). After the replacement, (11) and (13) are almost identical. Since the inverse hessian term is the same, and thus the calculation cost of influence function carries over. Therefore, I cannot imagine any use cases where RPS_LJE will be chosen over the influence function. Based on this, I really feel the paper has limited novelty and contribution. (in fig 4, one image out of 16 images of proposed method and influence function is different).

To improve the quality of the paper, the authors should 1. evaluate how well F(x) correlates with \sum_i ai K(x,x_i) 2. compare RPS_LJE with modification of RPS_L2 with different alpha and different optimization strategy 3. point out use cases where RPS_LJE may be more useful than the influence function.

From a practical viewpoint, point 3 is especially important. If the authors cannot do the third point, the paper would be better as a decomposition viewpoint of the influence function (if point 1 is achieved), but not a new method. If the authors can provide more (convincing) use cases in practice when RPS-LJE is better than the influence function, I will consider raising my score. 
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper addresses the problem of model explainability by looking at which training examples were responsible for the model’s predictions. The contributed model takes inspiration from RPS and changes its computation of the data importance factor such that it becomes less sensitive to the choice of hyper-parameters and leads to higher diversity in the explanations. The proposed RPS-LJE is compared against two baseline approaches from the state of the art.
main_review: **Originality:** 

The proposed approach seems of moderate originality. It builds on top of RPS and changes the way it computes data importance factors. In particular, the proposed computation of the data importance term is rewritten as a gradient step with a dynamic learning rate. The resulting formulation becomes, as acknowledged in the paper, similar to the influence function method. Moreover, there is no related work section. While this could be ok, the related work covered in the submission appears to be rather limited. Positioning the contribution wrt a larger body of literature would strengthen the contribution. A few works that the authors might want to consider:
- https://arxiv.org/pdf/1810.10118.pdf
- https://arxiv.org/abs/1802.06640 

and with a different flavor: 
- https://arxiv.org/abs/1412.6806
- https://arxiv.org/abs/1704.03296
- https://arxiv.org/abs/1910.08485
- https://proceedings.neurips.cc/paper/2017/file/0060ef47b12160b9198302ebdb144dcf-Paper.pdf 

**Quality and significance:**

The submission appears technically sound. The proposed approach is tested on three data debugging tasks (1 model per dataset) and compared against 2 baselines from the state of the art (RPS and influence function) and a random baseline. The reviewer's main concerns are wrt the experimental validation of the proposed approach and its motivation. See below.

- The choice of debugging tasks is unclear. Why train on a binary CIFAR-10 version (cars vs horses) as opposed to the standard `multi-class setup?
- The choice of methods to compare against is unclear and seems rather limited. Why limit the comparison to RPS and influence function? Why not consider [11] as well? Why is RPS removed from the German Credit Risk Analysis task? Why not use a model which is compatible with the RPS baseline?
- It appears that the experimental setup used in the paper is different than the one in the RPS paper. The paper considers corrupting 20-30% of the data, whereas 40% is corrupted in RPS. Why not stick to 40%?
- The paper claims that the proposed approach leads to more diverse explanations than RPS and shows this qualitatively. Would it be possible to quantify the diversity achieved by the proposed approach? e.g. by computing the % of explanations that are different from each other among the retrieved ones.
- The paper claims that the proposed approach leads to explanations that are faithful with the original model; however, the statement is only vaguely backed up. Could the authors quantify the agreement between original model output and the updated one?  (e.g. correlation) and compare the agreement with the one of other methods?
- Results are averaged across different runs but only the mean appears on the plots. How do the std look like? Are the observed improvements significant?
- What is the effect of considering learning rates even bigger than 1e-2? or in other words, how big does the learning rate have to be to start observing differences in the selected explanations?
- If we consider two images from the same class with similar appearances, do they lead to the same explanations?
- It seems that a natural baseline to the data debugging task would be to use the entropy at the output of a model to decide which data points to correct.
- Some experimental details are missing e.g.: which ResNet version is used? How is the RPS lambda chosen when making comparisons? How is the learning rate required by the proposed approach chosen?




- The paper does not motivate why explainability approaches which rely on understanding the influence of the training data on the test predictions are important.
- The paper does not motivate why explaining the predictions of a model is important.
- It is not clear why diversity in the explanations would be a desirable feature to improve the trustworthiness of a system. Could the authors elaborate on this?
- Could the authors comment on the use cases of the proposed approach? If the system fails, what happens? How can we mitigate this?
- What conclusions can we draw about the trustworthiness of the models using the proposed approach? What do we learn from those models that we couldn't learn from previous art?

**Clarity:**

Overall the paper is well written and easy to follow. Please find below some minor suggestions to improve its presentation.
- Figure 1: Include values of l2 weight. Does the observation about diversity hold for different l2 weights?
- Figure 2: Include details such as model used.
- Figure 3(b): Include choices of learning rates.
- Figure 4 contains a lot of plots. When discussing the figure, it would be good to point the reader towards each specific subplot from which conclusions are being drawn.
- Figure 5: It could be interesting to share associated importance values.
- The model notation introduced in Section 3 seems unnecessary.
- Introduce n (line 58).
- What is CNN in line 189 (the ResNet model?), same with RNN (the previously introduced LSTM?), it would be better if the authors could stick to the introduced names
- Although intuitive, random baseline should be explained.
- Appendix is rather packed, it would be helpful to refer to different relevant sections of the appendix in the main body of the paper.

**Comments about the checklist:**
- The code is not shared. Please revisit 3(a) and 4(c) and indicate that the code *will be* shared.
- Please review 4(b) and 4(d). Even if datasets are publicly available, they might come with their licenses, and unfortunately it is not always the case that consent was gathered from people.

**Post-rebuttal comments:**

The reviewer would like to thank the authors for their rebuttal which addressed a number of the questions raised in the reviews. After careful consideration and discussion with other reviewers, the reviewer agrees that the goal of the paper is sounds and that the proposed approach has some edge of novelty. Bridging IF and RPS-LJE may be of interest, although as acknowledged in the rebuttal, the benefits of the diversity achieved by the proposed approach remain unclear. 


The remaining main concern lies in the execution, and in particular the experimental evidence to support the claims and to ensure robustness of the conclusions of the paper. Unfortunately, the rebuttal did not take the opportunity to discuss the statistical significance of the results. The reviewer agrees that the variance of all methods might come from the randomness of data corruptions, but still thinks it would be worth understanding how robust each approach might be wrt this randomness. Moreover, the authors chose a different experimental setup than the one in previous art. While this might be ok, the decision still seems arbitrary. As suggested in the rebuttal, increasing the % of corruption might make differences between the random baseline and other approaches smaller, but considering different % could perhaps be useful to paint the full picture and potentially lead to more insightful discussion of the results. Beyond these points, unfortunately the rebuttal did not take the opportunity to discuss potential use cases/failure modes of the proposed approach, very briefly surfaced some of the limitations pointed by the reviewers, and did not comment on the conclusions that can be drawn from this research. 

Those are the main reasons why the reviewer remains on the fence and decides to keep their original score.

limitations_and_societal_impact: The paper discusses the broader impact of their work in the appendix. Although the checklist indicates that the limitations are discussed in the appendix, the reviewer could not find this discussion. The reviewer would strongly encourage the authors to move the broader impact to the main body of the paper. The reviewers would strongly encourage the authors to include a limitations discussion, also in the main body of the paper. This is crucial information that should not be disregarded.

What are the limitations of the proposed approach? It would seem that one of the limitations is that it would require storing all training data. Could the authors comment on that? Could we consider storing a subset of the training data that would be sufficient to provide explanations? How could we select the data points to consider?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The aim of this paper is to identify the contribution of the training data to the prediction. This work highlights the short comings of the existing Represented Point Selection (RPS-$l_2$) both empirically and by analyzing the derivation of RPS$-l_2$, and proposes an improved method called RPS-LJE. RPS-LJE is a clear improvement over RPS-$-l_2$ both qualitatively and quantitatively. Qualitatively, it is observed that the explanations produced by RPS-LJE isn't repetitive like RPS$-l_2$. Quantitatively, it performs comparably to, or better that the existing data-explanation approaches.
main_review: Originality: The approach presented in this work to address the shortcomings of RPS-$l_2$ is definitely novel

Please consider citing the following saliency methods for model explanation  -
https://arxiv.org/pdf/2006.16322.pdf
https://arxiv.org/abs/1703.01365

Quality:
The work is technically sound. The derivation of the RPS-LJE is pretty neat and intuitive. The effectiveness of the algorithm is further demonstrated empirically by the experiments where we see both qualitative and quantitative improvements.

Clarity: This work is well written and well organized.

Significance:
The results are significant.

----------------------
Post Rebuttal Comments:
After going through the rebuttal I still believe that the paper is an accept, though not a *clear accept*, and hence, I'm updating my score to **7**. 

Some comments for a future version of the paper:

I agree with DC2m that the (mean) test scores as reported aren't meaningful enough to make a quantitative argument that RPS-LJE is better than the baselines. However, the authors' response seems valid as well, i.e., the std is primarily influenced by the randomness of data corruptions across multiple runs. In such a case, the authors could report the fraction of examples for which RPS-LJE achieves the best score. Comparing the mean and std of this metric would be more meaningful.

Also, do consider citing: [Scaling Symbolic Methods using Gradients for Neural Model Explanation](https://arxiv.org/pdf/2006.16322.pdf)
limitations_and_societal_impact: Nil.
review_title: null

Meta-review: This paper proposes a variant of RPS called RPS-LJE that is claimed to offer several advantages over the previously proposed RPS-L2. The “explanation” acts on the last layer of deep nets only, atributing predictions to training points while treating the neural network as fixed. As such, it seems unreasonable to call this (or any related works that act similarly) explanatinos of “deep neural networks” vs of linear models wiith some fixed basis. The conversation centered around several claimed advantages — efficiency (it appears to be resolved in discussion that there is no efficiency gain over influence functions), a faithfulness advantage over RPS-L2, and the striking similarity and high agreement with influence function, raising the question of whether the method is really adding anything or if the novelty is all in the new perspective/derivation. 

The rebuttal and discussion were thoughtful and all reviewers were engaged. However, while they converged on several points of fact, a few others were left unresolved and their were some genuine normative disagreements about whether the contributions that withstood the test of the debate were sufficient to warrant acceptance. 

I will not hide the fact that this is a difficult metareview to write and that I am not sure how to resolve the remaining impasse. I am still seeking additional opinions and hope to make a more confident assessment but believe this is a genuinely difficult case. My current best assessment now is that this paper is truly borderline. However, I believe that with reasonable adjustments to claims and presentation per promises made in the discussion period, this paper is honest work and we can leave it to posterity (vs the review process) to determine the work's eventual impact. 
Decision: Accept (Poster)