Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The main contribution of this paper is SimplEx: a method that aims to explain a test prediction by decomposing it into the weighted sum of nearest neighbors (from a user-specified corpus) in the latent space. They claim this method is more precise and robust than existing explanation-by-example-comparison methods (namely, KNN-like explanations) and distinguish this method from (often gradient-based) feature saliency methods. The paper builds off work that try to explain model predictions that use comparisons in latent variable space (e.g., Concept Activation Vectors, Deep KNN). SimplEx works PROVIDED the model has a hidden layer that linearly maps to the output (this is the layer that is used for explaining predictions). 

- This constraint is significant for accurate weighting of examples. Note that if this is not satisfied by the model (say for instance a softmax is added right before the final output), it is possible to consider an earlier layer that satisfies the linear-mapping constraint as the output

Note too that if the latent space spanned by the user uploaded corpus does not contain the latent representation of the test prediction of interest (it likely will not), the test's latent is projected into the space. This means that it is possible to use this method on any input, regardless of the corpus. We can then measure how good this approximation is by calculating the residual between the corpus span and the test latent.

We can then "interpret the shift in latent space as resulting from a shift in the input space." This is the crux selling point of the paper, and it is accomplished using integrated (and projected) jacobians. This technique also lets us see the contribution of a feature from a corpus example.

The authors then proceed to (briefly) characterize their model on MNIST against dKNN inspired methods, and show a use case on a clinical risk dataset.

They claim the following novelty:

1. Freedom to choose comparison corpus, no need for this to equal the training set
2. The instance-level decompositions are valid in latent and output space, and are more robust than other explainability methods
main_review: The main contribution of this paper is SimplEx: a method that aims to explain a test prediction by decomposing it into the weighted sum of nearest neighbors (from a user-specified corpus) in the latent space. They claim this method is more precise and robust than existing explanation-by-example-comparison methods (namely, KNN-like explanations) and distinguish this method from (often gradient-based) feature saliency methods. The paper builds off work that try to explain model predictions that use comparisons in latent variable space (e.g., Concept Activation Vectors, Deep KNN). SimplEx works PROVIDED the model has a hidden layer that linearly maps to the output (this is the layer that is used for explaining predictions). 

- This constraint is significant for accurate weighting of examples. Note that if this is not satisfied by the model (say for instance a softmax is added right before the final output), it is possible to consider an earlier layer that satisfies the linear-mapping constraint as the output

Note too that if the latent space spanned by the user uploaded corpus does not contain the latent representation of the test prediction of interest (it likely will not), the test's latent is projected into the space. This means that it is possible to use this method on any input, regardless of the corpus. We can then measure how good this approximation is by calculating the residual between the corpus span and the test latent.

We can then "interpret the shift in latent space as resulting from a shift in the input space." This is the crux selling point of the paper, and it is accomplished using integrated (and projected) jacobians. This technique also lets us see the contribution of a feature from a corpus example.

The authors then proceed to (briefly) characterize their model on MNIST against dKNN inspired methods, and show a use case on a clinical risk dataset.

They claim the following novelty:

1. Freedom to choose comparison corpus, no need for this to equal the training set
2. The decompositions are valid in latent and output space

# Main Review

Provide a full review of the submission, including its originality, quality, clarity, and significance. See [https://neurips.cc/Conferences/2021/Reviewer-Guidelines](https://neurips.cc/Conferences/2021/Reviewer-Guidelines) for guidance on questions to address in your review, and /faq for how to incorporate Markdown and LaTeX into your review.

First, a brief note on Claim 1 — letting users choose the corpus of examples to decompose model predictions into is not particularly novel. The mentioned dKNN method could be easily used with any user-specified corpus, as the authors do in their "Experiments" section.

The submission is well written, and it dives deeply into the problem of understanding model prediction as a combination of human-interpretable examples AND features. This reviewer is not aware of other literature that comprehensively bridges these two methods of explainability, and I believe many fields in AI Interpretability can utilize or be inspired by this effort.

As it stands, the authors have this reviewer convinced that this method is useful in explaining many simple Classification problems. 

- Authors show that this method is better than using KNN to explain, since we don't want to include irrelevant corpus examples.
- Authors show that it can be used to inform users if the prediction is likely correct or not by relating it to examples in the corpus that were either correct or not.
- Authors show this working on both tabular data and image data (although interpreting image features is a bit more challenging)

In its current state, the submission lacks:

1. Accompanying code to play with results (authors have reported it will be published eventually)
2. Stories showing this method's effectiveness on problems beyond simple classification and architectures beyond established MLP and CNNs.
3. Quantitative (and even qualitative) comparisons to existing methods. For example, showing this method working on MNIST is actually rather boring, since no comparable results from other saliency methods are shown.
    - Additionally, this could take the form of a quick user study involving real doctors/nurses on the provided clinical risk dataset.

I also imagine this method is significantly slower than comparable ones (like dKNN) since it requires calculation of the integrated jacobian for every test example. This would then require larger compute to explain a large batch of predictions at once. It would be good if the authors included some performance metrics in their paper.

### Additional Comments

1. The introductory paragraph uses large, sweeping sentences that are both unprofessional in an academic paper and distracting from the point. Please reduce the wordiness.
2. Please check the paper for grammatical English errors. There are many instances where plural nouns are incorrectly substituted by their singular form, and several sentences early in the paper that I had trouble understanding:
    - "paterns", "exaplanations", "accesible", "developped"
    - [L188] "we do no longer need"
    - [L213] "an a corpus example"
3. Figures with screenshots of tabular data are difficult to read without zooming in.
4. I STRONGLY recommend choosing a color scheme other than green/red for Figure [3,7,8] for accessibility reasons. Consider a secondary encoding, or solutions proposed [here](https://www.visualisingdata.com/2019/08/five-ways-to-design-for-red-green-colour-blindness/#:~:text=The%20pink%2Dred%20through%20to,green%20hues%20used%20by%20default.) or [here](https://www.visualisingdata.com/2015/11/colour-swatch-alternatives-to-green-and-red/)
limitations_and_societal_impact: The method presented in the paper is described well, and the included examples (particularly of patient classification) show that this method can be appealing to those users of AI models that want to understand why a model makes certain predictions. Their claim that this method can generalize across models and datasets gives it even more potential impact. This reviewer is particularly intrigued by their proposition for future work — using this method to understand latent representations in unsupervised learning.

However, the authors themselves did not conduct experiments to show all their claims. They said it can work on regression problems, but showed no examples of it. There are no experiments comparing the results of SimplEx decompositions to feature saliency methods. There are no user studies showing that this would be helpful in the real world. And, while reporting performance metrics on interpretability methods is not the standard, I believe it would be good to discuss this point.

I believe this paper deserves to be included in the NeurIPS proceedings, but its story could be improved by addressing the above.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors propose to explain the behavior of black-box neural networks by approximating a top-level hidden layer (no non-linearities afterwards, except possibly softmax) as a linear combination of the hidden layers obtained from different examples. These examples may come from the training data, but it is not required. The contribution of each input feature is also evaluated using an extension of integrated gradient across multiple dimensions.
main_review: While explaining neural networks through a combination of examples was introduced earlier, the ability to use a corpus that differs from the training data is, to the best of my knowledge, novel (unless I missed some important work). The integrated Jacobian is a fairly straightforward extension of integrated gradients to multiple dimensions.

It is great that the approach can be employed with a variety of corpora for explainability. However, the authors should also have evaluated the quality of corpus decomposition (section 3.1) when the training data is used.

The experiments cover example decomposition reasonably well, but pay relatively little attention to the feature explanations (using the integrated Jacobian). The authors show the contribution of the features in a few figures, but there is very limited discussion.

In addition to techniques used for black-box model explainability, I would suggest that the authors also describe approaches to directly make models more interpretable, such as glass-box models.

The paper is generally fairly clear, although there are numerous typos: L5 centred, L44 paterns, L96 example (should be plural), L97 accesible, L98 exaplanations, and likely others.

The paper has the potential to be significant as explainability can be an important consideration for the release of machine learning products. The methods are also fairly simple, which could accelerate their adoption by machine learning researchers and practitioners.

[Post author response] After reading the other reviews and the author response, I have increased the score I assigned to this submission.

My remark about using the training data was unclear. I meant using the entire training data as the corpus for SimplEx decomposition. This point was partially addressed in the response to reviewers QnMz and coXq with corpora up to 1000 examples.

The additional experiment in the author response helps justify the use of the integrated Jacobians over integrated gradients (although an even more thorough exploration would be preferable).

I am also satisfied by the authors' commitment to discuss some related work in more detail.

Another typo I saw when reading the paper again: L249 "bellow"
limitations_and_societal_impact: Especially for sensitive data, such as in the medical domain, the corpus examples need to be carefully anonymized.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces SimplEx, a method that approximate the hidden representation of a test example using the linear combination of hidden representations of examples in a pre-defined corpus. Further, the hidden representation of the test instance is associated with each input feature of these examples in the corpus. The paper demonstrates the quality of approximation of SimplEx (Sec 3.1, SimplEx outperforms kNN baselines) and demonstrate its practical utility in the experiment of predicting cancer mortality (Sec 3.2) by allowing model users to interpret model predictions and determine whether to trust the prediction. Detailed math derivation and additional experiments with MNIST and time series data are included in the appendix.
main_review: Strength:
- A novel framework to interpret neural classification models with minimal assumptions on the model architecture.
- SimplEx bridges attribution-based (feature-based) interpretation methods and example-based interpretation method.
- The intuition of SimplEx is reasonable and mimics human learning process (by retrieving and comparing to past examples).
- Writing is coherent and self-contained. Illustrations are clear.

Weakness:
- It is questionable whether introducing weights (w^c) assigned to examples from the corpus improves interpretability empirically. The work may benefit from additional human evaluation, e.g., showing interpretation produced by SimplEx (Fig. 7) and produced by kNN Uniform/Distance/Random (hard-coding w_c).
- The SimplEx method may be computationally expensive (precompute the projected Jacobian for every instance in the corpus, perform a convex optimization procedure for every test instance). Some discussion on this topic would be helpful.

Questions:
- The assumptions in the Sec 3.2 use case is very strong — The doctors are required to withhold 1000 labeled examples to use SimplEx. Is this a hard constraint? How does the explanation quality change when fewer datapoints are used.
- I find it hard to understand Fig 2 and the related experiment. Does the figure suggest the patients with top 100 largest residues are supposed to be UK patients in the ideal setting (as their representations should be different from those of USA patients)? Does this suggest SimplEx can also be used for outlier/anomaly detection?
limitations_and_societal_impact: The authors describes assumption 2.1 as the main limitation to their approach. My concerns and questions are placed in the "main review" section.
I believe the work does not have foreseeable negative societal impact.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper introduces SimplEx, which is a method to post-hoc explain a model prediction by computing from a user-defined set of examples, how the set of examples decompose the prediction. For example, the clinical risk prediction of a patient can be explained by a set of other patients and how much each patient influenced the prediction. 

Contributions
-	A post-hoc explanation method, called SimplEx, were users can choose the set of examples from which the explanation should be constructed. This is advantageous when the training set is not available or a user wants customized explanations, e.g. a doctor wanting to understand a predictions in terms of other patients she/he knows.
-	SimplEx combines the ideas of feature saliency and example-based explanations. For each example, the method computes the percentage of this example contributing to explaining the prediction as well as which features of this example were important.
-	A latent-space generalization of Integrated Gradients [10]
-	Experimental evaluation that show SimplEx to be more precise and robust compared to baselines.

main_review: Strengths
-	The paper is well written and easy to follow
-	The method is interesting and supported by experimental results

Weaknesses
-	The paper leaves some natural questions open (see questions below).
-	Line 170 mentions that the corpus residual can be used to detect an unsuitable corpus, but there are no experiments to support this.

After authors' response
All the weakness points have been addressed by the authors' response. Consequently I have raised my score. In particular:
- The left open questions have all been answered.
- There indeed is an experiment to support this, thanks to the authors' for clarifying this, that connection was not clear to me previously.

Questions
-	Line 60: Why do you say that e.g. influence functions cannot be used to explain a prediction? The explanation of a prediction could be the training examples whose removal (as determined by the influence function) would lead to the largest score drop for a prediction.
-	How does the method scale as the corpus size or hidden dimension size is increased?
-	What happens if a too small corpus is chosen? Can this be detected?
-	What if we don’t know that a test example is crucially different, e.g. what if we don’t know that the patient of Figure 8 is “British” and we use the American corpus to explain it? Can this be detected with the corpus residual value?
-	In the supplementary material you mention how it is possible to check if a decomposition is unique. Do you do this in practice when conducting experiments? How do you choose a decomposition if it is not unique? What does it imply for the experiments (and the usage of the method in real-world applications) if the decomposition is not unique?

Typos, representation etc.
-	Line 50: An example of when a prototype model would be unsuitable would strengthen your argument.
-	Footnote 2: “or” -> “of”
-	Line 191: when the baseline is first introduced, [10] or other references would be helpful to support this approach
-	Line 319: “the the” -> “the”
-	Line 380: “at” -> “to”?

limitations_and_societal_impact: A broader impact section could be added. In a separate section (e.g. supplementary material), there could be an explicit discussion on when the method should not be used, e.g. as shown in Figure 8, the American corpus shouldn’t be used to explain the British patient. Also see last question above – what if we don’t know that the patient is British? Can this be detected? This should also be discussed in such a section.
review_title: null

Meta-review: This paper proposes a method for explaining machine learning models by decomposing the continuous representation of a test example into a combination of examples from a given prototype corpus. 

Reviewers find the method convincing, powerful (bridging two large framework of explainable ML: feature-based and example-based), well presented and well demonstrated. The authors have thoroughly addressed all outstanding concerns in the discussion period, including a small user study. I strongly urge the authors to take the time to implement the corresponding improvements in the manuscript.
Decision: Accept (Spotlight)