Explaining Latent Representations with a Corpus of Examples
Authors: Jonathan Crabbé
Zhaozhi Qian
Fergus Imrie
Mihaela van der Schaar
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: explainability
interpretability
latent
representation
decomposition
exampled-based
explanation
feature-based
feature
importance

Abstract: Modern machine learning models are complicated. Most of them rely on convoluted latent representations of their input to issue a prediction. To achieve greater transparency than a black-box that connects inputs to predictions, it is necessary to gain a deeper understanding of these latent representations. To that aim, we propose SimplEx: a user-centred method that provides example-based explanations with reference to a freely selected set of examples, called the corpus. SimplEx uses the corpus to improve the user’s understanding of the latent space with post-hoc explanations answering two questions: (1) Which corpus examples explain the prediction issued for a given test example? (2) What features of these corpus examples are relevant for the model to relate them to the test example? SimplEx provides an answer by reconstructing the test latent representation as a mixture of corpus latent representations. Further, we propose a novel approach, the integrated Jacobian, that allows SimplEx to make explicit the contribution of each corpus feature in the mixture. Through experiments on tasks ranging from mortality prediction to image classification, we demonstrate that these decompositions are robust and accurate. With illustrative use cases in medicine, we show that SimplEx empowers the user by highlighting relevant patterns in the corpus that explain model representations. Moreover, we demonstrate how the freedom in choosing the corpus allows the user to have personalized explanations in terms of examples that are meaningful for them.



Explaining Latent Representations with a Corpus of Examples
Jonathan Crabbé University of Cambridge j jc21338cam. ac. uk
Zhaozhi Qian
University of Cambridge zq2240maths. cam ac uk
Fergus Imrie
Mihaela van der Schaar University of Cambridge The Alan Turing Institute
UCLA imrie0g ucla. edu
UCLA mv4720cam ac. uk
Abstract
Modern machine learning models are complicated. Most of them rely on convoluted latent representations of their input issue prediction. To achieve greater transparency than a black-box that connects inputs to predictions, it is necessary gain a deeper understanding of these latent representations. To that aim, we propose SimplEx: a user-centred method that provides example-based explanations with reference to freely selected set of examples, called the corpus. SimplEx uses the corpus to improve the user's 's understanding of the latent space with post-hoc explanations answering two questions: (1) Which corpus examples explain the prediction issued for given test example? (2) What features of these corpus examples are relevant for the model to relate them the test example? SimplEx provides an answer by reconstructing the test latent representation as mixture of corpus latent representations. Further, we propose a novel approach, the Integrated Jacobian, that allows SimplEx to make explicit the contribution of each corpus feature the mixture. Through experiments on tasks ranging from mortality prediction to image classification, we demonstrate that these decompositions are robust and accurate. With illustrative use cases in medicine, we show that SimplEx empowers the by highlighting relevant patterns in the corpus that explain model representations. Morcover, we demonstrate how the freedom in choosing the corpus allows the have personalized explanations in terms of examples that are meaningful for them.
Introduction and related work
How can we make a machine learning model convincing? If accuracy is undoubtedly necessary, it is rarely sufficient. As these models are used in critical areas such as medicine, finance and the criminal justice system, their black-box nature appears as a major issue [1, 2, 3]. With the necessity to address this problem, the landscape of explainable artificial intelligence (XAI) developed [4. 5]. first approach XAI is to focus on while-box models that are interpretable by design. However, restricting to class of inherently interpretable models often comes the cost of lower prediction accuracy [6]. In this work, we rather focus on post-hoc explainability techniques. These methods aim at improving the interpretability of black-box models by complementing their predictions with various kinds explanations. In this way, it is possible to understand the prediction model without sacrificing its prediction accuracy.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Feature importance explanations are undoubtedly the most widespread type of post-hoe explanations. Popular feature importance methods include SHAP [7. 8, 9]. LIME [10], Integrated Gradients [11], Contrastive Hxamples [12] and Masks [13, 14, 15]. These methods complement the model prediction for an input example with a score attributed to each input feature. This score reflects the importance of each feature for the model to issue its prediction. Knowing which features are important for model prediction certainly provides more information on the model than the prediction by itself. However, these methods do not provide reason as to why the model pays attention to these particular features.
Another approach is to contextualize each model prediction with the help of relevant examples. In fact, recent works [16] have demonstrated that human subjects often find example- based explanations more insightful than feature importance explanations. Complementing the model's 's predictions with relevant examples previously seen by the model is commonly known as Case-Based Reasoning (CBR) [17, 18, 19]. The implementations of CBR generally involve models that create a synthetic representation of the dataset, where examples with similar patterns are summarized by prototypes [20, 21, 22]. At inference time, these models relate new examples to one or several prototypes to issue prediction. In this way, the patterns that are used by the model to issue a prediction are made explicit with the help of relevant prototypes. A limitation of this approach is the restricted model architecture. The aforementioned procedure requires to opt for a family of models that rely on prototypes to issue prediction. This family of model might not always be the most suitable for the task at hand. This motivates the development of generic post-hoc methods that make few or no assumption on the model.
The most common approach to provide example- explanations for wide variety of models mirrors feature importance methods. The idea is to complement the model prediction by attributing a score to each training example. This score reflects the importance of each training example for the model to issue its prediction. This score will typically be computed by simulating the effect of removing each training instance from the training set on the learned model [23]. Popular examples of such methods include Influence Functions [24] and Data-Shapley [25, 26]. These methods offer the advantage of being flexible enough to be used with a wide variety of models. They produce scores that describe what the model could have predicted if some examples were absent from the training set. This is very interesting in data valuation perspective. However, in an explanation perspective, is not clear how to reconstruct the model predictions with these importance scores.
So far, we have only discussed works that provide explanations of model output, which is the tip of the iceberg, Modern machine learning models involve many convoluted transformations to deduce the output from an input. These transformations are expressed in terms of intermediate variables that are often called latent variables. Some treatment of these latent variables is necessary if we want to provide explanations that take the model complexity into account. This motivates several works that push the explainability task beyond the realm of model outputs. Among the most noticeable contributions in this endcavour, we cite Concept Activation Vectors that create a dictionary between human friendly concepts (such the presence of stripes in an image) and their representation in terms of latent vectors [27]. Another interesting contribution is the Deep k-Nearest Neighbors model that contextualizes the prediction for example with its Nearest Neighbours in the space of latent variables, the latent space [28]. An alternative exploration of the latent space is offered by the representer theorem that allows, under restrictive assumptions, to use latent vectors decompose model's prediction in terms of its training examples [29].
Contribution In this work, we introduce a novel ap- proach called SimplEx that
Test Example
Corpus Exemples
lies at the crossroad of the
Immgge Cassifiaation
3
3
above research directions. SimplEx outputs host-hoc explanations in the form of Figure 1, where the model's prediction and latent repre- sentation for a test example is approximated
46% x
+37% x
+17%
Cirical Rai
a mix- ture of examples extracted from corpus of examples.
50% X
25%
17%
Figure 1: An example of corpus decomposition with SimplEx.
2

---

In each case, SimplEx highlights the role played by each feature of each corpus example in the latent space decomposition. SimplEx centralizes many functionalities that, to the best of our knowledge, constitute a leap forward from the previous state of the art. (1) SimplEx gives the user freedom to choose the corpus of examples whom with the model's predictions are decomposed. Unlike previous methods such as the representer theorem, there is no need for this corpus of examples to be equal to the model's training set. This is particularly interesting for two reasons: (a) the training set of a model is not always accessible (b) the user might want explanations in terms of examples that make sense for them. For instance, a doctor might want to understand the predictions of a risk model in terms of patients they know. (2) The decompositions of SimplEx valid, both in latent and output space. We show that, in both cases, the corpus mixtures discovered by SimplEx offer significantly more precision and robustness than previous methods such Deep k-Nearest Neighbors and the representer theorem. (3) SimplEx details the role played by each feature in the corpus mixture. This is done by introducing Integrated Jacobians, a generalization of Integrated Gradients that makes the contribution of each corpus feature explicit in the latent space decomposition. This creates a bridge between two research directions that have mostly developed independently: feature importance and example-based explanations [19, 30J. In Section 3 of the supplementary material, we report a user-study involving 10 clinicians. This study supports the significance of our contribution.
2 SimplEx
In this section, we formulate our method rigorously. Our purpose is to explain the black-box prediction for unseen test example with the help of set of known examples that we call the corpus. We start with a clear statement of the family of black-boxes for which our method applies. Then, we detail how the set of corpus examples can be used to decompose black-box representation for the unseen example. Finally, we show that the corpus decomposition can offer explanations at the feature level.
2.1 Preliminaries
Let X' C RRx be an input (or feature) space and y C Rdy be an output (or label) space, where dx and dy are respectively the dimension of the input and the output space. Our task is to explain individual predictions of a given black-box f: X' + V. In order to build our explainability method, we need to make an assumption on the family of black-boxes that we wish to interpret.
Assumption 2.1 (Black-box Restriction). We restrict to black-boxes : X' - y that can be decomposed as f = 1o g, where g X H maps input E X' to latent vector h g (x) H and H + y linearly maps' a latent vector h E H to an output y = I(h) = Ah E y. In the following, we call H C Rdd the latent space. Typically, this space has higher dimension than the output space d > dy.
Remark 2.1. In the context of deep-learning, this assumption requires that the last hidden layer maps linearly to the output. While it is often the case, it is crucial in the following since we will use the fact that linear combinations in latent space correspond to linear combinations in output space. Our purpose is to gain insights on the structure of the latent space.
Remark 2.2. This assumption is compatible with regression and classification models, we just need to elarify what we mean by output in the case of classification. If is a classification black-box that predicts the probabilities for each class, it will typically take the form in Assumption 2.1 up to normalizing map & (typically a softmax):
&b colog g. In this case, we ignore² the normalizing map & and define the output to be y (1 oog) (x).
Our explanations for rely on a set of examples that we call the corpus. These examples will typically (but not necessarily) be representative subset of the black-box training set. The corpus set has be understood as a set of reference examples that we want to use as building blocks to interpret unseen examples. In order index these examples, it will be useful denote by [m m22] the set of natural
numbers between the natural numbers and n2 with ni < R2. Further, we denote [m] [] n. the set of natural numbers between and n 2/ 1. The corpus of examples containing C' E N' examples x E X. In the following, superscripts are labels for examples and subscripts are labels for vector components. In this way, 2f has be understood as the component i of corpus example c.
set
[x" c [C]]
1 'The map can fact be affine. In the following, we omit the bias term E y that can be reabsorbed in g. "There is no loss of information the output allows reconstruct class probabilities p via p 0 (y).
3

---

2.2 A corpus of examples to explain a latent representation
Our purpose is to understand a prediction f(x) for an unseen test example with the help of the corpus. How can we decompose the prediction f(x) in terms of corpus predictions f(x*)? A naive attempt would be to express as mixture of inputs from the corpus C: E x with weights w E: [0, 1] that sum to one Σe= w = 1. The weakness of this approach is that the signification of the mixture weights is not conserved if the black-box f is not a lincar map: (( xx**) * >>=1 wx'(.
Fortunately, Assumption 2.1 offers us a better vector space to perform a corpus decomposition of the unseen example x. We first note that the map g induces latent representation of the corpus g(C) = [h" = g(x*) x CC C H. Similarly, x has latent representation h = g(x) E H. could therefore perform corpus decomposition in latent space h E w Now, by using the linearity of we can compute the black-box output of this
Following the above line of reasoning,
mixture in latent space: 1(Ee whh") = Ee w="(h") In this case, the weights that are used to decompose the latent representation h in terms of the latent representation of the corpus g(C)) also reflect the way in which the black-box prediction f(x) can be decomposed in terms of the corpus outputs f(C.. This hints that the latent space H is endowed with the appropriate geometry to make corpus decompositions. More formally, we think in terms of the convex hull spanned by the corpus. Definition 2.1 (Corpus Hull). The corpus convex hull spanned by corpus C with latent representation g(C) = [h = g(x9) x E CJ C. H. is the convex set
{
CH (C) (C)= Σ h w [0. (C]
Σ w 1
Remark 2.3. This is the set of latent vectors that are a mixture of the corpus latent vectors.
At this stage, is important to notice that an exact corpus decomposition is not possible if & CH(C). In such a case, the best we can do is to find the element h E CH((C that best approximates If H endowed with a norm [|,, this corresponds to the convex optimization problem
arg min
h hh-hi h ll
(1)
E CH(C)
By definition, the corpus representation h of h can be expanded mixture of elements from measure of importance in the
g(C): h - E wh The weight can naturally be interpreted reconstruction of h with the corpus. Clearly, w x 0 for some E [C] indicates that h does not play significant role in the corpus representation h of h. On the other hand, w" se 1 indicates that h generates the corpus representation h by itself.
At this stage, a natural question arises: how can we know if the corpus approximation h is good approximation for h? The answer is given by the residual vector h - h that measures the shift between the latent representation h = g(x) and the corpus hull (C) is natural to use this residual vector to detect examples that cannot be explained with the selected corpus of examples C.
Definition 2.2 (Corpus Residual). The corpus residual associated to a latent vector E H. and its corpus representation h CC((C) solving (1) is the quantity
rc (h)
h ||
min h E CH(C)
|| h h ||
hg
In Section 1.1 of the supplementary material, we show that the corpus residual also controls the quality of the corpus approximation in out- put space y. All the corpus-related quantities that we have introduced so far are summarized visually in Figure 2. Note that this Figure is simplification of the reality as C' will typically be larger than 3 and dx, dæ will typically be higher than 2. We are now endowed with a rigorous way to decompose test example in terms of corpus examples in latent space. In the next section, we detail how to pull-back this decomposition to input space.
h
h
CH(C)
X
H
h
Figure 2: Corpus convex hull and residual.
Note that this decomposition might not be unique, more details in Section I of the supplementary material.

---

2.3 Transferring the corpus explanation in input space
Now that we are endowed with corpus decomposition h = = would be convenient to have an understanding of the corpus decomposition in input space X. For the sake of notation, to draw a distinction between the latent representation of the unseen example x and its corpus decomposition h. If we want to understand the corpus decomposition in input space, a natural approach [11] fix baseline input x" together with its latent representation h g(x"). Let us now decompose the representation shift h - h in terms of the corpus:
=c wh that approximates h, it
will assume that the corpus approximation is good so that it is unnecessary
h we (h -hhh.)..
With this decomposition, understand the total shift in latent space h h in terms of individual contributions from each corpus member. In the following, we focus on the comparison between the baseline and single corpus example x" together with its latent representation h by keeping in mind that the full decomposition (2) can be reconstructed with the whole corpus. To bring the discussion in input space X, we interpret the shift in latent space h hh as resulting from a shift x - xx" the input space. We interested in the contribution of each feature to the latent space shift. To decompose the shift in latent space in terms of the features, we parametrize the shift in input space with a line y [0, 1] X' that goes from the baseline the corpus example: (1) x + L. (x x²) for t [0, 1]. Together with the black-box box, this line induces curve in latent space g y [0, 1] H that goes from the baseline latent representation h² to the corpus example latent representation h". Let us now use an infinitesimal decomposition of this curve to make the contribution of each input feature explicit. If we assume that is differentiable at (t). we can use a first order approximation of the curve at the vicinity of E 10,1) to decompose the infinitesimal shift latent space:
dx
Og
d:f
gt + 5t) - g 7c
öt + o(5t)
Dxg
dt
(t)
Infinitesimal shift in lalent space
dx
0g
(x 22 St o(8))
Dzg
(t)
where we used (4) = .! +1 (1f x) to obtain the second equality. In this decomposition, each
input feature contributes additively to the infinitesimal shift in latent space. It follows trivially that the contribution of the input feature corresponding input dimension [dx] is given by
0g
oi (t) = (Iς x
8t E H.
dr:
In order to compute the overall contribution of feature to the shift, we let öt - 0 and sum the infinitesimal contributions along the line y5. we assume" that g is almost everywhere differentiable, this sum converges to integral in the limit öt + 0. This motivates the following definitions.
Definition 2.3 (Integrated Jacobian & Projection). The integrated Jacobian between baseline (x", h" g(xx)) and corpus example (x", &²,h" h =g(x")) X x H. associated to feature E [dx] is
j = (xf z!!) f'" Ög
dt
E
dz, (t)
where ( = + (xx - x)) for & E [o, 1]. This vector indicates the shift in latent space induced by feature i of corpus example when comparing the corpus example with the baseline. To summarize this contribution the shift h hd described in (2), we define the projected Jacobian
h h
p proja de ( h
h ER,
where (..-) is an inner product for H and the normalization is chosen for the purpose of Proposition 1. Remark 2.4. The integrated Jacobian can be seen a latent-space generalization of Integrated Gradients [11]. In Section 1.3 of the supplementary material, we establish the relationship between the two quantities: IG, I(])).
TTis not restrictive, DNNs with ReLU activation functions satisfy this assumption for instance.
5

---

We summarize the Jacobian quantities in Fig- ure 3. By inspecting the figure, we notice that projected Jacobians encode the contribution of feature from corpus example e to the overall shift in latent space: po 0 implies that this feature creates a shift pointing the same direc- tion the overall shift; pf < 0 implies that this feature creates a shift pointing in the opposite direction and p 0 implies that this feature creates a shift in an orthogonal direction. We use the projections to summarize the contribution of each feature in Figures 1 8 & 9. The colors green and red indicate respectively positive and negative projection. In addition to these geometrical insights, Jacobian quantíties come with natural properties.
38
hd
H
P5
12
Figure 3: Integrated Jacobian and projection.
Proposition 2.1 (Properties of Integrated Jacobians). Consider a baseline (x h g(x))) and test example together with their latent representation (x, h g(x)) E X x H. If the shift h h admits decomposition (2), the following properties hold.
(A)
w
h
(B):
w p 1.
Proof. The proof is provided in Section 1.4 of the supplementary material.
These properties show that the integrated Jacobians and their projections are the quantities that we are looking for: they transfer the corpus explanation into input space. The first equality decomposes the shift in latent space in terms of contributions w arising from each feature of each corpus example. The second equality sets a natural scale to the contribution of each feature. For this reason, is natural to use w wfpf to measure the contribution of feature i of corpus example
3 Experiments
In this section, we evaluate quantitatively several aspects of our method. In first experiment, we verify that the corpus decomposition scheme described in Section yields good approximations for the latent representation of test examples extracted from the same dataset as the corpus examples. In realistic clinical use case, we illustrate the usage of SimplEx in a set-up where different corpora reflecting different datasets are used. The experiments are summarized below. In Section 2 of the supplementary material, we provide more details and further experiments with time series and synthetic data. The code for our method and experiments is available on the Github repository https:/ //gi thub. cwm/ comm/JJoaahannraabeeb /aimeaaaabb//mmee....eeab All the experiments have been replicated on different machines.
3.1 Precision of corpus decomposition
Description The purpose of this experiment is to check if the corpus decompositions described in Section 2 allows us to build good approximations of the latent representation of test examples. We start with a dataset ID that we split into a training set Durin and a testing set Dies- We train black-box f for given task on the training set Dusin. We randomly sample set of corpus examples from the training set C C Diran (we omit the true labels for the corpus examples) and a set of test examples from the testing set T C Diest- For each test example x E T, we build an approximation h g(x) with the corpus examples latent representations. In each case, we let the method use only K corpus examples to build the approximation. We repeat the experiment for several values of K.
for
Metrics We are interested in measuring the precision of the corpus approximation in latent space and in output space. To that aim, we use the R² score both spaces. In this way, R2 measures the precision of the corpus approximation h with respect to the true latent representation h. Similarly, R2' measures the precision of the corpus approximation
I(h) with respect to the true output y 1(h). Both of these metrics satisfy -o < R² < 1. A higher R² score is better with R²² = 1

---

1.000
0.95
0.90
0.75
SimplEx
SimpFEx
KNN Uniform
KNN Uniform
0.70
KNN Distance
0.825
KNN Distance
10
30
40
50
(a) Ri score for the latent approximation (b) Iy score for the output approximation
Figure Precision of corpus decomposition for prostate cancer (avg + std).
0.99
0.95
0.97
35 0.96
0.95
SimplEx
0.94
SimplEx
Uniform
KNN Uniform
KNN Distance
01333
KNN Distance
10
20
20
K
K
(a) R2 score for latent approximation (b) R$ score for the output approximation
Figure 5: Precision of corpus decomposition for MNIST (avg + std).
corresponding to perfect approximation. All the metrics are computed over the test examples T. The experiments are repeated 10 times to report standard deviations across different runs.
Baselines We compare our method (SimplEx) with baselines. A first approach, inspired by [28], consists in using the K-nearest corpus neighbours latent space build the latent approximation h. Building on this idea, we introduce two baselines (1) KNN Uniform that takes the average latent representation of the K-nearest corpus neighbours of h in latent space (2) KNN Distance that computes the same average with weights wf inversely proportional to the distance
he l| Finally, we use the representer theorem [29] to produce an approximation ý of with the corpus C. Unlike the other methods, the representer theorem does not allow to produce an approximation in latent space.
Datasets We enrolled in the American SEER program [31]. We consider the binary classification task of predicting cancer mortality for patients with prostate cancer. We train a multilayer perceptron (MLP) for this task. Since this task is simple. we show that corpus of C = 100 patients yields good approximations. (2) 70,000 MNIST images of handwritten digits [32]. We consider the multiclass classification task of identifying the digit represented on each image. We train convolutional neural network (CNN) for the image classification. This classification task is more complex than the previous one (higher dx and dy), we show that corpus of C = 1, 000 images yields good approximations in this case.
different datasets with distinct tasks for our experiment: (1) 240,486 patients
The results for SimplEx and the KNN baselines are presented Figure 4 & 5. Several things can be deduced from these results: (1) It is generally harder to produce a good approximation in latent space than in output space as R2 < Rý, for most examples (2) SimplEx produces the most accurate approximations, both in latent and output space. These approximations are of high quality with R² 1. (3) The trends are qualitatively different between SimplEx and the other baselines. The accuracy of SimplEx increases with K and stabilizes when small number of corpus members contribute (K in both cases). The accuracy of the KNN baselines increases with K,
Results
To enforce SimplEx to select examples, add a L penalty making the C K smallest weights vanish.
7

---

reaches a maximum for small K' and steadily decreases for larger K. This can be understood easily: when K increases beyond the number of relevant corpus examples, irrelevant examples will be added in the decomposition. SimplEx will typically annihilate the effect of these irrelevant examples by setting their weights w to zero in the corpus decomposition. The KNN baselines include the irrelevant corpus members in the decomposition, which alters the quality of the approximation. This suggests that K has to be tuned for each example with KNN baselines, while the optimal number of corpus examples to contribute is learned by SimplEx. (4) The standard deviations indicate that the performances of SimplEx are more consistent across different runs. This is particularly true in the prostate cancer experiment, where the corpus size C is smaller. This suggests that SimplEx is more robust than the baselines. (5) For the representer theorem, we have R3 = - (6.6 + 6.1). 10 for the prostate cancer dataset and R3, = - (7.2 + 6.6) for MNIST. This corresponds to poor estimations of the black-box output. We propose some hypotheses to explain this observation in Section 2.1 of the supplementary material.
3.2 Significance of Jacobian Projections
Description The purpose of this experiment is to check if SimplEx's s Jacobian Projections are good measure of the importance for each corpus feature in constructing the test latent representation h. In the same setting as in the previous experiment, we start with a corpus C of C 500 MNIST images. We build a corpus approximation for an example x E X' with latent representation h = g(x) E H. The precision of this approximation is reflected by its corpus residual (h). For each corpus example x E C, we would like to identify the features that are the most important in constructing the corpus decomposition of With SimplEx, this reflected by the Jacobian Projections pf. We evaluate these scores for each feature i E [dx] of each corpus example c [C]. For each corpus image x E C, we select the n most important pixels according to the Jacobian Projections and the baseline. In each case, we build mask m that replaces these most important pixels by black pixels. This yields corrupted corpus image x = m @ x", where @ denotes the Hadamard product. By corrupting all the corpus images, we obtain a corrupted corpus Ceor We analyse how well this corrupted corpus approximates h, this yields a residual re... (h).
Metric We are interested in measuring the effectiveness of the corpus corruption. This is reflected by the metric o (h) = TC.... (h) - rc rc(h). A higher value for this metric indicates that the features selected by the saliency method are more important for the corpus to produce a good approximation of h in latent space. We repeat this experiment for 100 test examples and for different numbers n of perturbed pixels.
Baseline As baseline for our experiment, we use Integrated Gradients, which close in spirit to our method. In a similar fashion, we compute the Integrated Gradients IG; for each feature E [dx] of each corpus example e [C] and construct corrupted corpus based on these scores.
Results The results are presented in the form of box plots in Figure 6. We observe that the corruptions induced by the Jacobian Projections are significantly more impactful when few pixels are perturbed. The two methods become equivalent when more pixels are perturbed. This demonstrates that Jacobian Projections are more suitable to measure the importance of features when performing latent space reconstruction, as it is the case for SimplEx.
3.3 Use case: clinical risk model across countries
Very often, clinical risk models are produced and validated with the data of patients treated at single site [33]. This can cause problems when these models are deployed at different sites for two reasons: (1) Patients from different sites can have different characteristics (2) Rules that are learned for one site might not be true for another site. One possible way to alleviate this problem would be to detect patients for which the model prediction is highly extrapolated and/or ambiguous. In this way, doctors from different sites can make an enlightened use of the risk model rather than blindly believing the model's 's predictions. We demonstrate that SimplEx provides a natural framework for this set-up.
As in the previous experiment, we consider dataset DUSA containing patients enrolled in the American SEER program [31]. We train and validate an MLP risk model with DISA- To give a realistic realization of the above use-case, we assume that we want to deploy this risk model in different site: the United Kingdom. For this purpose, we extract DUK from the set of 10,086 patients enrolled in the British Prostate Cancer UK program [34]. These patients are characterized by the
8

---

Methed
10
50
Number plaes perturbed
Figure 6: Increase in the corpus residual caused by each method (higher is better).
same features for both DUk and DUSA. However, datasets Duk and DUsa differ by a covariate shift: patients from DUk are in general older and at earlier clinical stages.
When comparing the two populations terms of the model, first interesting question ask is whether the covariate shift between PUSA and Duk affects the model representation. To explore this question, we take first corpus of American patients CUSA DUSA there is indeed a difference in terms of the latent representations, we expect the representations of test examples from Duk to be less closely approximated by their decomposition with respect CUSA. If this is true, the corpus residuals associated to examples of DuK will typically larger than the ones associated to DUSA. To evaluate this quantitatively. we consider a mixed set of test examples T sampled from both DUk and DUSA: T C Duk L DuSA. We sample 100 examples from both sources: IT Duk - 17
DUSAl 100. We then approximate the latent representation each example E g()) and compute the associated corpus residual Cuua (h). We sort the test examples from T by decreasing order of corpus residual and we use this sorted list to see if we can detect the examples from DUK. We use previous baselines for comparison, results are shown in Figure? 7.
Several things can be deduced from this experiment. (1) The results strongly suggest that the difference be- tween the two datasets DUSA and Duk is reflected their latent representations. (2) The corpus residuals from SimplEx offer the most reliable way to detect examples that are different from the corpus examples CUSA- None of the methods matches the maximal baseline since some examples of DUSa resemble ex- amples from Duk. (3) When the corpus examples representative of the training set, it is the case in the experiment, our approach based on SimplEx provides systematic way to detect test examples that have representations that are different from the ones produced training time. A doctor should be more sceptical with respect to model predictions associated to larger residual with respect CUSA as these arise from an extrapolation region of the latent space.
Number of UK patients detected
100
60
SimplEx
7NN Distance
7NN Uniform
20
Random
Maximal
Number patients inspected
Figure 7: Detecting UK patients (avg.+ std.).
Let us now make the case more concrete. Suppose that an American and British doctor use the above risk model predict the outcome for their patients. Each doctor wants to decompose the predictions the model in terms of patients they know. Hence, the American doctor selects a corpus of American patients CUSA C DuSa and the British doctor selects corpus of British patients Cuk C Duk. Both corpora have the same size CUSA = Cuk = 1, 000. We suppose that the doctors know the model prediction and the true outcome for each patient in their corpus. Both doctors sceptical about the risk model and want to use SimplEx to decide when it can be trusted. This leads them to natural question: is it possible to anticipate misclassification with the help of SimplExx
In Figure 8 & 9, we provide two typical examples of misclassified British patients from DUk \ CuK together with their decomposition in terms of the two corpora CUSA and CUK. These two examples

---

exhibit two qualitatively different situations. In Figure 8, both the American and the British doctors make the same observation: the model relates the test patient to corpus patients that are mostly misclassified by the model. With the help of SimplEx, both doctors will rightfully be sceptical with respect to the model's prediction.
In Figure 9, something even more interesting occurs: the two corpus decompositions suggest different conclusions. In the American doctor's perspective, the prediction for this patient appears perfectly coherent as all patients in the corpus decomposition have very similar features and all of them are rightfully classified. On the other hand, the British doctor will reach the opposite conclusion as the most relevant corpus patient is misclassified by the model. In this case, we have perfect illustration of the limitation of the transfer of risk model from one site (America) to another (United Kingdom): similar patients from different sites can have different outcomes. In both cases, since the test patient British, only the decomposition in terms of CUk really matters. In both cases, the British doctor could have anticipated the misclassion lassifiaccin of each patient with SimplEx.
Predic
Predic
True
Age
0S
Age RSA
6000
P
Comoriddiies Truuiimrrs
Comorbiatier
0.25
Frodic
True
51%
Clade
OFIIS Stage
USA
Slage
05.0
FSSA
Corpus
Toeatments
Stage
CF:58
UK
Precic
Tru
Corpus
Age PSA
60.0
PISA
63%
Comprtiaaiss
+ +.7%%
Conortraliaa
Cradda Siage
0.25
Saage
Figure 8: A first misclassified patient.
Predic
Predic,
Predic
True
DEXK
0.25
77%
17% X
6%
Predic
True
USA
Corpus
Predic True
Predic
Corpus
BRCK Truimens Blage
47%
39%
14%
0.25
Figure A second misclassified patient.
4 Discussion
We have introduced SimplEx, a method that decomposes the model representations at inference time in terms of a corpus. Through several experiments, we have demonstrated that these decompositions are accurate and can easily be personalized to the user. Finally, by introducing Integrated Jacobians, we have brought these explanations to the feature level.
We believe that our bridge between feature and example-based explainability opens up many avenues for the future. A first interesting extension would be to investigate how SimplEx can be used to
understand latent representations involved in unsupervised learning. For instance, SimplEx could be used to study the interpretability of self-expressivee latent representations learned by autoencoders [35]. second interesting possibility would be design rigorous scheme select the optimal corpus for a given model and dataset. Finally, a formulation where we allow the corpus to vary on the basis of observations would be particularly interesting for online learning.
10

---

Acknowledgement
The authors are grateful to Alicia Curth, Krzysztof Kacprzyk, Boris van Breugel, Yao Zhang and the 4 anonymous NeurIPS 2021 I reviewers for their useful comments on an earlier version of the manuscript. Jonathan Crabbé would like thank Bogdan Cebere for replicating the experiments in this manuscript, Jonathan Crabbé would like to acknowledge Bilyana Tomova for many insightful discussions and her constant support. Jonathan Crabbé funded by Aviva. Fergus Imrie is supported by by the National Science Foundation (NSF), grant number 1722516 122216..22 Zhaozhi Qian and Mihaela van der Schaar are supported by the Office of Naval Research (ONR), NSF 1722516.
References
[1] Zachary C. Lipton. The Mythos of Model Interpretabilityy Communications of the ACM, 61(10):35-43, 2016.
[2] Travers Ching, Daniel S. Himmelstein, Brett K. Beaulieu-Jones, Alexandr A. Kalinin, Brian Do, Gregory P. Way, Enrico Ferrero, Paul-Michael Agapow, Michael Zietz, Michael M. Hoffman, Wei Xie, Gail L. Rosen, Benjamin J. Lengerich, Johnny Israeli, Jack Lanchantin, Stephen Woloszynek, Anne E. Carpenter, Avanti Shrikumar, Jinbo Xu, Evan M. Cofer, Christopher A. Lavender, Srinivas C. Turaga, Amr M. Alexandari, Zhiyong Lu, David J. Harris, Dave DeCaprio, Yanjun Qi, Anshul Kundaje, Yifan Peng, Laura K. Wiley, Marwin H. S. Segler, Simina M. Boca, Joshua Swamidass, Austin Huang, Anthony Gitter, and Casey S. Greene. Opportunities and obstacles for deep learning in biology and medicine. Journal of The Royal Society Interface. 15(141):20170387, 2018.
[3] Erico Tjoa and Cuntai Guan. Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAL IEEE Transactions on Neural Networks and Leurning Systems, pages 1-21, 2020.
[4] Alejandro Barredo Arrieta, Natalia Diaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado. Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58:82- 115, 2020.
[5] Arun Das and Paul Rad. Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey. arXiv, 2020.
[6] Arun Rai. Explainable AI: from black box to glass box. Journal of the Academy of Marketing Science 2019 48:1, 1171):137114,, 2019.
[7] Lloyd Shapley. A value for n-person games. Contributions to the Theory of Games, 2(28):307- 317, 1953.
[8] Anupam Datta, Shayak Sen, and Yair Zick. Algorithmic Transparency via Quantitative Input Influence: Theory and Experiments with Learning Systems. In Proceedings 2016 IEEE Symposium on Security and Privacy, SP 20/6, pages 598-617. Institute of Electrical and Electronics Engineers Inc., 2016.
[9] Scott Lundberg and Su-In Lee. A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems, pages 4766-4775 2017.
[10] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should trust you?" Explaining the predictions of any classifier. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135-1144 1144. Association for Computing Machinery, 2016.
[11] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic Attribution for Deep Networks. 34th International Conference on Machine Learning, ICML 2017, 7:5109-5118, 2017.
[12] Amit Dhurandhar, Pin- Chen, Ronny Luss Chun-Chen Tu, Paishun Ting, Karthikeyan Shan- mugam, and Payel Das. Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives. Advances Neural Information Processing Systems, pages 592-603, 2018.
11

---

[13] Ruth C. Fong and Andrea Vedaldi. Interpretable Explanations of Black Boxes by Meaningful Perturbation, Proceedings of the IEEE International Conference on Computer Vision, pages 2017.
[14] Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. Proceedings of the IEEE International Conference on Com- puter Vision, pages 2950-2958, 2019.
[15] Jonathan Crabbé and Mihaela van der Schaar. Explaining Time Series Predictions with Dynamic Masks. International Conference Machine Learning. 38, 2021.
[16] Giang Nguyen, Daeyoung Kim, and Anh Nguyen. The effectiveness of feature attribution methods and its correlation with automatic evaluation scores. Advances in Neural Information Processing Systems, 35, 2021.
[17] R. Caruana, H. Kangarloo, J. D. Dionisio, U. Sinha, and D. Johnson. Case-based explanation of con-case-based learning methods. AMIA Symposim,, pages 212 215, 1999.
[18] Isabelle Bichindaritz and Cindy Marling. Case-based reasoning in the health sciences: What's next? In Artificial Intelligence in Medicine, volume 36, pages 127-135. Elsevier, 2006.
[19] Mark T. Keane and Eoin M. Kenny, How Case-Based Reasoning Explains Neural Networks: A Theoretical Analysis of XAI Using Post-Hoc popllanation-b-Exxmpae from a Survey of ANN-CBR Twin-Systems. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformaties), volume 1 11680 LNAI,
pages 155-171. Springer Verlag, 2019.
[20] Been Kim, Cynthia Rudin, and Julie Shah. The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification. Advances in Neural Information Processing Systems, 3:1952-19988 2015.
[21] Been Kim, Rajiv Khanna, and Oluwasanmi Koyejo. Examples are not Enough, Learn to Criticize! Criticism for Interpretability. Advances in Neural Information Processing Systems, 29, 2016.
[22] Karthik S. Gurumoorthy, Amit Dhurandhar, Guillermo Cecchi, and Charu Aggarwal. Efficient Data Representation by Selecting Prototypes with Importance Weights. Proceedings IEEE International Conference on Data Mining, ICDM, pages 260-269, 2017.
[23] R. Dennis Cook and Sanford Weisenberg. Residuals and influence in regression. New York: Chapman and Hall, 1982.
[24] Pang Wei Koh and Perey Liang. Understanding Black-box Predictions via Influence Functions. 34th International Conference on Machine Learning, ICML 2017, 5:2976-2987, 2017.
[25] Amirata Ghorbani and James Zou. Data Shapley: Equitable Valuation of Data for Machine Learning. 36th International Conference on Machine Learning, (CML.2019) pages 4053-4065, 2019.
[26] Amirata Ghorbani, Michael P. Kim, and James Zou. A Distributional Framework for Data Valuation. arXiv, 2020.
[27] Been Kim. Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). 35th International Conference on Machine Learning, ICML 2018, 6:4186-4199 2017.
[28) Nicolas Papernot and Patrick McDaniel. Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning. arXiv, 2018
[29] Chih-Kuan Yeh, Joon Sik Kim, Ian E. H. Yen, and Pradeep Ravikumar. Representer Point Selection for Explaining Deep Neural Networks. Advances in Neural Information Processing Systems, pages 9291-9301, 2018.
12

---

[30] Mark T. Keane and Eoin M. Kenny. The Twin-System Approach as One Generic Solution for XAI: An Overview of ANN-CBR Twins for Explaining Deep Learning. arXiv, 2019.
[31] Surveillance Research Program National Cancer Institute, DCCPS, Surveillance, epidemiology, and end results (seer) program. www.seer.cancer.gov.
[32] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142, 2012.
[33] Eric Wu, Kevin Wu, Roxana Daneshjou, David Ouyang, Daniel E. Ho, and James Zou. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nature Medicine, 2021.
[34] Prostate Cancer UK. Cutract. www.prostatccanceeuu.orgg
[35] Pan Ji, Tong Zhang, Hongdong Li, Mathieu Salzmann, and Ian Reid. Deep Subspace Clustering Networks. Advances in Neural Information Prucessing Systems, pages 24-33, 2017.
[36] Rushil Anirudh, Jayaraman J. Thiagarajan, Rahul Sridhar, and Peer-Timo Bremer. MARGIN: Uncovering Deep Neural Networks using Graph Signal Analysis. arXiv,, 2017.
[37] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: extension of MNIST to handwritten letters. arXiv, 2017.
[38] Alberto Abadie. Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects. Journal of Economic Literature, 2020.
[39] Alberto Abadie, Alexis Diamond, and Jens Hainmueller. Synthetic control methods for compar- ative case studies: Estimating the effect of California's Tobacco control program. Journal of the American Statistical Association, 105(490):4935055. 2010.
[40] Susan Athey, Mohsen Bayati, Nikolay Doudchenko, Guido Imbens, and Khashayar Khosravi. Matrix Completion Methods for Causal Panel Data Models. arXiv, 2017.
[41] Muhammad Amjad, Devavrat Shah, and Dennis Shen. Robust Synthetic Control. Journal of Machine Learning Research, 19(22):1-51, 2018.
[42] Jennifer Gordetsky and Jonathan Epstein. Grading of prostatic adenocarcinoma: current state and prognostic implications. Diagnostic Pathology, 11(1), 2016.
13


Reviews:
[{"review_id": "oFxKqfPnS44", "reviewer": "Reviewer_h3ev", "summary": "The main contribution of this paper is SimplEx: a method that aims to explain a test prediction by decomposing it into the weighted sum of nearest neighbors (from a user-specified corpus) in the latent space. They claim this method is more precise and robust than existing explanation-by-example-comparison methods (namely, KNN-like explanations) and distinguish this method from (often gradient-based) feature saliency methods. The paper builds off work that try to explain model predictions that use comparisons in latent variable space (e.g., Concept Activation Vectors, Deep KNN). SimplEx works PROVIDED the model has a hidden layer that linearly maps to the output (this is the layer that is used for explaining predictions). \n\n- This constraint is significant for accurate weighting of examples. Note that if this is not satisfied by the model (say for instance a softmax is added right before the final output), it is possible to consider an earlier layer that satisfies the linear-mapping constraint as the output\n\nNote too that if the latent space spanned by the user uploaded corpus does not contain the latent representation of the test prediction of interest (it likely will not), the test's latent is projected into the space. This means that it is possible to use this method on any input, regardless of the corpus. We can then measure how good this approximation is by calculating the residual between the corpus span and the test latent.\n\nWe can then \"interpret the shift in latent space as resulting from a shift in the input space.\" This is the crux selling point of the paper, and it is accomplished using integrated (and projected) jacobians. This technique also lets us see the contribution of a feature from a corpus example.\n\nThe authors then proceed to (briefly) characterize their model on MNIST against dKNN inspired methods, and show a use case on a clinical risk dataset.\n\nThey claim the following novelty:\n\n1. Freedom to choose comparison corpus, no need for this to equal the training set\n2. The instance-level decompositions are valid in latent and output space, and are more robust than other explainability methods", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of this paper is SimplEx: a method that aims to explain a test prediction by decomposing it into the weighted sum of nearest neighbors (from a user-specified corpus) in the latent space. They claim this method is more precise and robust than existing explanation-by-example-comparison methods (namely, KNN-like explanations) and distinguish this method from (often gradient-based) feature saliency methods. The paper builds off work that try to explain model predictions that use comparisons in latent variable space (e.g., Concept Activation Vectors, Deep KNN). SimplEx works PROVIDED the model has a hidden layer that linearly maps to the output (this is the layer that is used for explaining predictions). \n\n- This constraint is significant for accurate weighting of examples. Note that if this is not satisfied by the model (say for instance a softmax is added right before the final output), it is possible to consider an earlier layer that satisfies the linear-mapping constraint as the output\n\nNote too that if the latent space spanned by the user uploaded corpus does not contain the latent representation of the test prediction of interest (it likely will not), the test's latent is projected into the space. This means that it is possible to use this method on any input, regardless of the corpus. We can then measure how good this approximation is by calculating the residual between the corpus span and the test latent.\n\nWe can then \"interpret the shift in latent space as resulting from a shift in the input space.\" This is the crux selling point of the paper, and it is accomplished using integrated (and projected) jacobians. This technique also lets us see the contribution of a feature from a corpus example.\n\nThe authors then proceed to (briefly) characterize their model on MNIST against dKNN inspired methods, and show a use case on a clinical risk dataset.\n\nThey claim the following novelty:\n\n1. Freedom to choose comparison corpus, no need for this to equal the training set\n2. The decompositions are valid in latent and output space\n\n# Main Review\n\nProvide a full review of the submission, including its originality, quality, clarity, and significance. See [https://neurips.cc/Conferences/2021/Reviewer-Guidelines](https://neurips.cc/Conferences/2021/Reviewer-Guidelines) for guidance on questions to address in your review, and /faq for how to incorporate Markdown and LaTeX into your review.\n\nFirst, a brief note on Claim 1 — letting users choose the corpus of examples to decompose model predictions into is not particularly novel. The mentioned dKNN method could be easily used with any user-specified corpus, as the authors do in their \"Experiments\" section.\n\nThe submission is well written, and it dives deeply into the problem of understanding model prediction as a combination of human-interpretable examples AND features. This reviewer is not aware of other literature that comprehensively bridges these two methods of explainability, and I believe many fields in AI Interpretability can utilize or be inspired by this effort.\n\nAs it stands, the authors have this reviewer convinced that this method is useful in explaining many simple Classification problems. \n\n- Authors show that this method is better than using KNN to explain, since we don't want to include irrelevant corpus examples.\n- Authors show that it can be used to inform users if the prediction is likely correct or not by relating it to examples in the corpus that were either correct or not.\n- Authors show this working on both tabular data and image data (although interpreting image features is a bit more challenging)\n\nIn its current state, the submission lacks:\n\n1. Accompanying code to play with results (authors have reported it will be published eventually)\n2. Stories showing this method's effectiveness on problems beyond simple classification and architectures beyond established MLP and CNNs.\n3. Quantitative (and even qualitative) comparisons to existing methods. For example, showing this method working on MNIST is actually rather boring, since no comparable results from other saliency methods are shown.\n    - Additionally, this could take the form of a quick user study involving real doctors/nurses on the provided clinical risk dataset.\n\nI also imagine this method is significantly slower than comparable ones (like dKNN) since it requires calculation of the integrated jacobian for every test example. This would then require larger compute to explain a large batch of predictions at once. It would be good if the authors included some performance metrics in their paper.\n\n### Additional Comments\n\n1. The introductory paragraph uses large, sweeping sentences that are both unprofessional in an academic paper and distracting from the point. Please reduce the wordiness.\n2. Please check the paper for grammatical English errors. There are many instances where plural nouns are incorrectly substituted by their singular form, and several sentences early in the paper that I had trouble understanding:\n    - \"paterns\", \"exaplanations\", \"accesible\", \"developped\"\n    - [L188] \"we do no longer need\"\n    - [L213] \"an a corpus example\"\n3. Figures with screenshots of tabular data are difficult to read without zooming in.\n4. I STRONGLY recommend choosing a color scheme other than green/red for Figure [3,7,8] for accessibility reasons. Consider a secondary encoding, or solutions proposed [here](https://www.visualisingdata.com/2019/08/five-ways-to-design-for-red-green-colour-blindness/#:~:text=The%20pink%2Dred%20through%20to,green%20hues%20used%20by%20default.) or [here](https://www.visualisingdata.com/2015/11/colour-swatch-alternatives-to-green-and-red/)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The method presented in the paper is described well, and the included examples (particularly of patient classification) show that this method can be appealing to those users of AI models that want to understand why a model makes certain predictions. Their claim that this method can generalize across models and datasets gives it even more potential impact. This reviewer is particularly intrigued by their proposition for future work — using this method to understand latent representations in unsupervised learning.\n\nHowever, the authors themselves did not conduct experiments to show all their claims. They said it can work on regression problems, but showed no examples of it. There are no experiments comparing the results of SimplEx decompositions to feature saliency methods. There are no user studies showing that this would be helpful in the real world. And, while reporting performance metrics on interpretability methods is not the standard, I believe it would be good to discuss this point.\n\nI believe this paper deserves to be included in the NeurIPS proceedings, but its story could be improved by addressing the above.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "YtYMLD28lRW", "reviewer": "Reviewer_Zgwu", "summary": "In this paper, the authors propose to explain the behavior of black-box neural networks by approximating a top-level hidden layer (no non-linearities afterwards, except possibly softmax) as a linear combination of the hidden layers obtained from different examples. These examples may come from the training data, but it is not required. The contribution of each input feature is also evaluated using an extension of integrated gradient across multiple dimensions.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While explaining neural networks through a combination of examples was introduced earlier, the ability to use a corpus that differs from the training data is, to the best of my knowledge, novel (unless I missed some important work). The integrated Jacobian is a fairly straightforward extension of integrated gradients to multiple dimensions.\n\nIt is great that the approach can be employed with a variety of corpora for explainability. However, the authors should also have evaluated the quality of corpus decomposition (section 3.1) when the training data is used.\n\nThe experiments cover example decomposition reasonably well, but pay relatively little attention to the feature explanations (using the integrated Jacobian). The authors show the contribution of the features in a few figures, but there is very limited discussion.\n\nIn addition to techniques used for black-box model explainability, I would suggest that the authors also describe approaches to directly make models more interpretable, such as glass-box models.\n\nThe paper is generally fairly clear, although there are numerous typos: L5 centred, L44 paterns, L96 example (should be plural), L97 accesible, L98 exaplanations, and likely others.\n\nThe paper has the potential to be significant as explainability can be an important consideration for the release of machine learning products. The methods are also fairly simple, which could accelerate their adoption by machine learning researchers and practitioners.\n\n[Post author response] After reading the other reviews and the author response, I have increased the score I assigned to this submission.\n\nMy remark about using the training data was unclear. I meant using the entire training data as the corpus for SimplEx decomposition. This point was partially addressed in the response to reviewers QnMz and coXq with corpora up to 1000 examples.\n\nThe additional experiment in the author response helps justify the use of the integrated Jacobians over integrated gradients (although an even more thorough exploration would be preferable).\n\nI am also satisfied by the authors' commitment to discuss some related work in more detail.\n\nAnother typo I saw when reading the paper again: L249 \"bellow\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Especially for sensitive data, such as in the medical domain, the corpus examples need to be carefully anonymized.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "P6pC7FnSkO0", "reviewer": "Reviewer_QnMz", "summary": "This paper introduces SimplEx, a method that approximate the hidden representation of a test example using the linear combination of hidden representations of examples in a pre-defined corpus. Further, the hidden representation of the test instance is associated with each input feature of these examples in the corpus. The paper demonstrates the quality of approximation of SimplEx (Sec 3.1, SimplEx outperforms kNN baselines) and demonstrate its practical utility in the experiment of predicting cancer mortality (Sec 3.2) by allowing model users to interpret model predictions and determine whether to trust the prediction. Detailed math derivation and additional experiments with MNIST and time series data are included in the appendix.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- A novel framework to interpret neural classification models with minimal assumptions on the model architecture.\n- SimplEx bridges attribution-based (feature-based) interpretation methods and example-based interpretation method.\n- The intuition of SimplEx is reasonable and mimics human learning process (by retrieving and comparing to past examples).\n- Writing is coherent and self-contained. Illustrations are clear.\n\nWeakness:\n- It is questionable whether introducing weights (w^c) assigned to examples from the corpus improves interpretability empirically. The work may benefit from additional human evaluation, e.g., showing interpretation produced by SimplEx (Fig. 7) and produced by kNN Uniform/Distance/Random (hard-coding w_c).\n- The SimplEx method may be computationally expensive (precompute the projected Jacobian for every instance in the corpus, perform a convex optimization procedure for every test instance). Some discussion on this topic would be helpful.\n\nQuestions:\n- The assumptions in the Sec 3.2 use case is very strong — The doctors are required to withhold 1000 labeled examples to use SimplEx. Is this a hard constraint? How does the explanation quality change when fewer datapoints are used.\n- I find it hard to understand Fig 2 and the related experiment. Does the figure suggest the patients with top 100 largest residues are supposed to be UK patients in the ideal setting (as their representations should be different from those of USA patients)? Does this suggest SimplEx can also be used for outlier/anomaly detection?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors describes assumption 2.1 as the main limitation to their approach. My concerns and questions are placed in the \"main review\" section.\nI believe the work does not have foreseeable negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "IvECYVfK7dZ", "reviewer": "Reviewer_coXq", "summary": "This paper introduces SimplEx, which is a method to post-hoc explain a model prediction by computing from a user-defined set of examples, how the set of examples decompose the prediction. For example, the clinical risk prediction of a patient can be explained by a set of other patients and how much each patient influenced the prediction. \n\nContributions\n-\tA post-hoc explanation method, called SimplEx, were users can choose the set of examples from which the explanation should be constructed. This is advantageous when the training set is not available or a user wants customized explanations, e.g. a doctor wanting to understand a predictions in terms of other patients she/he knows.\n-\tSimplEx combines the ideas of feature saliency and example-based explanations. For each example, the method computes the percentage of this example contributing to explaining the prediction as well as which features of this example were important.\n-\tA latent-space generalization of Integrated Gradients [10]\n-\tExperimental evaluation that show SimplEx to be more precise and robust compared to baselines.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n-\tThe paper is well written and easy to follow\n-\tThe method is interesting and supported by experimental results\n\nWeaknesses\n-\tThe paper leaves some natural questions open (see questions below).\n-\tLine 170 mentions that the corpus residual can be used to detect an unsuitable corpus, but there are no experiments to support this.\n\nAfter authors' response\nAll the weakness points have been addressed by the authors' response. Consequently I have raised my score. In particular:\n- The left open questions have all been answered.\n- There indeed is an experiment to support this, thanks to the authors' for clarifying this, that connection was not clear to me previously.\n\nQuestions\n-\tLine 60: Why do you say that e.g. influence functions cannot be used to explain a prediction? The explanation of a prediction could be the training examples whose removal (as determined by the influence function) would lead to the largest score drop for a prediction.\n-\tHow does the method scale as the corpus size or hidden dimension size is increased?\n-\tWhat happens if a too small corpus is chosen? Can this be detected?\n-\tWhat if we don’t know that a test example is crucially different, e.g. what if we don’t know that the patient of Figure 8 is “British” and we use the American corpus to explain it? Can this be detected with the corpus residual value?\n-\tIn the supplementary material you mention how it is possible to check if a decomposition is unique. Do you do this in practice when conducting experiments? How do you choose a decomposition if it is not unique? What does it imply for the experiments (and the usage of the method in real-world applications) if the decomposition is not unique?\n\nTypos, representation etc.\n-\tLine 50: An example of when a prototype model would be unsuitable would strengthen your argument.\n-\tFootnote 2: “or” -> “of”\n-\tLine 191: when the baseline is first introduced, [10] or other references would be helpful to support this approach\n-\tLine 319: “the the” -> “the”\n-\tLine 380: “at” -> “to”?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A broader impact section could be added. In a separate section (e.g. supplementary material), there could be an explicit discussion on when the method should not be used, e.g. as shown in Figure 8, the American corpus shouldn’t be used to explain the British patient. Also see last question above – what if we don’t know that the patient is British? Can this be detected? This should also be discussed in such a section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for explaining machine learning models by decomposing the continuous representation of a test example into a combination of examples from a given prototype corpus. 

Reviewers find the method convincing, powerful (bridging two large framework of explainable ML: feature-based and example-based), well presented and well demonstrated. The authors have thoroughly addressed all outstanding concerns in the discussion period, including a small user study. I strongly urge the authors to take the time to implement the corresponding improvements in the manuscript.
Decision: Accept (Spotlight)