Beyond Examples: Constructing Explanation Space for Explaining Prototypes
Authors: Hyungjun Joo
Seokhyeon Ha
Jae Myung Kim
Seungyub Han
Jungwoo Lee
Venue: ICLR 2022 Submitted (2022)
Keywords: Interpretable machine learning
XAI
Uncertainty
Prototype-based classification

Abstract: As deep learning has been successfully deployed in diverse applications, there is ever increasing need for explaining its decision. Most of the existing methods produced explanations with a second model that explains the first black-box model, but we propose an inherently interpretable model for more faithful explanations. Our method constructs an explanation space in which similarities in terms of human-interpretable features at images share similar latent representations by using a variational autoencoder. This explanation space provides additional explanations of the relationships, going beyond previous classification networks that provide explanations by distances and learned prototypes. In addition, our distance has more intrinsic meaning by VAE training techniques that regulate the latent space. With user study, we validate the quality of explanation space and additional explanations.

Under review as a conference paper at ICLR 2022
BEYOND EXAMPLES: CONSTRUCTING EXPLANATION SPACE FOR EXPLAINING PROTOTYPES
Anonymous authors Paper under double-blind review
ABSTRACT
As deep learning has been successfully deployed in diverse applications, there is ever increasing need for explaining its decision. Most of the existing methods produced explanations with second model that explains a given black-bos model, but we propose an inherently interpretable model for faithful explanation. Our method constructs an explanation space in which similarities in terms of human- interpretable features at images share similar latent representations by using a variational autoencoder. This explanation space provides additional explanation of the relationship, going beyond previous classification networks that provide explanation with distances and learned prototypes. In addition, our distance has more intrinsic meaning by VAE training techniques that regulate a latent space. With human evaluation, validate the quality of explanation space and additional explanations.
INTRODUCTION
"Erplonation" is essentially based on understanding. When a fact is in doubt, an explanation is provided to resolve question. Therefore, explanation must be human-interpretablle and reliable. An explanation that is not human-interpretable does not resolve the issue, and an explanation that is not reliable may create confusion. In practice, it is difficult to find an explanation that perfectly satisfies even one condition. In this situation, when an insufficient explanation is given, use may wonder why the explanation is given. We aim to address this question by providing a reliable explanation and providing additional explanations for given explanation in this paper.
With advances in deep learning, its performance improves rapidly. Advances have increased the use of deep neural networks (DNNs), which is used in areas where model decisions significantly affect people such as healthcare (Caruana et al.) 2015). radiation (Reardon) 2019), and autonomous driving. As result, questions about the rationality of model predictions for industrial and social use have been raised, and the call for explanation growing (Goodman Flaxman, 2017 Doshi- Velez
& Kim 2017). Since most DNNs black-bor box models, explanation of model prediction is often
provided by a second model that explains a given black- box model (Ribeiro et al.) 2016b Chen et al.,
2016b]CChen
2018a: Lundherg & Lee 2017 Guo al. 2018). This second model is called post-hoe interpretable model, which has been used for flexibility (Ribeiro et al. 2016a). However, this can lead people to misunderstanding the explanation because it is not offered by the model that provides the prediction. Therefore, as Rudin (2019) claimed, it is necessary to use an inherently-interpeetal to accurately explain DNNs without any mistakes, which results from an architecture to provide intrinsic explanation.
model in order
We build an inherently interpretable model, by referring to methods that generate prototypes and conduct classification using Buclidean-distame in a latent space (Li et al., 2018; Chen et al.. 2019). These prototype-base -aased explanation methods provide explanations with examples and distances and show efficacy in diverse domains. However, prototype-base methods are powerful but not enough to understand a model prediction fully, as explanations must be both numaa-interpretable and reliable. As most of the existing prototype-based explanation models suffer from the problem that similar latent representations do not guarantee similarity in shape, and distances often provide explanations that are not human-intee iitraaprteetrpebblee... Users can raise doubts, but have choice but accept the given distances only as numerical values. Since there is no way to verify how distances are constructed, the reliability of the explanations is compromised.

---

Under review as a conference paper at ICLR 2022
2
3
O
6
Input
6
4
2
8
O
2
(4) Explanation given by prototypes and distances
8
8
2
2
2 2
2
2
2
2 2
2
2
8 8
9 9 9 9
8
8
2
9 5
8
8
2
2
2
2222
2
2
2
2
2
2 8
9 9 4 4
Prototype:
Prnotypes: Chos
2
2
2
9
6 6
2
2
2
2
2
5
5
5
5
2
2
2
2
8
9 4 4
6
2 2
Inpur
Procotypes
Protocypes
Prototyp:: Class
(b) Relationships between imput and prototypes
(c) Relationships between prototypes
(4) Prosotype distribution visualization
Figure I: Basic explanation: (a) Learned prototypes that represent the training dataset distances between the input and prototypes. Additional explanations(b-d): (b) Relationships between input and prototypes. Bottom of the figure explains that the explanation space enables us to see how input 9 gradually becomes prototype 4. (c) Relationships between prototypes. Providing gradual changes from prototype to another prototype that explains how model discriminates prototypes. (d) Prototype distribution visualization with two dimensions based standard deviation demonstrating that prototypes are not luckily chosen.
To solve these problems, construct explanation space that provides gradual explanations and reliable distance. By using VAE, a latent space can be visualized as an image that users can recognize. We also use VAE training techniques to regulate latent space ensure latent representations share similarities in appearance. This means that distance in the latent space can have more "intrinsic meaning". We call the latent space obtained through the process the explanation space. Our model provides additional gradual explanation as in Figure Mcanwhile, the prototypes and the distances between the input data and prototypes are given basic explanation Figure aa). Through these relationships, users are able to understand how distances are composed. Instead of providing the distances between the input and prototypes, it is possible to better understand a model by showing which features are changing in terms of relationship. Therefore, our proposed model allows users to interpret distances more reliably.
In this paper, developed the model that performs classification through distances between prototype distributions and the encoded input distribution the latent space of the Variational Autoencoder (VAE)Kingma) & Welling 2014). After VAE encodes input data in Gaussian distribution form, compare the encoded input distribution and generated prototype distributions through four measures with various number of prototype distributions provide explanations with high classification accuracy. Furthermore, we use the method (Sensoy et al. (0188) of placing Dirichlet distribution on the class probabilities improve accuracy by reducing the uncertainty of the model.
Contributions. Our key contributions are as follows:
By constructing the explanation space, we can visualize additional gradual explanations. Through gradual explanations, relationships are provided rather than just providing each prototype image.
By our dense explanation space approach using VAE, similar latent representations tend to resemble each other in appearance. Therefore, distance explanation is more meaningful because similarity is guaranteed.
We present method find a baseline number of prototypes to describe the training dataset. In experiments, we show changes in accuracy and explanations of the model by varying numbers of prototypes.
2

---

Under review as a conference paper at ICLR 2022
2 RELATED WORK
Explanation Methods for DNN While there are diverse suites of methods to explain DNN, it can be categorized based on the access of the model's internal information and how they provide explanations (Jeyakumar et al. (020). The model-agnosti method is to approximate the relationships between input data and output decision while treating the model as a black-box. There are methods to create a local explanation (Ribeiro et al. 2016b] Chen et al. 2018a Lundberg & Lee| [017]] [hen jet al., 2018b) by selecting features that play an essential role in making decisions and methods providing an interpretable model through a global approximate (Guo et al. 2018). On the other hand, in knowing the internal information of the model, the model-transparent method provides explanations by saliency map using gradient in classifier (Simonyan al.] 2014; Smmikovv et al. Sundararajan et al., 2017). class activation map using global average pooling (Zhou et al. Selvaraju al. 2017). and feature map based on attention (Liu al., 2020). The third method
2017
2016
provides explanations by examples capturing relationship between test input and the model's decision. Considering that examples are based on training data (Bien & Tibshirani] 2011 Koh & Liang 2017
Ankk& & Plister 2020) or learned prototype data (Li et al. 2018 et al. [019]] Hase et al. 2019
our work is closely aligned with these prototype-based classification models. The example based
method shows its efficacy to explain complex concepts in psychology (Aamodt & Plaza 1994] and
education (Renkl) 2014) domains, and is also accepted by the general public as the most preferred
explanation method to explain DNN (Jeyakumar et al. 2020).
The following three prototype-hase based classification methods are similar to our method in that they use the distance between the encoded input and prototypes for classification and explanation. Li et al. (2018) learn and visualize the prototypes for the entire data through autoencoder, Chen et al. (2019) go further from here, characterizing training image patches as class-specific prototypes for partial explanations, while Hase et al. (2019) feature a hierarchical demonstration of prototypes through related groups for non-flat explanations. As most of prototype- based explanation methods are based on Li et al. (0118), we use it as a baseline model. Our framework goes further from these methods, expands prototypes from vectors to distribution in latent space and constructs explanation space for diverse explanations with reliability. In addition to prototype-base based methods, there are several methods that use the distance in latent space understand the internal behavior of model to
provide explanation using VAE (Antorán et al. 2021 O' S'Saughnessy et al. 2020 or GAN Lang
ct al. 2021
Yang et al. 2021
Uncertainty in deep learning The largest body of research on estimating uncertainty in deep learning is Bayesian neural networks (Neal 2012] Gal Ghahramani] 2015 Kendall & Gal 2017). To obtain predictive uncertainty, Bayesian neural networks use prior distributions on model parameters and infer the posterior distribution. Although exact inference in Bayesian methods is intractable, a range of approximate techniques has been proposed, such as variational inference (Kingma et al.)
2015 Gal & Ghahramani (016) and stochastic gradient hamiltonian monte carlo (SG-HMC) (Chen ct al. 2014). However, in practice, these Bayesian methods are outperformed by Deep Ensembles (Lakshminarayanan et al., 2017). This non-bayesian technique uses multiple neural networks trained from different initializations and averages their predictions as the model output. It is simple and effective but requires high computational costs at training and test time. Another non-bayesian technique is based on evidential deep learning (Sensoy et al., 2018 Amini et al. 2019), which use to estimate uncertainty. These methods use a single deterministic neural network and apply prior distributions on model predictions, not model parameters.
3 METHODS
Our model consists of three parts: (i) VAE layer, (ii) Distance measuring layer (iii) Uncertainty layer, as shown in F The VAE layer is used to encode input into form of distribution and decode the values obtained at prototype distributions and relationships with recognizable image data. The distance between prototype distributions and encoded input distribution is used as input for one cally-connected layer to create inherently interpretable classification model. Moreover, since classification is performed with distances in the latent space of VAE, we use an uncertainty layer to increase accuracy.
3

---

Under review as a conference paper at ICLR 2022
VAE layers
Distance measuring layer
Uncertainty layer
Input image x
Pully-connected layer h
Encoder
Az
a2
022
Uncertainty calculation
:
Decoder
Hm
logits 1
Reconsiructed image &
Figure 2: Overall model architecture. The entire model trained end-to-end. Since the uncertainty
layer was used reduce uncertainty in the training process, classification proceeds through logits. The explanation is provided by showing the learned prototypes, the distance between the prototypes and Ma, and their relationship each other through the latent space.
3.1 USING VARIATIONAL AUTO ENCODER
Let D = (X,Y] } be the training dataset consists of images x RN and labels y E (I...., KY K.. The encoder f g(z) x) of the VAE maps an input image to Gaussian distribution d ~ N(H.. (4z,, o= E 1 which reparameterized to generate latent vector Rdd. After that, the generated z is reconstructed in the form of image x g(z) through decoder g p(x(z). Un- like autoencoder, VAE regulates the latent space created by the encoder as isotropic unit Gaussian (p(z) (0, I)), which is why used VAE feature extractor. By regularizing latent space and sampling latent vectors from distribution, VAE constructs appropriate dense latent space that creates similar latent representations from similar-shaped images, which plays essential role in the interpretability of our method described in Section 4
Furthermore, our model leams prototype distributions dm , d; ~ NN(H4 of?) E RR and visualizes 14; as prototype. For that, the
independent multivariate Gaussian distri-
butions D = {dl, structures of disentangled VAE are applied in order disentangle the explanation components prototype distributions through dimensions. A method of encouraging the independence of represen- tation distribution q(z) by lowering the KI. divergence between ((z) and 7(z) == II; ((2)), ). and methods of lowering the KL divergence of encoder and isotropic unit Gaussian p(z) constrain the latent space stronger are additionally used with vanilla VAE. Applying these methods would result in the following objective:
-BKpx////)))//BBLL(xxxxx)))/)) -(K((())k
In this objective function, 3 and 7 are hyperparameters that control the power of the constraint terms.
3.2 CLASSIFICATION NETWORK USING DISTANCE
Once the input is encoded by latent distribution d,., logits based the distance from the prototype distributions are produced for the classification. This process divided into (i) measuring the distance between learned D (d,, dz, ... d } and d, in the latent space and (ii) producing the logits by measured distance using the fully-connected layer for the classification.
Since prototype distributions D and encoded input d, both are in the form of Gaussian distribution, we use 2- Wasserstein distance (Givens et al.] 1984) as distance measure. The 2- Wasserstein distance W for multivariate Gaussian distribution N (14, E,), ( E2) Rd Σg IRd x IRd)) can
4

---

Under review as a conference paper at ICLR 2022
be computed as:
W = //1 P2ll + Tr(E1 + Ex
Since we assume the prototype distributions D to be independent and encoder / predicts diagonal covariance matrices, E, diag(o!..... .....,(,)) and the above equation simplified as:
W Σ 10d p2)² (o} -
Three other measurements KI KL-divergence (KLD), Jensen-Shannon divergence (JSD), Jensen-Tsallis distance (JTD) (Choi et al. 2019) are also experimented in Section 5.1
The distances between D and d used as input of fully-connected layer h to generate logits. The reason for classification by passing only one fully-connected layer is to construct a weight matrix. By constructing weight matrix using h, the model can numerically confirm the impact of each prototype distribution on class selection. This weight matrix gives interpretability to the model's s prediction even when the number of given prototype distributions m and the class K are different.
3.3 USING UNCERTAINTY LAYER
To repeat sampling could increase the uncertainty of classification based distances in a latent space. Therefore, by considering uncertainty, we also improve the classification performance by about 1% with the method (Sensoy et al., 2018) that uses the Dirichlet distribution to reduce the uncertainty of a model. In the training process, the uncertainty loss Luncentin calculated using the Dirichlet parameters obtained through logits 1. The detailed method for uncertainty layer is in Appendix A..5
After going through all of these processes, the final objective function is composed as follows:
y(x,y) Xunocerain (haaeerrtiinnhaaaaaaa (D, f(x))) + Cvve (x.
where Luncertain is the uncertainty classification term based on cross-entropy loss and D and d₂ is the VAE term that described at Section 3.1 Moreover, À is hyperparameter for adjusting the ratio between them.
CONSTRUCTING INTERPRETABLE EXPLANATIONS
4.1 RELIABLE EXPLANATION SPACE
Our method increases the reliability of the explanation through distance and the prototypes. As noted in Brendel & Bethge (019)), the interpretation of previous prototype-based classification methods is difficult because only the Euclidean distance between prototypes and latent representations is given, while having a similar latent representation does not guarantee that images share similarities in human-interpretable features. Therefore, by sampling with distributions and regularizing latent space through VAF, our model constructs dense explanation space that ensures similar latent representations share similarities in appearance. Figure 3 makes this more evident. Looking the visualized latent space by I-SNE (Van der Maaten & Hinton 2008), embedded data by the previous method tends to separate clusters, often make the case that the distance between two points the same class farther than the distance between two different class points, as shown in Figure 3(a). In contrast, the proposed model increases the reliability of explanation through distance by showing that each class is clustered together and exist close to each other that forms dense latent space. Furthermore, by showing latent representations contained within the prototype distribution d, as images, the prototype's 's reliability can also be increased by their similarities. As illustrated in Figure [(d), features within the distribution similar to the prototype μæ at basic explanation, demonstrating that prototypes are not coincidentally chosen.
We made quantitative comparisons through user study as well as qualitative comparisons to ensure the reliability of the explanation space. We measured the user's understanding through an interactive reconstruction experiment (Ross et al. (02I) to show the reliability of the explanation space of our method. This experiment is to generate an image similar to randomly sampled image by changing
5

---

Under review as a conference paper at ICLR 2022
Lict al.
VAE
FactorVA
Figure 3: Visualizations for the latent space of FashionMNIST by three models. (a) Li et 2018 (b) Proposed model with VAE. (c) Proposed model with FactorVAE.
each dimension value of the latent space through a slide using the decoder of the models. As the user explores the latent space and generates an image similar to the sampled image, we can see how the configuration of the latent space is easy understand for the user. The evaluation was conducted through Amazon Mechanical Turk. order to improve the quality of the evaluation, all experiments conducted through 20 Mechanical Turk Masters. The success rate by model is shown in Table [ Detailed settings for experiments are in Appendix A.2
As Ross et al. (2021) concluded that higher success rate in- teractive reconstruction tasks shows improved interpretability. we concluded that the explanation space by our method improved inter- pretability because the success rate obtained through the interactive reconstruction experiment showed higher value than the baseline model. The distance value in the explanation space having more in- trinsic meaning because the explanation space created by our method showed improved interpretability compared to the baseline model. Therefore, the experimental results could be used as quantitative eval- uation to support that our method created a reliable explanation space.
Table I: Success rate by base- line model (Li et al. 2018). proposed model with VAE, FactorVAE
Success rate
ct al.
0.27+00.8
VAE
0.48110.1
Factor VAE
0.54.10.29
4.2 SELECTING THE NUMBER OF PROTOTYPE DISTRIBUTIONS
In the prototype-based explanation model, the number of prototypes is critical. As can be seen in the model structure, the number of prototypes affects the input of the fully-connected layer and inevitably affects accuracy. Moreover, since explanations are also provided through prototypes, it is important to set the number of prototypes. However, since there was no information to refer other than the number of classes determine the number of prototypes that can represent the dataset, the previous methods (Li et al.) 2018 Chen et al. [2099] Hase et al. 2019) specified the number of prototypes to an arbitrary and heuristic value determined by a human.
We propose method to produce the baseline number of prototype distributions in our framework. Since we have set the prototype distributions as independent Gaussian distributions rather than vectors, prototype distributions cover the latent space. Furthermore, prototype distributions representative of the whole dataset, it shares similarities with the Gaussian Mixture Model (GMM) in that the whole dataset is represented in the form of multiple Gaussian distributions. We choose an optimal number of components of the GMM in latent space as a baseline number of prototype distributions. We use the Bayesian Information Criterion (BIC), which is known be the most appropriate criterion for GMM (Steele & Raftery 2010). The experimental results for this can be seen in the Section 5
4.3 EXPLAINING THE RELATIONSHIP OF EXPLANATIONS
The explanations of our model are divided into basic explanation and explanations of the basic expla- nation. For fundamental explanations. visualized images of the learned prototypes and the distance between encoded input and prototype distributions are given as previous prototype-brat based explanation methods. To explain this basic explanation, we additionally provide two gradual explanations. These

---

Under review as a conference paper at ICLR 2022
explanations can be obtained because method constructs dense explanation space, unlike previous methods. As a result, it is possible to generate interconnection between latent representations as images. We represent changing elements by interpolating these latent representations and define them as relationships.
The relationships between inputs and prototypes can be visualized by model. In situations where distance is given for interpretable information, the factors that can reduce the distance are critical for understanding the model's prediction. Additionally, relationships between prototypes are given. Since the prototypes are representative of the whole dataset, relationships between prototypes show which features are needed change from one class to another class. This explanation can be used as basic information about how the model discriminates classes.
After the model is trained, basic explanation and explanations for basic explanation are visualized as follows. First, the prototype, provided as the fundamental explanation, is visualized by sending (4; of the prototype distribution through the decoder g as prototype. Next, the relationships between prototype μ4; and prototype Mj are visualized as sending 14; + (uz 14)r/k /k for 0, 0,1,...,4 k through the decoder g. which can controlled by increasing the value of for detailed explanation and decreasing the value of k for a brief explanation. Finally, the relationships between inputs and prototypes are provided in the same way, and visualizations of prototype distributions are based on values and values.
The disadvantage of our model comes from this visualization. Many training samples are required to generate explanation images. In addition, visualized images of our explanations are fundamentally generated through VAE, making it challenging to generate human- recoonizacenii images for complex datasets that are difficult for VAE generate the observable image. However, this is problem related to VAE's expressive power, and there are many high-resolution expressive methods (Lee et al.) 2020; Vahdat Kautz 2020 based on VAE, so applying these methods would solve these problems in future work.
5 EXPERIMENTS
Experiment details MNIST and Fashion-MNIST (Xiao et al. 2017) datasets are used for our experiments and splited by 0.9, 0.1 for training dataset and test dataset. For training our model, we use Adam (Kingma & Ba 2015) optimizer with learning rate of 0.0001 and weight decay of 0.00005, and set hyperparameter À to 50. To compare various VAE structures, we use vanilla VAE
(3 = 1, y = 0), 3-VAE (3 = 4, 7 0) (Higgins et al. (2017) and FactorVAE (3 = 1,
7 = 6.4) (Kim
Mnih 2018). We train 500 epochs with VAE and VAE for about 2 hours, and about 3 hours were used with Factor using single 1080 Ti GPU.
Table 2: Accuracy by measures (%) on Fashion-MNIST
VAE uncertain
Beta- VAE
Factor VAE
Measure
original
original
uncertain
original
uncertain
KLD
91.3+0.2
92.150.2
91.61:0.
92.2
91.2+:022
92.410.1
JSD
91.910.2
92,410.2
91.910.2
92.441..3
91.910.3
92.411.2
JTD
91.0110.1
92.110.2
91.41:0.2
92.1
92.110.2
92.011.3
WSD
91.8+10.1
93.0 + 0.1
92.2+0.2
92.6 0.2
91.3++0.2
92.8 + 0.1
5.1 CLASSIFICATION ACCURACY
The accuracy of our model on MNIST is 99.33 with providing visualized explanations in Figure [ Furthermore, shown in Table 2 our model achieved testing accuracy 93.00=1..137
on Fashion- Fashion-MNIST. This result is comparable to non-interpretable classification networks such as VGG16 (93.5%) and GoogleNet (93.7%) reported in the dataset website. Moreover, achieve higher accuracy than the baseline model's (Li et al. (2018) accuracy 91.03+0.32 that only provides basic explanation. We construct baseline model with same layers and same hyperparameters.
As shown in Table we conduct experiments with four measurements and three VAE structures. According measures, results of experiments using Wasserstein-distance show the highest accuracy
7

---

Under review as a conference paper at ICLR 2022
4.93
7.14
6.70
7.09
6.77
6.38
6.70
6.35
5.80
5.49
Input
6.62
6.85
7.63
4.19
6.23
Distance becomes färther
Prototypes
6.19
5.20
5.74
6.89
7,31
(a)
(c)
Figure 4: Explanations of classification network with 20 prototype distributions and FactorVAB (a) Prototypes and distance between input and prototype distributions in the latent space. (b)
Relationships between input and prototypes. (c) Relationships between prototypes.
through three VAE structures. In addition, it achieves the highest diversity and stability of the learned prototypes as it provides a usable gradient when distributions are supported on non-overlapping domains (Kolouri et al., 2019). This indicates that Wasserstein-distance is the most suitable mea- surement creating prototypes using distances between distributions. The role of the uncertainty layer to increase accuracy is also clearly revealed. In most of the 12 cases, accuracy increased when the uncertainty layer applied. Among three VAE structures, the vanilla VAE shows the highest accuracy, but FactorVAF. also shows no significant difference in accuracy. However, FactorVAF forms more reliable explanation space as Section 4 it used create Figures
5.2 PROVIDED EXPLANATIONS: BEYOND EXAMPLES
Our model generates explanations for prediction as Figure In Figure 4 a), for the black T-shirt input image, our model shows prototypes with the lowest distance having shapes of tops. On the other hand, the distances from the prototypes that looked like bags, trousers, and shoes are higher than 6.5, indicating that the prototypes and distance information provide accurate explanations. Furthermore, fully-connected layer is applied as a weight matrix corresponding to a similarity connection between these distances and logits, showing relevance between prototype distributions and classes. However, this explanation alone makes it difficult to understand why the distance from the prototypes in the same T-shirt class is different and whether the different distances came out even if the prototypes were in the form of dresses. Therefore, our method provides additional explanations for this basic explanation.
Additionally, our model provides the relationships between prototypes and inputs as an explanation of the factors that influenced distance, such as Figure b). For example, from relationships between the input and prototypes for class dress, the nearest prototype can be seen with only the thickness of the torso changing, followed by the next prototype with longer sleeves and the farthest prototype with no sleeves. Since prototypes with changes in the sleeves have similar distances, we can assume that the model sensitive to sleeves in the relationship between input and dress prototypes. The bigger the change in the sleeves of the clothes, the farther the distance becomes. In addition, relation- ships between prototypes are visualized as additional information for how the model discriminates prototypes, such as Figure c)..
We conduct a user study to show that the relationship helps us to understand model predictions. With the explanations from model, subjective comparison is made when (i) basic explanation (Figure 4(a)), and (ii) basic explanation and the relationship between input and prototype (Figure 4(a), (b)) given. For each set of explanations, we studied the results of correct/wrong classification. We prepare the following questions to check whether users were able to respond to factors that influenced model predictions. In the case of correct classification, the distance between prototypes within the same class. In the case of misclassification, we asked the factors that made the distance from the prototype of the wrong class close. The relationships with the input and following prototypes were given during the survey: (i) Correct classification: two prototypes of the same class as input. (ii) Wrong classification: the prototype closest the input among misclassified classes, and the prototype closest to the input among true classes. Additional information about the survey included Appendix A.3
ask the factors that affect differences in
8

---

Under review as a conference paper at ICLR 2022
90
Ac0,9)
80
75
VAE
Beta-VAL
FactorvAE
100
Prototype distributions number
Figure 5: Accuracy and visualizations of prototypes. (a) Classification aceuracy by changing numbers of prototype distributions (b) Visualization of prototypes pa(i = I,..., m) at number of prototype distributions m = [2, 10, 30, 100]
We conducted two surveys through Amazon Mechanical Turk with 50 Mechanical Turk masters. Each survey consists of 12 questions, 6 per correct classification and misclassification. The average re- sponse rate for answering the factor is in Table This result shows that participants were better able to answer factors for model pre- dictions when given the additional explanations of the relationship between inputs and prototypes. It shows that relationships enables us to better understand the factors that compute distances.
Table 3: Response rate with- out/with relationships
Without
With
Correct cls.
0.77
0.83
Wrong cls.
0.60
0.71
5.3 CHANGING THE NUMBER OF PROTOTYPE DISTRIBUTIONS
As mentioned above, chose the optimal number of Gaussian Mixture Model's components proper number of prototype distributions. We use the EM algorithm to fit Gaussian Mixture Model to the dataset on latent space. First, trained VAE encodes μ values of the entire dataset to place them on the latent space. Since our prototype distributions consist of independent multivariate Gaussian distributions, we use the Gaussian Mixture Model with each component having its own diagonal covariance matrix. As the first local minimum BIC score indicates strong evidence for the model (Fraley& & Raftery| 1998), we determine that the number of components at the first local minimum the optimal number of components. The number of components obtained is 20.3, 19.8, 23.8 on average in VAE, B- VAE, and FactorVAE Through these results, the baseline number of prototype distributions is set as 20. To verify this, experiment with numbers of prototype distributions m = 3, 10, 20, 30, 40, 60, 100] and the results are Figure 5 In Figure a()), the accuracy begins converge around 20. In addition, when the number of prototypes extensively slowly. In Figure btb) when the number of prototypes increases from a number smaller than the number of dataset classes, various appearances appear. However, when the number of prototypes increases excessively, similar appearance prototypes are inevitable. These results indicate that it better to set the number of Gaussian mixture model components as the number of prototype distributions.
small, the model converges
6 CONCLUSION
The main contribution of our paper is to further enhance human understanding of model predictions by providing additional explanations of relationships. Further to existing prototype-based explanation methods that generate explanations by prototype examples and distances, our model visualizes relationships between input and prototypes that can explain the factors relevant to distance. Moreover, due to creating a dense explanation space, the distance became possible to function as a numerical value that guarantees similarity, which led to increase reliability.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Agnar Aamodt and Enric Plaza.
Case-based reasoning: Foundational issues, methodological varia- tions, and system approaches, Al communications, 7(1):39-59, 1994.
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. arXiv preprint arXi:: 2019.
Javier Antorán, Umang Bhatt, Tameem Adel, Adrian Weller, and José Miguel HemándezzLLoato. Getting a CLUE: A method for explaining uncertainty estimates. In International Conference on Learning Representations (ICLR), 2021.
Serean Arık and Tomas Pfister. Protoattend: Attention-based prototypical learning. Journal of Machine Learning Research (JMLR), 21:1-35, 2020.
Jacob Bien and Robert Tibshirani. Prototype selection for interpretable classification. The Annals of Applied Statistics, pp. 2403-2424, 2011.
Wieland Brendel and Matthias Bethge. Approximating cnns with bag-of- log-of-local-feature models works surprisingly well on imagenet. In International Conference on Learning Representations (ICLR), 2019.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 2/th ACM SIGKDD international conference knowledge discovery and data mining, pp- 1721-1730, 2015.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: Deep learning for interpretable image recognition. Advances Neural Information Processing Systems (NIPS), 32:8930-8941, 2019.
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning explain: An information- theoretic perspective on model interpretation. In International Conference on Machine Learning (ICML), pp. 883-892. PMLR, 2018a.
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. L-shapley and e-shapley: Efficient model interpretation for structured data. arXiv preprint arXiv: 1eXivv1808.000 1808.026100 201 8b.
Tianqi Chen, Emily B Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In Proceedings of the 31st International Conference on International Conference on Machine Learning (ICML.. pp. 11-1683, 2014.
Yunho Choi, Kyungjae Lee, and Songhwai Oh. Distributional deep reinforcement learning with mixture of gaussians. In 2019 International Conference on Robotics and Automation (ICRA), pp. 9791-9797. IEEE, 2019.
Finale Doshi Delei-Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint ar arXiv: 1702.08602 2017.
Chris Fraley and Adrian Raftery. How many clusters? which clustering method? answers via model-based cluster analysis. The computer journal, (1(8)5578-588 1998.
Y Gal and Z Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In 33rd International Conference on Machine Learning (ICML), volume 3, pp. 1651-1660 2016.
Yarin Gal and Zoubin Ghahramani. Bayesian convolutional neural networks with bernoulli approxi- mate variational inference. arXiv preprint arXiv: 150.00258,, 2015.
Clark R Givens, Rae Michael Shortt, et al. A class of wasserstein metrics for probability distributions.
The Michigan Mathematical Journal, 31(2):231-24) 1984.
Bryce Goodman and Seth Flaxman. European union regulations on algorithmic Docision-making and "right explanation". Al magazine, 38(3):50-57, 2017.
10

---

Under review as a conference paper at ICLR 2022
Wenbo Guo, Sui Huang, Yunzhe Tao, Xinyu Xing, and Lin Lin. Explaining deep learning models-a bayesian non-perretrric approach. Advances in Neural Information Processing Systems (NIPS), 2018:4514-4524, 2018.
Peter Hase, Chaofan Chen, Oscar Li, and Cynthia Rudin. Interpretable image recognition with hierarchical prototypes. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume pp. 32-40, 2019.
Irina Higgins. Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In 5th International Conference on Learning Representations (ICLR), 2017.
Jeya Vikranth Jeyakumar, Joseph Noor, Yu-Hsi Cheng, Luis Garcia, and Mani Srivastava. How can i explain this you? empirical study of deep neural network explanation methods. Advances in Neural Information Processing Systems (NIPS), 2020.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Proceedings of the 3 31xt International Conference on Neurul Information Processing Systems (NIPS), pp. 5580-5590, 2017.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning (ICML), pp. 2649-2658. PMLR, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations (ICLR), 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations (ICLR), 2014.
Diederik P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparam- eterization trick. In Proceedings of the 28th International Conference Neural Information Processing Systems (NIPS), pp. 2575-2583 2583, 2015.
International Conference on Machine Learning (ICML), pp. 1885-1894. PMLR, 2017. Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In
Soheil Kolouri, Phillip E Pope, Charles E. Martin, and Gustavo K Rohde. Sliced wasserstein auto- encoders. In International Conference Learning Representations (ICLR), 2019.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive
uncertainty estimation using deep ensembles. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS), pp. 6405-6416, 2017.
Oran Lang, Yossi Gandelsman, Michal Yarom, Yoav Wald, Gal Elidan, Avinatan Hassidim, William T Freeman, Phillip Isola, Amir Globerson, Michal Irani, al. Explaining style: Training a gan to explain classifier in stylespace. arXiv preprint arXiv: 104. 13369, 2021.
Wonkwang Lee, Donggyun Kim, Seunghoon Hong, and Honglak Lee. High-fidelity synthesis with disentangled representation. In European Conference on Computer Vision (ECCV), pp. 157-174. Springer, 2020.
Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 32, 2018.
Wenqian Liu, Runze Li, Meng Zheng, Srikrishna Karanam, Ziyan Wu, Bir Bhanu, Richard Radke, and Octavia Camps. Towards visually explaining variational autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8642-8651, 1, 2020.
Scott M Lundberg and Su-In Lee. unified approach to interpreting model predictions. Advances in Neural Information Processing Systems (NIPS), 30:4765 2017.

---

Under review as a conference paper at ICLR 2022
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Matthew R. O' O'Shaughnessy, Gregory Canal, Marissa Connor, Christopher Rozell, and Mark A. Davenport. Generative causal explanations of black-box classifiers. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems (NIPS), 2020.
Sara Reardon. Rise of robot radiologists. Nature, 576(7787):854 S54, 2019.
Alexander Renkl. Toward an instructionally oriented theory of example- haaeleebaased learning. Cognitive science, 3-31), 2014.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Model-agnostic interpretability of machine learning. arXiv preprint arXiv: 1606. 05386, 2016a.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference knowledge discovery and data mining, pp. 1135-1144, 2016b.
Andrew Ross, Nina Chen, Elisa Zhao Hang, Elena L Glassman, and Finale Doshi- Velez. Evaluating the interpretability of generative models by interactive reconstruction. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-211) 2019.
Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In Proceedings of the 32nd International Conference Neural Information Processing Systems (NIPS), pp. 3183-3193, 2018.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 2017.
Russell Steele and Adrian E Raftery. Performance bayesian model selection criteria for gaussian mixture models. Frontiers of statistical decision making and bayesian analysis, 2:113-130, 2010.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference Machine Learning (ICML), pp.
PMLR, 2017.
Arash Vabdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems (NIPS), 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1)) 2008
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprini arXiv: 077477777 2017.
Ceyuan Yang, Yujun Shen, and Bolei Zhou. Semantic hierarchy emerges in deep generative rep- resentations for scene synthesis. International Journal of Computer Vision, 129(5)144)--146. 2021.
Bolei Zhou, Aditya Khosla, Ágata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 2921-2929. IEEE Computer Society, 2016.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 MODEL. ARCHITECTURE DETAILS
In our proposed model, we use vanilla VAE, 3-VAE. and FactorVAE for the VAE layer. For VAE and B-VAE, we use Convolutional Neural network for the encoder and a Deconvolutionll Neural Network for the decoder. Detailed architecture explanation is provided in Table For FactorVAE. discriminator is additionally used for distinguishing whether the input was drawn from g(z) or g(z) = I ((z,) calculate KL divergence between ((z) and @(z). The discriminator made of 6 MLP layers with 1000 hidden units each and leaky ReLU is used for non-linearity, that generates 2 logits as outputs. We use Adam optimizer for discriminator with learning rate 0.0001 and weight decay 0.00005.
A.2 DETAILS FOR INTERACTIVE RECONSTRUCTION EXPERIMENT
We constructed the experiments in the same way in the paper Ross et al. (2021). Through the interactive reconstruction method shown in the paper, we evaluate the models: Li et al.) (2018). proposed model with VAE, and proposed model with FactorVAE. The evaluation through Amazon Mechanical Turk, and for a fair evaluation, everyone paid $3.75 per evaluation. In order improve the quality of the evaluation, all experiments were conducted through Mechanical Turk Masters.
conducted
The experiments
conducted with models trained through the MNIST dataset. Success considered the user-generated image and the sampled image had a similarity of at least 75%. In addition, it is designed that can skip the question only if they generate images for least 45 seconds. Experiments were conducted with 20 participants per model and 7 questions with random images were provided. An example interactive reconstruction experiment shown Figure
Working efficiently you can, try change the dials on the left align the pictures on right. As you get close aligmment, you may find helpful to overlay the two pictures by clicking the button below. Finally, you can also relabel the controls on the left by clicking the labels.
If you actively try solve the task for 45 seconds but cannot, you'll be able to skip it and move on without penalty.
Dis
<l
4
Ciz
x2
Cureet abgumant 9.2% Tarast aleument 75%
Lia
lis
Fix
Picture
Target
Click the Jabols 2enarn.
Overlay Pictures
Figure 6: Example of interactive reconstruction experiment.
13

---

Under review as a conference paper at ICLR 2022
A.3 USER STUDY ABOUT RELATIONSHIPS
The survey conducted after showing precautions the participants first. Participants asked to answer the factors that influenced the model's decision by looking at the given information. At this time, since the given information may be insufficient to judge the factor, two options were provided: the option of not knowing and the option of guessing. In the case of the participants who judged that they could guess factors, they were also asked to answer which factors they guessed. Consent form and example of survey without relationships is shown in Figure 7 Consent form and example of survey with relationships shown in Figure g
The surposo this ovalunte retabiiity by of dirterent toarning wigoitthme and their eutpors
eenen surneye, survyy games pieces intormanio cimpurs, probotypest The numbors witte bolow the prototypes are the distonce coroulated the model possitie.
repermmnnt
70
Iaper
your pretii recpppiioon
loast 18 years of ago to participott Your pa tooppation any part of this study is viluntary. You the right retuae any ropornsssio soions.
flogetiee withetes from the study any
Presstpee
Weooooooooono ondo 1d str betwece potitgpc trosem cass appoar difiwet?
associsted with your participatio mosaarc beyond everyday life, As rosearch part cipant, you will receive compo noathon you complioto the snang, the nowi raid with 1.0 You sot nicewvo onde paymont and of the study.
Figure 7: Consent form and example of survey without relationships.
"ho pose of this studs to avallato the intoppstabiiiiy of difforent moot koaming wkgesthhmw
"ho oxper imoot cxperimnn: infermation lngutio, prutes prototyynes
will conpiet of sonio survoss. The survey givos you throe oi oocs of weliian eddap- bnterme axpatloannd proticlguae rnpart
Al model, For quections, you can anowor as simply as possible.
prrtespation repertmeent
lnact 19 yoars apo participato. Your zor toopptttion in any part of study voluntary. You have the to rafuse parttolpeno anopethorn or withcrow from the study at any timo any repermassiioss
What twstnee made the iifanion on petheeen portttpprr of earw daas apper feeeet?
ricipann, your rocoivo "hhere no lin own risks associated with your car rioppation in chis rescarch beyo nd those everyday
complete stury, you will bee rewarded with bot You recerve code for paymment at undior the study.
P ancevr visi factora canises.
Figure 8: Consent form and example of survey with relationships.
A.4 DETAIL EXPLANATIONS FOR MEASUREMENTS
We use KL-divergence, Lensen-Shannov divergence (JSD), Jensen-Tsallis distance (JTD) and Wasser- stein distance for measuring distance between prototype distributions D {d), d2..... d and encoded input distribution dz.
14

---

Under review as a conference paper at ICLR 2022
Since it not described in detail in this paper, we provide additional explanations in this section.
For two multivariate Gaussian distributions p = N (11, E1), = (2) E R", RR, D₁ E R R²), KL-divergence is as follows:
ΣΣ log Σ
d + tr (Σ + (#2 m1) E1)"''""'(0 (12 mn)
(I(pll
(1)
From independent distributions D and encoded input distribution that has diagonal covariance matrix the above equation simplifies to:
Σ = .i.... ).
(01)² + (AG) 12)
1
log
(2)
2(02)2
2
Through this equation, with (p + 4) Gensen-Shanme divergence also simplifies in the same manner because Jensen-Shannon divergence defined as:
JSD(pl = ((pll)) + K Lg)lmmm)
(3)
Unlike JSD, Jensen-Tsallis divergence the Tsallis entropy instead of the Shannon entropy. The Tsallis entropy sp.x(p) is generalization of the Shannon entropy which becomes Shannon entropy when g converges to I and & equals In previous works, they specific Jensen-Tsallis divergence with and k 4 and refer to JTD since it satisfies the conditions of metric. In our condition, JTD is as follows:
(TD)) = Σ [1 + +N( - (N(uli (o)) + (o2)3) (4)
By using KL-divergence, JSD, and JTD as a distance measure, the proposed model learns prototypes illustrated in Figure 9 Using these metric for distance, the model accuracy is high enough, but the prototypes do not form observable images, which is contrary to our purpose of explaining the model predictions. This is because these three measures fail provide usable gradients when distributions are supported on non-overlapping domains. For this reason, the model fails to provide usable gradients when the prototype distributions and the embedded input distribution do not overlap, thus not forming proper latent space as shown in the Figure which makes prototypes unstable.
A.5 DETAILED ANALYSIS DUE TO APPLICATION UNCERTAINTY LAYER
In the process of forming the latent space in our proposed model, VAE formulates the reconstruction loss through the latent vector sampled from the embedded input distribution. Since classification was performed through the distance in the latent space, tried to improve the classification accuracy by lowering the uncertainty of the model. By adding the calculation process to logits without changing the structure of the model, it possible to predict uncertainty and increase the accuracy by making the model robust against adversarial perturbations. Furthermore, since the model is not trained to handle uncertain information, it not only increases accuracy but also makes similar features exist in similar positions in the latent space. The embedded dataset visualized with t-SNE is illustrated in Figure 10 showing that data of the same class forms more aggregated clusters in latent space when the uncertainty layer is applied. Through this latent space property, prototypes form more obvious observations that can express the whole dataset better, as Figure because prototype distributions are learned through distances at latent space.
For detailed explanations of the uncertainty layer setting in our model is as follows. Given a sample Xs, let there be one-hot vector with ground truth class j. then Yy and you = 0 for all k # j. and logit that comes through the distance measuring layer of the model. We compute the evidence as exponential of logit 4, Then use the evidence to derive the uncertainty probability
K/S, where total S; E)=(ees (eig 1), and parameters of Dirichlet distribution os est + for class
15

---

Under review as a conference paper at ICLR 2022
Table 4: VAE network architecture used for experiments with MNIST and FashionMNIST
VAE Encoder
VAE Decoder
Input x2x32 greyscale image
Input z E 10
Conv2d
Conv2d (256 channels, 1x1 kernels, stride pad 0)
(32 channels, 4x4 kernels, stride pad I)
BatchNorm, Leak)ReLU(0.2)
BatchNorm, Leaky
Conv2d
ConyTranspose2d
(32 channels, 4x4 kernels, stride pad I)
(64 channels, 4x4 kernels, stride pad 0)
BatchNorm, Lea))ReLU(0.2)
BatchNorm, Leaky)eLU(00)))
Conv2d
ConyTranspose2d
(64 channels, 4x4 kernels, stride pad
(32 channels, 4 kernels, stride pad
BatchNorm, (ea.))eLU((.2)
BatchNorm, LeakyRLLUO0))
Conv2d
ConyTranspose2d
(256 channels, 4x4 kernels, stride 1, pad 0)
(32 channels, 4x4 kernels, stride pad
BatchNorm, Lea))ReLU(0.2)
BatchNorm, LeakyReLU(0.2))
Conv2d
ConvTransposel) (1 channels, 4 kemnlls, stride 2, pad
12 channels, 1x1 kernels, stride I, pad 0)
Sigmoid
(6)
(c)
Askle boot
Bag
Snenicer
Shin
Sandal
Cost
Pullover
T-shinhop
(c)
Figure 9: Prototypes made with the proposed model using (a) KL-divergence, (b) JSD, and (c) JTD as distance measurements. Visualizations for their latent space by t-SNE (d) KI KI-divergence, (e) JSD, (f) JTD. Models are trained using vanilla VAE at the VAE layer with 1000 epochs.
k. Through the obtained parameter ask and S,, which is the sum of total evidence and class K, the cross-entropy loss changes as follows:
K
C. Σ (b)) - wb(ss))
(5)
Where v-(-) is digamma function, and after this process it has the same effect specifying the class of uncertainty. Also, we add regularizing term for misleading evidence. For predicted parameters og with true class j. misleading evidence is represented Oz for all k of j. With equation 5
16

---

Under review as a conference paper at ICLR 2022
Original
Lsing Uncertainty layer
(0)
(b)
Figure 10: T-SNE visualizations for the embedded FNsHionMNIST using three types of VAE for the
VAE layer. (a) vanilla VAE, (b) 3- 3-V,, (c) FactorVAE.
Original
Using
Uncertainty
layer
Figure 11: Visualized prototypes from proposed model with three types of VAE for the VAE layer. (a) vanilla VAE, (b) 3-VAE, (c) FactorVAE.
regularize them using KL divergence with the uniform Dirichlet distribution as follows:
A
Luncertain
L.
KL (p: a;) [D (p: (1,..., 1))]
where d is the annealing coefficient that starts from 0 and becomes 1 over training epoch t 50, D (p: (I...., 1)) is the uniform Dirichlet distribution, and &; y; + y:) O Oi; for removing non-misleading evidence.
A.6 ADDITIONAL RESULTS FOR MNIST EXPERIMENTS
As shown in Figure 12 a), prototypes with the shape of O are the closest to the input. Distances to numbers such as 2, 3, 8, 9, which have curved parts similar to O, are closer than the distance to numbers such from the input was similar. Further explanations are given by the relationships between input and prototypes and the relationships between each prototype, shown in Figure 12 b2),, (c). We infer
4, 7, which have sharp shapes. Moreover, in the same class. the distance
17

---

Under review as a conference paper at ICLR 2022
Prototypes
3
7
O
9
5
4
8
O
3.41
6.68
6.88
3.05
5.34
5.85
6.13
5.33
3.41
5.59
Input
2
9
8
3
5
5
7
6
2
4.88
5.23
4.93
5.14
5.67
6.75
7.87
6.93
6.84
5.06
Distanaces
3
3
5
5
5
5
6
3.05
Rotation
3.41
3 3
3
3
0
0
0
0
3 3
3
3
3
3
1
Thickness 1, Width 1
3333
3
2
2
2
22
5.67
50000.5550 5
5.5 5 5 5
333
3
4
4
4
4
4
6.75
0
0
0
0
5
5
33335
3
5
0-5
Smeart
3
333
33356
6
6
6
6
6.84
0
0
6
6
6
6
6 6 6 6 6
3
3
7
7
7
7
7.87
0
0
3
3
3
3
3
8
8
8
8
Smear
0-6
3
3
3
3
9
9
9
9
9
5.23
0 0 9
9
9.9
3
3
9
4
3
3
9
9
6.88
0
? 7 7 7 7 7 7
Prototype: Class 3
Prototypes: Class 0-9
0-9
9-7
(b)
Figure 12: (a) Basic explanation with prototypes and distance between embedded input and prototype distributions. (b) Relationships between input and prototypes. (c) Relationships between prototypes
the factors that caused the model to predict distance farther by the relationships between the input and the prototypes. Also, by examining the relationship between each prototype, we can find out how the model distinguishes the classes. If the explanation is insufficient, more detailed explanations can be obtained by visualizing more interpolation values between prototypes.
A.7 STABILITY: HOw SHOULD THE EXPLANATION RESPOND TO SIMILAR INPUTS?
Interpretable explanation models have provide consistent explanations for similar inputs for stability. This is because the reliability of explanations is not guaranteed if different explanations are provided for similar inputs. To demonstrate the stability of our method, we measure the changes in distances provided as explanation when the input is distorted. For fairness of measurements, the distance change is normalized by distance before the input distorted. We measure these distance changes for each prototype of all inputs. In Table our method shows much more stability for input distortions. In particular, there is large difference in terms average and median in the case the baseline model, which means explanations that provide significantly different distances exist when the inputs are distorted. Based on our experiment, we claim our method provides a more reliable explanation users than the bascline model.
18

---

Under review as a conference paper at ICLR 2022
Table 5: Distance changes by baseline model, proposed model with VAE, FactorVAE
FactorVANe
VAE
Baseline model
Distortion
Median
Mean
Median
Mean
Median
Mean
0.1
0.0222
0.0294
0.0293
0.0390
0.0701
0.1580
0.2
0.0421
0.0535
0.0595
0.0735
0.1535
0.4666
0.3
0.0502
0.0634
0.0730
0.0923
0.1908
0.6697
0.4
0.0630
0.0784
0.0907
0.90
0.2575
0.1177
0.5
0.0732
0.0912
0.1044
0.1283
0.3224
1.6922
19


Reviews:
[{"review_id": "pAhBJPK4z1H", "reviewer": "Reviewer_bDHd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper improved existing prototype-based DNN by enhancing interpretability and while also improving the performance. I enjoyed reading the paper and I think the proposed solutions will benefit other prototype-based DNN models for image classification. I have a couple of questions that I wish the authors could answer.\n\n1) in the original ProtoPnet paper, the authors Li et al designed separation cost and clustering cost in their objective, with the goal to push images closer to prototypes of the same label, and away from ``incorrect'' prototypes. I don't see such designs in this paper. Is there a reason that you don't choose to include these terms?\n\n2) In some prototype-based models, they also include a \\emph{diversity} term, which encourages protoypes to be different, to avoid redundancy. There are no such designs in the proposed model. Do prototypes have redundancy issues? do some of them look very similar to each other?\n\n3) I'm wondering if you can discuss the connection between your explanations for basic explanations and counterfactual explanations, because it seems by changing certain features, following your basic explanations and explanations for basic explanations, you can guide the model to get a different prediction. You did mention that \"Visualized images of our explanations are fundamentally generated through VAE, making it challenging to generate human-recognizable images for complex datasets that are difficult for VAE to generate the observable image.\" Would that be a reason that your model cannot be used to generate counterfactual explanations?", "summary_of_the_paper": "This paper extends existing series of prototype-based DNN for image classification. Their model uses VAE to ensure similar latent representations share similarities in appearance, a problem that many existing prototype-based CNN models suffer; They also include an uncertainty layer which improves the predictive performance. The authors also conducted human evaluations via Amazon Mechanical Turks to validate the interpretability of their model ", "summary_of_the_review": "Overall I enjoyed reading the paper. I only have a few questions, which basically asked the authors to clarify their design choices, especially those that are different from existing works. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "as4UzWZTxRk", "reviewer": "Reviewer_HXZp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The key issue that explanation spaces try to address is that having similar latent representations does not guarantee that images share similarities in human-discernible ways, so the method uses VAE to regularize the latent representations. This is an interesting idea.\n\nI have a few questions about the experimental results:\n- How was reliability measured (that FactorVAE forms a more reliable explanation space) in Section 5.1?\n- It looks like there are two orange clusters in Figure 3b?", "summary_of_the_paper": "The paper extends prototypes-based explanations by proposing what they call explanation spaces describing the relationships between input and prototypes, the relationship between prototypes, and how prototypes are distributed. They construct explanation spaces using VAEs and suggest a way to find the optimal number of prototypes.\n", "summary_of_the_review": "The idea is interesting but not a lot of novelty, as some existing papers (e.g. Li's Deep Learning for Case-Based Reasoning through Prototypes paper) already provide notions of distances between inputs and prototypes, prototypes and test data points, etc.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Gj82c4JTS2X", "reviewer": "Reviewer_QGcu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper introduces an interesting concept, *explanation space*, and an interesting way of constructing networks that incorporate some constraints, i.e. the distance layer, into the model. As a result, part of the internal components are not black boxes to humans any more. It also comes with an easier way to visualize the internal activations of the model by the decoder part of a VAE. \n\nHowever, even though I appreciate the potential merits in the paper’s idea of building better networks in terms of interpretability, I have several concerns about the motivation of the technique and inadequate evaluations that may not be sufficient to convince the reader. Please find my detailed review as follows.\n\n### Novelty\n\nThe idea of combining distance layer with VAE is novel and the concept of *explanation space* is also first proposed in this paper. These two techniques make this paper stand out from other VAE-based explanations. However, these techniques are not well-motivated to me and the particular application of the proposed visualizations, explanation space, is also missing. To clarify, I have two following questions regarding the motivation of the paper:\n\n1. What is the concrete problem this paper tries to solve? As the authors argue that “However, examples are powerful but not enough” in the introduction of the paper, the sentence seems not to be completed because the most important object is missing -- the example-based explanations are not enough for what (questions)? I am interested in what kind of questions that the authors have in mind that a human user may have but is not able to fully answer by the current explanation approaches. For example, with a classifier that differs dogs from cats, what does the concrete question look like, which requires a set of explanations, i.e. explanation space, instead of examples generated by prior work. Without a clarification like that, I am not sure how the evaluations provided in the result of the paper justify the significance and validate the efficiency of the proposed approach.   \n\n2. The current motivation for using the uncertainty layer seems to be quite weak. The authors do not justify what motivates the use of an uncertainty layer on the top of the network. The only relevant information I am able to find is that by using the uncertainty layer the accuracy of the model is improved by 1%. This is not convincing enough to serve as a strong motivation because :1) the improvement of performance is strongly related to the data distribution, which does not serve as a motivation if using other datasets. For example, 1% increase can be very significant for larget datasets, i.e. ImageNet, but pretty trivial for the datasets used in this paper, FashionMNIST and MNIST. 2) the improvement of the accuracy should instead be the result of employing the uncertainty layer instead of the motivation to do so. The way the current paper justifies the use of this layer does not convince me of the necessity of it. And i am curious to see how the changes in explanations may look like with the 1% decrease of the network’s performance. \n\nA minor point: understanding the model’s internal behavior by the distances of representations learned by VAE or any encoder-decoder networks are not first proposed in this paper . Some related work [1] seems worth mentioning and discussing. \n\n\n### Technical Quality \n\nI have three concerns regarding the analysis and the experiments in this paper.\n\n1. The empirical findings seem to be the main contributions of this paper, however, some descriptions seem to convey the message that the proposed explanation enjoys some nice properties but they are either not clearly defined or proved. As the authors emphasize in the abstract that “... but we propose an inherently interpretable model for more faithful explanations.”, I am not able to locate clear definitions in the paper about “inherently interpretable” and a measurable definition of “faithfulness” to support the argument that the proposed method is **more** faithful. Further, what are the baselines for the authors to derive the conclusion that the proposed method is **more faithful** (I will come back to the baseline issue in the next bullet point). \n\n2. The empirical evaluations are not strong enough to support the claims that the authors make about the contributions. Firstly, most of the experiments are conducted on MNIST and FashionMNIST but these two data distributions are sometimes too simple to generalize to higher dimensions, i.e. ImageNet. It is not new to the community that a lot of algorithms will work perfectly on MNIST and FashionMNIST but fail dramatically on more complicated colorful images as in practice you may not actually need a deep net to achieve a great performance on MNIST. For example, the K-nearest neighbors classifier is both explainbale and accurate on MNIST. I would encourage the authors to expand the empirical evaluations to datasets with higher dimensions in the input features before rushing to make any conclusions. \n\n3. Baselines are missing in the evaluation section. The major part of the evaluation section seems to focus on explore that the proposed method can bring to the user. However, before rushing into the exploration part of the method, there is a missing part of comparing the proposed method with the prior work. Even though the authors emphasize that the proposed method provides a set of examples instead of just one, which does not exist in the previous work. However, it does not seem to be unfair to compare one or several examples samples from the explanation space with some prior work that provides instance-based explanations [2, 3, 4, 5, 6] unless I misunderstand the paper and please help me understand why such comparisons are not useful to show the proposed explanation space provides better explanations. These baselines may not provide an apple-to-apple comparison but are worth discussing and performing some comparisons in the ballpark. \n\n\n### Clarity\nThe writing of the introduction section is a bit confusing to me because I miss the part about what the problem the paper aims to answer and how the results in the evaluation sections help to show that the proposed method solves the problem. Figure 1 and 2 are clear to help readers to understand the proposed architecture and I appreciate that. \n\n### Significance\n\nWith the aforementioned review, the significance of this paper can vary. One the one side, if the authors can help me understand the what is the subject of the explanation, that is the question requiring an answer like the one proposed in the paper and can not be answered by prior work, and why the current evaluation is sufficient to support the claim, I would recommend this paper because the contributions are significant to the explanation community. On the other hand, the current manuscript does not seem to be able to convince me that the contributions are sound and significant. Therefore, I am on the negative side but will decrease my confidence in the score because I am willing to increase it once my concerns are resolved. \n\n\n[1] Yang, Ceyuan et al. “Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis.” Int. J. Comput. Vis. 129 (2021): 1451-1466.\n \n[2] Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning - Volume 70(ICML'17). JMLR.org, 1885–1894.\n\n[3] Kim, Been et al. “Examples are not enough, learn to criticize! Criticism for Interpretability.” NIPS(2016).\n\n[4] Yeh, Chih-Kuan et al. “Representer Point Selection for Explaining Deep Neural Networks.” NeurIPS(2018).\n\n[5] Pruthi, Garima et al. “Estimating Training Data Influence by Tracking Gradient Descent.” ArXivabs/2002.08484 (2020): n. Pag.\n\n[6] Goyal, Yash et al. “Counterfactual Visual Explanations.” ArXiv abs/1904.07451 (2019): n. pag.\n", "summary_of_the_paper": "The main contributions of this papa are 1) a particular architecture that combines Variational Auto-Encoder (VAE) that makes the predictions based on the distances between instances and proto-types; and 2) a series of visualizations in the latent space that explains the model’s predictions, i.e. explanation space. In order to verify their assumptions that their networks provide faithful and rich explanations, the authors conduct user studies on MNIST and FashionMNIST datasets.", "summary_of_the_review": "Overall I recommend for a rejection because the current version of the paper is not well-motivated and the evaluations are not sufficient to support the claims and contributions made by this paper. I am open to any discussions and will increase my score if my concerns are resolved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GBojo4gjNi", "reviewer": "Reviewer_RHQ4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The continuous nature of the embedding space of a VAE makes it more suitable than an AE. This is an improvement over the previous work of (Oscar Li et al. AAAI 2018).\nTraining VAE may have several issues: it may require many training samples to be able to generate explanation images of adequate quality. In the paper, generated samples are based on toy datasets MNIST or FMNIST. It is not clear how complicated the image generation process would become if the method be applied to more serious datasets. My concern is that at some point the challenge will become high-quality image generation, rather than a classification task.  \nAlso, the decoder of the VAE may degrade the classification performance (C. Chen Neurips 2019). Too much regularization may degrade the classification performance. Too little regularization (VAE) may result in a poor generation of the explanation images.\n \nDo these \"extra\" explanations this method is providing offer anything more than a simple prototype? In other words, if the authors had a comparison study in their mechanical turk experiment with PDL (Oscar Li et al. AAAI 2018), would there have been any benefit over using their method? The paper unfortunately hasn't done an experiment with PDL, only comparing to the \"no explanation\" scenario.\n\nI am not a native English speaker but there are too many grammatical issues in the paper. For example, in pp.2 paragraph 2: “Among our model, (i) relationships …” , Meanwhile, prototypes … are given for explanation.”. “we developed the model that progresses…”. Or in page 9, “it is proper to use …”\n", "summary_of_the_paper": "The paper tries to provide an inherently interpretable neural network. Unlike the previous method of (O. Li, AAAI 2018) the paper uses VAE instead of AE. It considers the distance between the distribution of the input and the distribution of prototypes in the latent space of a VAE (the explanation space). The paper also provides a method for determining the number of prototypes using the Bayesian Information Criterion (BIC) for choosing the optimal number of components in the corresponding GMM. \n", "summary_of_the_review": "I see values in the paper as using a VAE instead of AE is a natural improvement over previous work. Nevertheless, I have major concerns as described above, in particular, the paper is missing a comparison with previous works especially (Oscar Li et al. AAAI 2018). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to create an explanation space to describe the relationships between input data and prototypes (and also between the prototypes themselves). It constructs such a space suing VAEs and conducts experiments to validate the effectiveness and interpretability of the method.

Strengths:
- The proposed method is interesting and intuitive

Weakness:
- Novelty of the idea is limited
- Missing experiment comparison with some important previous work
- Some claims are not well supported by the empirical results
Decision: Reject