Beyond Examples: Constructing Explanation Space for Explaining Prototypes

1 INTRODUCTION . `` Explanation '' is essentially based on understanding . When a fact is in doubt , an explanation is provided to resolve a question . Therefore , explanation must be human-interpretable and reliable . An explanation that is not human-interpretable does not resolve the issue , and an explanation that is not reliable may create confusion . In practice , it is difficult to find an explanation that perfectly satisfies even one condition . In this situation , when an insufficient explanation is given , a use may wonder why the explanation is given . We aim to address this question by providing a reliable explanation and providing additional explanations for a given explanation in this paper . With advances in deep learning , its performance improves rapidly . Advances have increased the use of deep neural networks ( DNNs ) , which is used in areas where model decisions significantly affect people - such as healthcare ( Caruana et al. , 2015 ) , radiation ( Reardon , 2019 ) , and autonomous driving . As a result , questions about the rationality of model predictions for industrial and social use have been raised , and the call for explanation is growing ( Goodman & Flaxman , 2017 ; Doshi-Velez & Kim , 2017 ) . Since most DNNs are black-box models , explanation of model prediction is often provided by a second model that explains a given black-box model ( Ribeiro et al. , 2016b ; Chen et al. , 2018a ; Lundberg & Lee , 2017 ; Guo et al. , 2018 ) . This second model is called a post-hoc interpretable model , which has been used for flexibility ( Ribeiro et al. , 2016a ) . However , this can lead people to misunderstanding the explanation because it is not offered by the model that provides the prediction . Therefore , as Rudin ( 2019 ) claimed , it is necessary to use an inherently-interpretable model in order to accurately explain DNNs without any mistakes , which results from an architecture to provide intrinsic explanation . We build an inherently interpretable model , by referring to methods that generate prototypes and conduct classification using Euclidean-distance in a latent space ( Li et al. , 2018 ; Chen et al. , 2019 ) . These prototype-based explanation methods provide explanations with examples and distances and show efficacy in diverse domains . However , prototype-based methods are powerful but not enough to understand a model prediction fully , as explanations must be both human-interpretable and reliable . As most of the existing prototype-based explanation models suffer from the problem that similar latent representations do not guarantee similarity in shape , and distances often provide explanations that are not human-interpretable . Users can raise doubts , but have no choice but to accept the given distances only as numerical values . Since there is no way to verify how distances are constructed , the reliability of the explanations is compromised . Input Prototypes Prototypes : Class 2 Prototypes Input ( b ) Relationships between input and prototypes ( c ) Relationships between prototypes ( d ) Prototype distribution visualization ( a ) Explanation given by prototypes and distances Prototype : Class 8 Prototype : Class 6 Figure 1 : Basic explanation : ( a ) Learned prototypes that represent the training dataset and distances between the input and prototypes . Additional explanations ( b-d ) : ( b ) Relationships between input and prototypes . Bottom of the figure explains that the explanation space enables us to see how input 9 gradually becomes prototype 4 . ( c ) Relationships between prototypes . Providing gradual changes from prototype to another prototype that explains how model discriminates prototypes . ( d ) Prototype distribution visualization with two dimensions based on standard deviation demonstrating that prototypes are not luckily chosen . To solve these problems , we construct an explanation space that provides gradual explanations and reliable distance . By using VAE , a latent space can be visualized as an image that users can recognize . We also use VAE training techniques to regulate a latent space to ensure latent representations share similarities in appearance . This means that a distance in the latent space can have more `` intrinsic meaning '' . We call the latent space obtained through the process the explanation space . Our model provides additional gradual explanation as in Figure 1 . Meanwhile , the prototypes and the distances between the input data and prototypes are given as basic explanation in Figure 1 ( a ) . Through these relationships , users are able to understand how distances are composed . Instead of providing the distances between the input and prototypes , it is possible to better understand a model by showing which features are changing in terms of relationship . Therefore , our proposed model allows users to interpret distances more reliably . In this paper , we developed the model that performs classification through distances between prototype distributions and the encoded input distribution in the latent space of the Variational Autoencoder ( VAE ) ( Kingma & Welling , 2014 ) . After VAE encodes input data in a Gaussian distribution form , we compare the encoded input distribution and generated prototype distributions through four measures with various number of prototype distributions to provide explanations with high classification accuracy . Furthermore , we use the method ( Sensoy et al. , 2018 ) of placing a Dirichlet distribution on the class probabilities to improve accuracy by reducing the uncertainty of the model . Contributions . Our key contributions are as follows : • By constructing the explanation space , we can visualize additional gradual explanations . Through gradual explanations , relationships are provided rather than just providing each prototype image . • By our dense explanation space approach using VAE , similar latent representations tend to resemble each other in appearance . Therefore , distance as explanation is more meaningful because similarity is guaranteed . • We present a method to find a baseline number of prototypes to describe the training dataset . In experiments , we show changes in accuracy and explanations of the model by varying numbers of prototypes . 2 RELATED WORK . Explanation Methods for DNN While there are diverse suites of methods to explain DNN , it can be categorized based on the access of the model ’ s internal information and how they provide explanations ( Jeyakumar et al. , 2020 ) . The model-agnostic method is to approximate the relationships between input data and output decision while treating the model as a black-box . There are methods to create a local explanation ( Ribeiro et al. , 2016b ; Chen et al. , 2018a ; Lundberg & Lee , 2017 ; Chen et al. , 2018b ) by selecting features that play an essential role in making decisions and methods providing an interpretable model through a global approximate ( Guo et al. , 2018 ) . On the other hand , in knowing the internal information of the model , the model-transparent method provides explanations by saliency map using gradient in classifier ( Simonyan et al. , 2014 ; Smilkov et al. , 2017 ; Sundararajan et al. , 2017 ) , class activation map using global average pooling ( Zhou et al. , 2016 ; Selvaraju et al. , 2017 ) , and feature map based on attention ( Liu et al. , 2020 ) . The third method provides explanations by examples capturing relationship between test input and the model ’ s decision . Considering that examples are based on training data ( Bien & Tibshirani , 2011 ; Koh & Liang , 2017 ; Arık & Pfister , 2020 ) or learned prototype data ( Li et al. , 2018 ; Chen et al. , 2019 ; Hase et al. , 2019 ) , our work is closely aligned with these prototype-based classification models . The example-based method shows its efficacy to explain complex concepts in psychology ( Aamodt & Plaza , 1994 ) and education ( Renkl , 2014 ) domains , and is also accepted by the general public as the most preferred explanation method to explain DNN ( Jeyakumar et al. , 2020 ) . The following three prototype-based classification methods are similar to our method in that they use the distance between the encoded input and prototypes for classification and explanation . Li et al . ( 2018 ) learn and visualize the prototypes for the entire data through autoencoder . Chen et al . ( 2019 ) go further from here , characterizing training image patches as class-specific prototypes for partial explanations , while Hase et al . ( 2019 ) feature a hierarchical demonstration of prototypes through related groups for non-flat explanations . As most of prototype-based explanation methods are based on Li et al . ( 2018 ) , we use it as a baseline model . Our framework goes further from these methods , expands prototypes from vectors to distribution in latent space and constructs explanation space for diverse explanations with reliability . In addition to prototype-based methods , there are several methods that use the distance in latent space to understand the internal behavior of a model or to provide explanation using VAE ( Antorán et al. , 2021 ; O ’ Shaughnessy et al. , 2020 ) or GAN ( Lang et al. , 2021 ; Yang et al. , 2021 ) . Uncertainty in deep learning The largest body of research on estimating uncertainty in deep learning is Bayesian neural networks ( Neal , 2012 ; Gal & Ghahramani , 2015 ; Kendall & Gal , 2017 ) . To obtain predictive uncertainty , Bayesian neural networks use prior distributions on model parameters and infer the posterior distribution . Although exact inference in Bayesian methods is intractable , a range of approximate techniques has been proposed , such as variational inference ( Kingma et al. , 2015 ; Gal & Ghahramani , 2016 ) and stochastic gradient hamiltonian monte carlo ( SG-HMC ) ( Chen et al. , 2014 ) . However , in practice , these Bayesian methods are outperformed by Deep Ensembles ( Lakshminarayanan et al. , 2017 ) . This non-bayesian technique uses multiple neural networks trained from different initializations and averages their predictions as the model output . It is simple and effective but requires high computational costs at training and test time . Another non-bayesian technique is based on evidential deep learning ( Sensoy et al. , 2018 ; Amini et al. , 2019 ) , which we use to estimate uncertainty . These methods use a single deterministic neural network and apply prior distributions on model predictions , not model parameters . 3 METHODS . Our model consists of three parts : ( i ) VAE layer , ( ii ) Distance measuring layer ( iii ) Uncertainty layer , as shown in Figure 2 . The VAE layer is used to encode input into a form of distribution and decode the values obtained at prototype distributions and relationships with recognizable image data . The distance between prototype distributions and encoded input distribution is used as input for one fully-connected layer to create an inherently interpretable classification model . Moreover , since classification is performed with distances in the latent space of VAE , we use an uncertainty layer to increase accuracy . 3.1 USING VARIATIONAL AUTO ENCODER . Let D = { X , Y } be the training dataset consists of images x ∈ RN and labels y ∈ { 1 , . . . , K } . The encoder f : q ( z|x ) of the VAE maps an input image to a Gaussian distribution dz ∼ N ( µz , σz2 ) ( µz , σz ∈ Rd ) which is reparameterized to generate latent vector z ∈ Rd . After that , the generated z is reconstructed in the form of image x̂ = g ( z ) through decoder g : p ( x|z ) . Unlike autoencoder , VAE regulates the latent space created by the encoder as isotropic unit Gaussian ( p ( z ) ∼ N ( 0 , I ) ) , which is why we used VAE as a feature extractor . By regularizing latent space and sampling latent vectors from distribution , VAE constructs appropriate dense latent space that creates similar latent representations from similar-shaped images , which plays an essential role in the interpretability of our method described in Section 4 . Furthermore , our model learns prototype distributions as m independent multivariate Gaussian distributions D = { d1 , d2 , . . . , dm } , di ∼ N ( µi , σ2i ) ∈ Rd and visualizes µi as prototype . For that , the structures of disentangled VAE are applied in order to disentangle the explanation components at prototype distributions through dimensions . A method of encouraging the independence of representation distribution q ( z ) by lowering the KL divergence between q ( z ) and q̄ ( z ) : = ∏d j=1 q ( zj ) , and methods of lowering the KL divergence of encoder f and isotropic unit Gaussian p ( z ) to constrain the latent space stronger are additionally used with vanilla VAE . Applying these methods would result in the following objective : 1 n n∑ i=1 [ Eq ( z|xi ) [ logp ( xi|z ) ] − βKL ( q ( z|xi ) ‖p ( z ) ) ] − γKL ( q ( z ) ‖q̄ ( z ) ) . In this objective function , β and γ are hyperparameters that control the power of the constraint terms .

Summary: The paper tries to provide an inherently interpretable neural network. Unlike the previous method of (O. Li, AAAI 2018) the paper uses VAE instead of AE. It considers the distance between the distribution of the input and the distribution of prototypes in the latent space of a VAE (the explanation space). The paper also provides a method for determining the number of prototypes using the Bayesian Information Criterion (BIC) for choosing the optimal number of components in the corresponding GMM.