CDeepEx: Contrastive Deep Explanations

1 INTRODUCTION . Deep neural networks ( DNN ) have shown extraordinary performance on computer vision tasks such as image classification ( Szegedy et al. , 2015 ; Simonyan and Zisserman , 2014 ; Springenberg et al. , 2015 ; He et al. , 2016 ) , image segmentation ( Chen et al. , 2018 ) , and image denoising ( Zhang et al. , 2017 ) . The first example of such a performance was on image classification , where it outperformed other computer vision methods which were carefully handcrafted for image classification ( Krizhevsky et al. , 2012 ) . Following this success , DNNs continued to grow in popularity . Although the performances of DNNs on different tasks are getting close to human expertise ( Rajpurkar et al. , 2017 ) and in some cases surpass them ( Springenberg et al. , 2015 ) , there is hesitation to use them when interpretability of the results is important . Accuracy is a well-defined criterion but does not provide useful understandings of the actual inner workings of the network . If the deployment of a network may result in inputs whose distribution differs from that of the training or testing data , interpretability or explanations of the network ’ s decisions can be important for securing human trust in the network . Explanations are important in settings such as medical treatments , system verification , and human training and teaching . Naturally , one way of getting an explanation is asking the direct question , `` Why did the DNN choose this answer ? '' Humans often also seek contrasting explanations . For instance , they maybe more familiar with the contrasting answer , or they want to find the subtle differences in input which change the given answer to the contrasting one . This way of questioning can be phrased as , `` Why did the DNN not choose B ( over A ) ? '' In this work , we present a framework to answer this type of question . We learn a model over the input space which is capable of generating synthetic samples similar to the input . Then , we ask how we can alter this synthetic input to change the classification outcome . Our proposed framework is not based on heuristics , does not need to change the given network , is applicable as long as the given model can handle backpropagation ( no further requirements for layers ) , and can run much faster than methods with input perturbation . The only overhead of this method is the assumed availability of a latent model over the input . If this latent model is not available , we can learn such a model using generative adversarial methods or variational auto encoders . Learning this latent space needs to be done only a single time and is independent of the learned classifier to be explained . 2 RELATED WORK . There are different ways to categorize interpretability methods such as the one discussed by Shrikumar et al . ( 2017 ) . Here we categorize the existing approaches into three overlapping categories . 2.1 NETWORK VISUALIZERS . The first group of methods try to understand units of the network ( Erhan et al. , 2009 ; Yosinski et al. , 2015 ; Bau et al. , 2017 ) . These methods test each individual unit or a set of units in the network to gain insight into what network has learned . The disadvantage of these methods is that they need to check all the units to see which one ( or combination of units ) is responsible for a concept . Morcos et al . ( 2018 ) showed it is unlikely that only a single unit learns a concept . Rather , a combination of units usually combined to represent a concept . This , in turn , makes these methods inapplicable in practice when the network contains thousands of units . These methods are example-based explanations . That is , they generate an explanation for a single input . By contrast , Fong and Vedaldi ( 2018 ) proposed a method to determine whether the network learned a concept based on a set of probe images and pixel-level annotated ground truth which may not be readily available or easy to obtain for many tasks . 2.2 INPUT SPACE VISUALIZERS . The second category corresponds to networks that try to explain the network ’ s decision in the space of the input image . Ribeiro et al . ( 2016 ) proposed a method to find out which parts of the image have the largest contribution to the decision of the network by making changes to the image and forwarding the new image through network . Zintgraf et al . ( 2017 ) proposed a similar approach with a more clever way of sampling the image parts . These methods need to consider changing each dimension of images and get explanation for each change in order that the aggregated results are visually coherent . Zhou et al . ( 2016 ) proposed a method which forwards the image through the network , records the activations in the last pooling or convolution layer , and uses them as an explanation . Selvaraju et al . ( 2017 ) proposed a similar method , but which uses the back-propagated signal for a more accurate explanation . There are two potential difficulties with these approaches . First , they assume that the explanation can be summarized in the last pooling or convolution layer which has a broad receptive field . Explanations with such broad receptive field can not give useful information in explaining fine-grained datasets . Second , these methods are restricted to use in convolutional networks . Simonyan et al . ( 2013 ) used the gradient of the output with respect to the pixels of the input image to generate a heat map . They showed that their proposed method is closely related to DeconvNets ( Zeiler and Fergus , 2014 ) . The difference is in the handling of backpropagation of ReLU units . The weakness of these methods is that the generated backpropagated signal in image space is not visually specific . They need to use heuristics in backpropagation to make the results more specific and useful to humans , such as changing the backpropagation rules for ReLU unit ( Springenberg et al. , 2015 ) . Kindermans et al . ( 2018 ) showed the unreliability of these methods to shifts in input . Some of these methods needs a reference input image ( Shrikumar et al. , 2017 ) whose choice can greatly change the generated explanation ( Kindermans et al. , 2018 ) . While preparing this manuscript , we discovered xGEMs ( Joshi et al. , 2018 ) , a preprint available on arXiv . It is similar to our work in that it uses a GAN to regularize the explanation and also seeks to find a “ why not ” explanation . Their work is different in that it does not formulate a constrained optimization problem as we do ( and our results show the importance of our constraints ) and they focus on producing a “ morph ” from one class to another , rather than highlighting the differences directly in the input image ( as we do ) . 2.3 JUSTIFICATION EXPLANATIONS . Finally , there are methods that learn to justify the classification of a network by producing textual or visual justifications ( Hendricks et al. , 2016 ; Park et al. , 2018 ) . Although related to image descriptions or definitions , they differ in that their goal is to explain why . However , the justification network is trained to match human explanations . Therefore , they are not direct explanations of how the learned classification network makes its decisions , but rather what humans would like as the explanation . 2.4 SUMMARY . In summary , the existing approaches in literature have at least one of the following downsides . I By learning an explanation network , they use another black box to explain a current black box ( Park et al. , 2018 ) . II They are not always applicable since they need specific layers or architecture ( Zhou et al. , 2016 ; Selvaraju et al. , 2017 ) . III They use heuristics during backpropagation to generate explanations ( Springenberg et al. , 2015 ; Zeiler and Fergus , 2014 ) . IV They need of a set of probe images or concepts which may not be available or can not be easily obtained ( Shrikumar et al. , 2017 ) . V They need network alteration to record the activations ( Zeiler and Fergus , 2014 ; Bau et al. , 2017 ; Shrikumar et al. , 2017 ) . VI They need of considerable amount of computational time to provide an explanation ( Zintgraf et al. , 2017 ; Ribeiro et al. , 2016 ) . Our model does not suffer from the aforementioned shortcomings , except that we require a generative model of the input space . Note that this input-space model does not force the explanation . Its purpose is similar to that of reference input images , except ( as required by other methods ) it does not need any domain knowledge . The input-space model can be estimated entirely independently from the network to be explained . 3 PROPOSED METHOD . First , we introduce the notation used in this work in Section 3.1 . Then , in Section 3.2 , we describe how to learn a latent space capable of generating natural looking images similar to the input space . Last , in Section 3.3 , we describe our method on how to generate explanations from the latent representation of input space . The overall framework is summarized in Procedure 1 and Figure 1 . 3.1 TERMINOLOGY . D : Rn → Rc is the given discriminator network , for which we want to generate explanations . G : Rk → Rn generates natural looking images similar to the ones in the dataset . We choose G to be an adversarial network trained on the input dataset used for D. I ∈ Rn is the input image ; z ∈ Rk is a latent variable ; and Iz is the output of G , i.e . Iz = G ( z ) . ytrue is the correct label for image I , and yprobe is the class label for the class of interest . Thus , the question we would like to answer is “ Why did D produce label ytrue and not label yprobe for input I ? ” Procedure 1 Generating the explanation on given D and I 1 : Learn a function G : Rk → Rn . See 3.2 2 : Find a representation for input I using procedure 2 3 : Find ze from Equation 2 4 : Return the explanation G ( z0 ) −G ( ze ) Procedure 2 Getting latent representation on the input I 1 : procedure LEARN z0 ( G , I , λ , loss ( . ) ) 2 : z0 ∼ N ( 0 , 1 ) 3 : while G ( z0 ) 6≈ I do 4 : z0← z0 −λ∇zloss ( I , G ( z ) ) 5 : ∆z0 = G ( z0 ) − I 6 : return z0 , ∆z0 3.2 LEARNING INPUT DISTRIBUTION . The question “ why not class yprobe ? ” implies a query image about which the question is being asked . We need to capture the manifold of natural looking images similar to this input to be able to answer this question in a meaningful way for a human . Learning a compact manifold enables us to move along this manifold instead of directly optimizing on the input space , i.e. , each pixel . There are different ways to find this mapping including variational auto encoders ( Kingma and Welling , 2013 ) and generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) . In this work , we used GANs to map latent space into input space . We used the method proposed by Arjovsky et al . ( 2017 ) to train the GAN . The structure of the networks is similar to that proposed by Radford et al . ( 2015 ) .

Summary: The idea proposed in this paper is to aid in understanding networks by showing why a network chose class A over class B.  To do so, the goal is to find an example that is close to the original sample, but belongs to the other class. As is mentioned in the paper, it is crucial to stay on the data manifold for this to be meaningful. In the paper, an approach using a GAN to traverse the manifold is proposed and the experimental evaluation is done on MNIST.