GANMEX: Class-Targeted One-vs-One Attributions using GAN-based Model Explainability

Attribution methods have been shown as promising approaches for identifying key features that led to learned model predictions . While most existing attribution methods rely on a baseline input for performing feature perturbations , limited research has been conducted to address the baseline selection issues . Poor choices of baselines limit the ability of one-vs-one explanations for multi-class classifiers , which means the attribution methods were not able to explain why an input belongs to its original class but not the other specified target class . Achieving one-vs-one explanation is crucial when certain classes are more similar than others , e.g . two bird types among multiple animals , by focusing on key differentiating features rather than shared features across classes . In this paper , we present GANMEX , a novel approach applying Generative Adversarial Networks ( GAN ) by incorporating the to-be-explained classifier as part of the adversarial networks . Our approach effectively selects the baseline as the closest realistic sample belong to the target class , which allows attribution methods to provide true one-vs-one explanations . We showed that GANMEX baselines improved the saliency maps and led to stronger performance on perturbation-based evaluation metrics over the existing baselines . Existing attribution results are known for being insensitive to model randomization , and we demonstrated that GANMEX baselines led to better outcome under the cascading randomization of the model . 1 INTRODUCTION . Modern Deep Neural Network ( DNN ) designs have been advancing the state-of-the-art performance of numerous machine learning tasks with the help of increasing model complexities , which at the same time reduces model transparency . The need for explainable decision is crucial for earning trust of decision makers , required for regulatory purposes Goodman & Flaxman ( 2017 ) , and extremely useful for development and maintainability . Due to this , various attribution methods were developed to explain the DNNs decisions by attributing an importance weight to each input feature . In high level , most attribution methods , such as integrated gradient ( IG ) ( Sundararajan et al . ( 2017 ) ) , DeepSHAP ( Lundberg & Lee ( 2017 ) ) , DeepLift ( Shrikumar et al . ( 2017 ) ) and Occlusion ( Zeiler & Fergus ( 2013 ) ) , alter the features between the original values and the values of some baseline instance , and accordingly highlight the features that impacts the model ’ s decision . While extensive research has been conducted on the attribution algorithms , research regarding the selection of baselines is rather limited , and it is typically treated as an afterthought . Most existing methodologies by default apply a uniform-value baseline , which can dramatically impact the validity of the feature attributions ( Sturmfels et al . ( 2020 ) ) , and as a result , existing attribution methods showed rather unperturbed output even after complete randomization of the DNN ( Adebayo et al . ( 2018 ) ) . In a multi-class classification setting , existing baseline choices do not allow specifying a target class , and this has limited the ability for providing a class-targeted or one-vs-one explanation , meaning explaining why the input belongs to class A and not a specific class B . These explanations are crucial when certain classes are more similar than others , as often happens for example when the classes have a hierarchy among them . For example , in a classification task of apples , oranges and bananas , a model decision for apples vs oranges should be based on their color rather than the shape since both an apple and orange are round . This would intuitively only happen when asking for an explanation of ‘ why apple and not orange ’ rather than ‘ why apple ’ . In this paper , we present GAN-based Model EXplainability ( GANMEX ) , a novel methodology for generating one-vs-one explanations leveraging GAN . In a nutshell , we use GANs to produce a baseline image which is a realistic instance from a target class that resembles the original instance . A naive use of GANs can be problematic because the explanation generated would not be specific to the to-be-explained DNN . We lay out a well-tuned recipe that avoids these problems by incorporating the classifier as a static part of the adversarial networks and adding a similarity loss function for guiding the generator . We showed in the ablation study that both swapping in the DNN and adding the similarity loss are critical for resulting the correct explanations . To the best of our knowledge , GANMEX is the first to apply GAN for explaining DNN decisions , and furthermore the first to provide a realistic baseline image , rather than an ad-hoc null instance . We showed that GANMEX baselines can be used with a variety of attribution methods , including IG , DeepLIFT , DeepSHAP and Occlusion , to produce one-vs-one attribution superior compared with existing approaches . GANMEX outperformed the existing baseline choices on perturbation-based evaluation metrics and showed more desirable behavior under the sanity checks of randomizing DNNs . Other than its obvious advantage for one-vs-one explanations , we show that by replacing only the baselines and without changing the attribution algorithms , GANMEX greatly improves the saliency maps for binary classifiers , where one-vs-one and one-vs-all are equivalent . 2 RELATED WORKS . 2.1 ATTRIBUTION METHODS AND SALIENCY MAPS . Attribution methods and their visual form , saliency maps , have been commonly used for explaining DNNs . Given an input x = [ x1 , ... , xN ] ∈ RN and model output S ( x ) = [ S1 ( x ) , ... , SC ( x ) ] ∈ RC , an attribution method for output i assign contribution to each pixel AS , c = [ a1 , ... , aN ] . There are two major attribution method families : Local attribution methods that are based on infinitesimal feature perturbations , such as gradient saliency ( Simonyan et al . ( 2014 ) ) and gradient * input ( Shrikumar et al . ( 2016 ) ) , and global attribution methods that are based on feature perturbation with respect to a baseline input ( Ancona et al . ( 2018 ) ) . We focus on global attribution methods since they tackle the gradient discontinuity issue in local attribution methods , and they are known to be more effective on explaining the marginal effect of a feature ’ s existence ( Ancona et al . ( 2018 ) ) . In this paper , we discussed five popular global attribution methods below : Integrated Gradient ( IG ) ( Sundararajan et al . ( 2017 ) ) calculates a path integral of the model gradient from a baseline image x̃ to the input image x : IGi = ( xi − x̃i ) ∫ 1 α=0 ∂xiS ( x̃+ α ( x− x̃ ) ) dα . The baseline is commonly chosen to be the zero input and the integration path is selected as the straight path between the baseline and the input . DeepLIFT ( Shrikumar et al . ( 2017 ) ) addressed the discontinuity issue by performing backpropagation and assigns a score C∆xi∆t to each neuron in the networks based on the input difference to the baseline ∆xi = xi− x̃i and the difference in the activation to that of the baseline ∆t = t ( x ) − t ( x̃ ) , that satisfies the summation-to-delta property ∑ i C∆xi∆t = ∆t . Occlusion ( Zeiler & Fergus ( 2013 ) ; Ancona et al . ( 2018 ) ) applies full-feature perturbations by removing each feature and calculating the impacts on the DNN output . The feature removal was performed by replacing its value with zero , meaning an all zero input was implicitly used as the baseline . DeepSHAP ( Chen et al . ( 2019 ) ; Lundberg & Lee ( 2017 ) ; Shrikumar et al . ( 2017 ) ) was built upon the framework of DeepLIFT but connecting the multipliers of attribution rule ( rescale rule ) to SHAP values , which are computed by ‘ erasing features ’ . The operation of erasing one or more features require the notion of a background , which is defined by either a distribution ( e.g . uniform distribution over the training set ) or single baseline instance . For practical reasons , it is common to choose a single baseline instance to avoid having to store the entire training set in memory . Expected Gradient ( Erion et al . ( 2019 ) ) is a variant of IG that calculates the expected attribution over a prior distribution of baseline input , usually approximated by the training set XT , meaning EGi = Ex̃∼XTα∼U ( 0,1 ) ( x − x̃ ) i∂xiS ( x̃ + α ( x − x̃ ) ) where U is the uniform distribution . In other words , the baseline of IG is replaced with a uniform distribution over the samples in the training set . A crucial property of the above methods is their need for a baseline , which is either explicitly or implicitly defined . In what follows we show that these methods are greatly improved by modifying their baseline to that chosen by GANMEX . 2.2 THE BASELINE SELECTION PROBLEM . Limited research has been done on the problem of baseline selection so far . A simple ” most natural input ” , such as zero values of all numerical features is commonly chosen as the baseline . For image inputs , uniform images with all pixels set to the max/min/medium values are commonly chosen . The static baselines frequently cause the attribution to only focus on or even overly highlight the area where the feature values are different from the baseline values , and hide the feature importance where the input values are close to the baseline values ( Sundararajan & Taly ( 2018 ) ; Adebayo et al . ( 2018 ) ; Kindermans et al . ( 2017 ) ; Sturmfels et al . ( 2020 ) ) . Several none-static baselines have been proposed in the past , but each of them suffered from its own downsides ( Sturmfels et al . ( 2020 ) ) . Fong & Vedaldi ( 2017 ) used blurred images as baselines , but the results are biased toward highlighting high-frequency information from the input . Bach et al . ( 2015 ) make use of the training samples by finding the training example belonging to the target class closest to the input in Euclidean distance . Even though the concept of minimum distance is highly desirable , but in practice , the nearest neighbor selection in high dimensional space can frequently lead to poor outcome , and most of the nearest neighbors are rather distant from the original input . Along the same concept , expected gradient simply samples over all training instances instead of identifying the closest instance ( Erion et al . ( 2019 ) ) . Expected gradient benefits from ensembling in a way similar to that of SmoothGrad , which averages over multiple saliency maps produced by imposing Gaussian noise on the original image ( Smilkov et al . ( 2017 ) ; Hooker et al . ( 2019 ) ) . We claim however that averaging over the training set does not solve the issue ; for example , due to the foreground being located in different sections of the images , the average image would often resemble a uniform baseline . 2.3 ONE-VS-ONE AND ONE-VS-ALL ATTRIBUTION . In multi-class settings , while one-vs-all explanationAS , co ( x ) was designed to explain why the input x belong to its original class co and not the others , one-vs-one explanations aim to provide an attribution AS , co→ct ( x ) ∈ RN that explains why x belong to co and not the specified target class ct . Most existing attribution methods were primarily designed for one-vs-all explanation , but was proposed to extend to one-vs-one by simply calculating the attribution with respect to the difference of the original class probability to the target class probability Sdiff ( x ) = Sco ( x ) −Sct ( x ) ( Bach et al . ( 2015 ) ; Shrikumar et al . ( 2017 ) ) . It is easy to think of examples where this somewhat naive formulation will not provide correct one-vs-one explanation . Taking the example of fruit classification , for both apples and oranges the explanation could easily be the round shape , and taking the difference between those will result in an arbitrary attribution . We claim that without a class-targeted baseline , the modified attributions will still omit the ” vs-one ” aspect of the one-vs-one explanation . Take IG for example , AS , diff ( x ) = AS , co ( x ) −AS , ct ( x ) . With zero baseline , the target class score Sct ( x ) and its gradient will likely stay close to zero along the straight path from the input to the zero baseline , meaning that AS , ct ( x ) ≈ 0 because the instance never belongs to the target class . With this in mind the one-vs-one explanation is not very informative with respect to the target class ct. Few class-targeted baselines were proposed in the past . The minimum distance training sample ( MDTS ) described in Section 2.2 is class-targeted as the sample was selected from the designated target class . While the original expected gradient was defined for one-vs-all explanation only , we extended the method to one-vs-one by sampling the baselines only from the target class . However , as mentioned in Section 2.2 , MDTS is frequently hindered by the sparsity of the training set in the high dimensional space , and expected gradient suffers from undesired effects caused by uncorrelated training samples . The problem of baseline selection , especially for the one-vs-one explainability , has presented a challenging problem , because the ideal baseline choice can simply be absent from the training set .

Summary: This paper considers the less-explored baseline selection issue in attribution methods for one-vs-one explanations of multi-class classifiers. The key insight is to construct the closest and realistic target class baseline. To this end, an existing image-to-image translation GAN model, namely StarGAN, is leveraged to transform an input example to another example in a target class yet is close to the input. This baseline can be integrated with a variety of attribution methods, including integrated gradient, DeepLIFT, Occlusion, and deepSHAP, and shows consistent improvements over zero baseline and minimum distance training sample for one-vs-one explanations. The experiments are conducted on three datasets – MNIST, SVHN, and apple2orange.