An evaluation of quality and robustness of smoothed explanations

1 INTRODUCTION . Explanation methods attribute a numerical value to each data feature in order to quantify its relative importance towards the model ’ s prediction . Such attributions help to better understand and trust complex models like deep neural networks ( DNNs ) . In safety-critical tasks , such an understanding is a prerequisite to the deployment of DNNs , because a domain expert will never make important decisions based on a model ’ s prediction unless that model is trustworthy . Moreover , explanations can help to understand the reasons behind the decision of a model , and when it comes to model debugging , they can reveal the presence of any spurious data correlations that may lead to faulty predictions during inference ( Ribeiro et al. , 2016 ) . In the context of image classification with deep neural networks , several explanation methods have been proposed based on the gradient with respect to input , also called gradient-based explanations ( Baehrens et al. , 2010 ; Bach et al. , 2015 ; Selvaraju et al. , 2017 ; Sundararajan et al. , 2017 ; Springenberg et al. , 2015 ) . The explanation generated by these methods , a saliency map , highlights the parts of the image that contributed to the prediction . Recent work has shown that gradient-based explanations of neural networks can be fragile and can be easily manipulated via adversarially perturbed inputs ( Ghorbani et al. , 2019 ; Dombrowski et al. , 2019 ; Heo et al. , 2019 ; Viering et al. , 2019 ; Kindermans et al. , 2019 ) . That is , one can find a small-norm perturbation to be added to an input ( often imperceptible ) , such that the focus of the explanation changes towards irrelevant features while the model ’ s output remains unchanged . This , in turn , can make explanations inappropriate to help end-users gain trust in a model ’ s prediction . The large curvature of the decision surface of neural networks has been identified as one of the causes of fragility for gradient-based explanations ( Ghorbani et al. , 2019 ; Dombrowski et al. , 2019 ; Wang et al. , 2020 ) . To make explanations more robust , a class of approaches proposed smoothing the explanation or making the decision surface of neural networks more smooth ( Wang et al. , 2020 ; Dombrowski et al. , 2019 ; Ivankay et al. , 2020 ) . We refer to these approaches as smoothing approaches . It is worth mentioning that similar methods have been proposed in the context of adversarial robustness , with the aim of flattening the decision surface of neural networks in order to reach more robust predictions ( Moosavi-Dezfooli et al. , 2019 ; Qin et al. , 2019 ) . Here , we provide a thorough investigation of the explanations derived by smoothing approaches in terms of explanation quality and robustness . We employ various tests to assess the quality of these explanations . Each test evaluates a desirable property for explanations , such as : sensitivity to changes in the model , fidelity to the predictor function , etc . In terms of robustness , we show that explanations derived by smoothing approaches only provide robustness against additive ` p norm attacks . Specifically , in this work , we show that compared to additive attacks , attacks based on the combination of spatial transformation ( Xiao et al. , 2018 ) and/or color transformation ( Laidlaw & Feizi , 2019 ) together with additive perturbations are more effective in manipulating these explanations . Our contributions can be summarized as follows : • We study the effectiveness of smoothing approaches to achieve robust explanations . We present results on evaluating both the quality and robustness properties of smoothed explanations . • We assess the quality of smoothed explanations via presenting the results of various quality tests . Our results demonstrate the pros and cons of smoothed explanations with respect to the following quality aspects : sensitivity to model parameters , class discriminativeness , Infidelity , and sparseness . • We present results for different combination of additive and non-additive attacks , and show that they are able to manipulate explanations derived by smoothing approaches more successfully . Combining different types of perturbations to achieve stronger attacks has been a topic of investigation in the context of adversarial examples ( Jordan et al. , 2019 ) . To the best of our knowledge , this is the first time such attacks have been used in the context of explanations . Related works . There have been several works aiming to make explanations more robust . These works mostly focused on either modifying the explanation method itself or modifying the predictor model to achieve robust explanations . Wang et al . ( 2020 ) introduced Uniform Gradient , which is similar to Smooth Gradient unless it uses Uniform noise , and showed that it can hardly be manipulated by additive attacks . Dombrowski et al . ( 2019 ) proved that a network with soft-plus activations has a more robust Gradient explanation compared to a ReLU network , given that the parameter β of the soft-plus function is chosen to be sufficiently small . Consequently , they proposed the β-smoothing approach in which they substitute the ReLU activations of a trained network by softplus functions with a small β parameter . Wang et al . ( 2020 ) introduced a regularization term called Smooth Surface Regularization ( SSR ) to the training objective of a DNN . This training objective penalizes the large curvature of a DNN by regularizing the eigenvalue of the input hessian with the maximum absolute value . Moreover , they showed that adversarial training ( Madry et al. , 2018 ) also leads to more robust explanations . This fact can also be deduced from the results of ( MoosaviDezfooli et al. , 2019 ) as they showed that adversarial training leads to a significant decrease in the curvature of the loss surface with respect to inputs . Anders et al . ( 2020 ) proposed an attack in which they adversarially manipulate the model instead of the input in order to manipulate the explanation . Then they propose a modification to the existing explanation methods to make them more robust against such manipulated models . Lakkaraju et al . ( 2020 ) proposed a framework for generating robust and stable black box explanations based on adversarial training . Chen et al . ( 2019 ) introduced a regularization term to the training objective of neural networks to achieve robust Integrated Gradient explanations . Finally , Dombrowski et al . ( 2020 ) developed a theoretical framework to derive bounds on the maximum manipubality of explanations and proposed three different techniques to boost the robustness of explanations . In this work , we show that the robustness of smoothed explanations can be affected by employing a combination of additive and non-additive attacks . Furthermore , we present a through evaluation of the different quality aspects of smoothed explanations . 2 BACKGROUND . First , we provide the definition of an explanation map and then briefly describe the explanation methods we used in this paper . Then we continue with introducing the attacks to explanations and the smoothing approaches we are going to study in this paper . Consider a model f : Rd → RK which classifies an input x ∈ Rd into one of the K classes . An explanation map , denoted by hf ( x ) : Rd → Rd , associates a score to each feature of the input indicating the relevance of that feature towards the model ’ s prediction . For instance , in the context of image classification , saliency maps associate a score to each pixel of the input image resulting in a heatmap that highlights important regions of the image leading to the model prediction . In this work , we focus on the gradient-based explanations and mainly on the Gradient method . Given a model f and an input x , the Gradient explanation is defined as∇xf ( x ) . Since other gradient-based explanation methods make use of the gradients with respect to input , we argue that our results could be extended to those explanation methods as well . We will also consider two smoothed variants , namely Smooth ( Smilkov et al. , 2017 ) and Uniform Gradient ( Wang et al. , 2020 ) methods . 2.1 ATTACKS TO MANIPULATE EXPLANATIONS . Similarly to common adversarial attacks ( Goodfellow et al. , 2015 ; Moosavi-Dezfooli et al. , 2016 ; Szegedy et al. , 2014 ) , recent work has shown that explanations can also be manipulated by adding a small and almost imperceptible perturbation to the input ( Ghorbani et al. , 2019 ; Dombrowski et al. , 2019 ) . We refer to this class of attacks as explanation attacks . There have been various formulations for explanation attacks ( Ghorbani et al. , 2019 ; Dombrowski et al. , 2019 ) . In this work , we will use the formulation introduced by Dombrowski et al . ( 2019 ) . In this attack , the attacker tries to find a perturbed input for which the explanation is manipulated to be very similar to a given target explanation map while the output of the model remains approximately unchanged . Note that the target map could be any heatmap in general ; however , we used the explanation of a target image as a target map in this work . Below , we will give a formal definition of this attack . Definition 1 ( Targeted manipulation attack ) . An explanation hf ( x ) for model f ( x ) is vulnerable to attack at input x if there exist a perturbed input xadv , such that hf ( xadv ) is similar to a given target map ht but the model ’ s output remains unchanged . An attacker finds xadv by minimizing the following objective function : L = ∥∥hf ( xadv ) − ht∥∥2 + γ1 ‖f ( xadv ) − f ( x ) ‖2 + γ2Lreg ( x , xadv ) ( 1 ) The first term in ( 1 ) ensures the similarity of the manipulated explanation to the target map , the second term ensures the similarity between the model output for the original and perturbed inputs , and the third term regularizes the perturbation to ensure perceptual similarity between the original and perturbed images . Note that Lreg is defined by the attacker according to the type of the perturbation . The relative weighting of the terms in ( 1 ) is controlled by the hyper-parameters γ1 and γ2 . 2.2 TOWARDS ROBUST EXPLANATIONS . Recent works have tried to define the robustness of explanations in terms of the sensitivity of input gradients to changes in the input data ( Wang et al. , 2020 ; Dombrowski et al. , 2019 ) . Wang et al . ( 2020 ) define the robustness of explanations by the Lipschitz continuity coefficient of the input gradients ; a smaller coefficient means that the explanation is less sensitive to the changes in the input and hence more robust . In this regard , a class of approaches to generate robust explanations have been proposed in the recent works , which are either based on smoothing out the explanation maps or flattening the decision boundary of the model itself . Broadly , these approaches can be classified into two categories : ( 1 ) Post-hoc approaches do not require retraining of the network and can be applied as a post-processing step . ( 2 ) Ad-hoc approaches to robust explanations require retraining of the network and hence are more costly . In this work , we consider Smooth Gradient ( Smilkov et al. , 2017 ) , Uniform Gradient ( Wang et al. , 2020 ) , and β-smoothing ( Dombrowski et al. , 2019 ) as post-hoc approaches . The first two methods involve smoothing the explanation map , while the third one smooths the decision surface of the model . All three approaches act on pre-trained models , and hence are characterized as post-hoc . Among the ad-hoc methods , we study the explanations generated by adversarially trained networks , and networks trained with curvature regularization ( CURE ) ( Moosavi-Dezfooli et al. , 2019 ) , which is a similar approach to SSR ( Wang et al. , 2020 ) 1 . 1We experiment only with CURE , because with the publicly available code of SSR we were not able to reproduce the results in ( Wang et al. , 2020 ) .

Summary: This paper evaluates the quality and robustness of explanations of three post-hoc smoothing approaches (Smooth Gradient, Uniform Gradient, B-smoothing), and two ad-hoc smoothing approaches (CURE, Adv). It evaluates the quality of explanations based on the model parameter sensitivity, class sensitivity, sparseness, infidelity. It also evaluates the robustness of explanations to combinations of additive, spatial transformation, and recoloring attacks by comparing similarities between target and explanation maps. All evaluation is performed on publicly available benchmark datasets such as ImageNet and MS COCO. Based on their experimental results, the authors made several claims about the quality and robustness compared to the vanilla gradient method. For example, the authors claim that most of the smoothing methods are less sensitive to perturbation of model parameters so they may not be helpful to debug a model, and they may not be "robust" to non-additive types of attacks.