Robust Models Are More Interpretable Because Attributions Look Normal

1 INTRODUCTION . Feature attribution methods are widely used to explain the predictions of neural networks ( Binder et al. , 2016 ; Dhamdhere et al. , 2019 ; Fong & Vedaldi , 2017 ; Leino et al. , 2018 ; Montavon et al. , 2015 ; Selvaraju et al. , 2017 ; Shrikumar et al. , 2017 ; Simonyan et al. , 2013 ; Smilkov et al. , 2017 ; Springenberg et al. , 2014 ; Sundararajan et al. , 2017 ) . By assigning an importance score to each input feature of the model , these techniques help to focus attention on parts of the data most responsible for the model ’ s observed behavior . Recent work ( Croce et al. , 2019 ; Etmann et al. , 2019 ) has observed that feature attributions in adversarially-robust image models , when visualized , tend to be more interpretable—the attributions correspond more clearly to the discriminative portions of the input . One way to explain the observation relies on the fact that robust models do not make use of nonrobust features ( Ilyas et al. , 2019 ) whose statistical meaning can change with small , imperceptible changes in the source data . Thus , by using only robust features to predict , these models naturally tend to line up with visibly-relevant portions of the image . Etmann et al . take a different approach , showing that the gradients of robust models ’ outputs more closely align with their inputs , which explains why attributions on image models are more visually interpretable . In this paper , we build on this geometric understanding of robust interpretability . With both analytical ( Sec . 3 ) and empirical ( Sec . 5 ) results , we show that the gradient of the model with respect to its input , which is the basic building block of all gradient-based attribution methods , tends to be more closely aligned with the normal vector of a nearby decision boundary in robust models than in “ normal ” models . Leveraging this understanding , we propose Boundary-based Saliency Map ( BSM ) and Boundary-based Integrated Gradient ( BIG ) , two variants of boundary attributions ( Sec . 4 ) , which base attributions on information about nearby decision boundaries ( see an illustration in Fig . 1a ) . While BSM provides theoretical guarantees in the closed-form , BIG generates both quantitatively and qualitatively better explanations . We show that these methods satisfy several desireable formal properties , and that even on non-robust models , the resulting attributions are more focused ( Fig . 1b ) and less sensitive to the “ baseline ” parameters required by some attribution methods . To summarize , our main contributions are as follows . ( 1 ) We present an analysis that sheds light on the previously-observed phenomeon of robust interpretability showing that alignment between the normal vectors of decision boundaries and models ’ gradients is a key ingredient ( Proposition 1 ) . In particular , we derive a closed-form result for one-layer networks ( Theorem 1 ) and empirically validate the take-away of our theorem generalizes to deeper networks . ( 2 ) Motivated by our analysis , we introduce boundary attributions , which leverage the connection between boundary normal vectors and gradients to yield explanations for non-robust models that carry over many of the favorable properties that have been observed of explanations on robust models . ( 3 ) We empirically demonstrate that one such type of boundary attribution , called Boundary-based Integrated Gradients ( BIG ) , produces explanations that are more accurate than prior attribution methods ( relative to ground-truth bounding box information ) , while mitigating the problem of baseline sensitivity that is known to impact applications of Integrated Gradients Sundararajan et al . ( 2017 ) ( Section 6 ) . 2 BACKGROUND . We begin by introducing our notations . Throughout the paper we use italicized symbols x to denote scalar quantities and bold-face x to denote vectors . We consider neural networks with ReLU as activations prior to the top layer , and a softmax activation at the top . The predicted label for a given input x is given by F ( x ) = argmaxc fc ( x ) , x ∈ Rd , where F ( x ) is the predicted label and fi ( x ) is the output on the class i . As the softmax layer does not change the ranking of neurons in the top layer , we will assume that fi ( x ) denotes the pre-softmax score . Unless otherwise noted , we use ||x|| to denote the ` 2 norm of x , and the ` 2 neighborhood centered at x with radius as B ( x , ) . Explainability . Feature attribution methods are widely-used to explain the predictions made by DNNs , by assigning importance scores for the network ’ s output to each input feature . Conventionally , scores with greater magnitude indicate that the corresponding feature was more relevant to the predicted outcome . We denote feature attributions by z = g ( x , f ) , z , x ∈ Rd . When f is clear from the context , we simply write g ( x ) . While there is an extensive and growing literature on attribution methods , our analysis will focus closely on the popular gradient-based methods , Saliency Map ( Simonyan et al. , 2013 ) , Integrated Gradient ( Sundararajan et al. , 2017 ) and Smooth Gradient ( Smilkov et al. , 2017 ) , shown in Defs 1-3 . Definition 1 ( Saliency Map ( SM ) ) The Saliency Map gS ( x ) is given by gS ( x ) : = ∂f ( x ) ∂x . Definition 2 ( Integrated Gradient ( IG ) ) Given a baseline input xb , the Integrated Gradient gIG ( x ; xb ) is given by gIG ( x ; xb ) : = ( x− xb ) ∫ 1 0 ∂f ( ( x−xb ) t+xb ) ∂x dt . Under review as a conference paper at ICLR 2022 # $ # $ % % ! ! ! '' `` # $ % '' ! '' '' `` # Definition 3 ( Smooth Gradient ( SG ) ) Given a zero-centered Gaussian distributionN with a standard deviation σ , the Smooth Gradient gSG ( x ; σ ) is given by gSG ( x ; σ ) : = E ∼N ( 0 , σ2I ) ∂f ( α+ ) ∂x . Besides , we will also include results from DeepLIFT ( Shrikumar et al. , 2017 ) and grad × input ( element-wise multiplication between Saliency Map and the input ) ( Simonyan et al. , 2013 ) in our empirical evaluation . As we show in Section 3.2 , Defs 1-3 satisfy axioms that relate to the local linearity of ReLU networks , and in the case of randomized smoothing ( Cohen et al. , 2019 ) , their robustness to input perturbations . We further discuss these methods relative to others in Sec . 7 . Robustness . Two relevant concepts about adversarial robustness will be used in this paper : prediction robustness that the model ’ s output label remains unchanged within a particular ` p norm ball and attribution robustness that the feature attributions are similar within the same ball . Recent work has identified the model ’ s Lipschitz continuity as a bridge between these two concepts ( Wang et al. , 2020c ) and some loss functions in achieving prediction robustness also bring attribution robustness ( Chalasani et al. , 2020 ) . We refer to robustness as prediction robustness if not otherwise noted . 3 EXPLAINABILITY , DECISION BOUNDARIES , AND ROBUSTNESS . In this section , we begin by discussing the role of decision boundaries in constructing explanations of model behavior via feature attributions . We first illustrate the key relationships in the simpler case of linear models , which contain exactly one boundary , and then generalize to piecewise-linear classifiers as they are embodied by deep ReLU networks . We then show how local robustness causes attribution methods to align more closely with nearby decision boundaries , leading to explanations that better reflect these relationships . 3.1 ATTRIBUTIONS FOR LINEAR MODELS . Consider a binary classifier C ( x ) = sign ( w > x + b ) that predicts a label { −1 , 1 } ( ignoring “ tie ” cases where C ( x ) = 0 , which can be broken arbitrarily ) . In its feature space , C ( x ) is a hyperplane H that separates the input space into two open half-spaces S1 and S2 ( see Fig . 2a ) . Accordingly , the normal vector n̂ of the decision boundary is the only vector that faithfully explains the model ’ s classification while other vectors , while they may describe directions that lead to positive changes in the model ’ s output score , are not faithful in this sense ( see v in Fig . 2a for an example ) . In practice , to assign attributions for predictions made by C , SM , SG , and the integral part of IG ( see Sec . 2 ) return a vector characterized by z = k1n̂ + k2 ( Ancona et al. , 2018 ) , where k1 6= 0 and k2 ∈ R , regardless of the input x that is being explained . In other words , these methods all measure the importance of features by characterizing the model ’ s decision boundary , and are equivalent up to the scale and position of n̂ . 3.2 GENERALIZING TO PIECEWISE-LINEAR BOUNDARIES . In the case of a piecewise-linear model , such as a ReLU network , the decision boundaries comprise a collection of hyperplane segments that partition the feature space , as in H1 , H2 and H3 in the example shown in Figure 2b . Because the boundary no longer has a single well-defined normal , one intuitive way to extend the relationship between boundaries and attributions developed in the previous section is to capture the normal vector of the closest decision boundary to the input being explained . However , as we show in this section , the methods that succeeded in the case of linear models ( SM , SG , and the integral part of IG ) may in fact fail to return such attributions in the more general case of piecewise-linear models , but local robustness often remedies this problem . We begin by reviewing key elements of the geometry of ReLU networks ( Jordan et al. , 2019 ) . ReLU activation polytopes . For a neuron u in a ReLU network f ( x ) , we say that its status is ON if its pre-activation u ( x ) ≥ 0 , otherwise it is OFF . We can associate an activation pattern denoting the status of each neuron for any point x in the feature space , and a half-space Au to the activation constraint u ( x ) ≥ 0 . Thus , for any point x the intersection of the half-spaces corresponding to its activation pattern defines a polytope P ( see Fig . 2b ) , and within P the network is a linear function such that ∀x ∈ P , f ( x ) = w > Px + bP , where the parameters wp and bP can be computed by differentiation ( Fromherz et al. , 2021 ) . Each facet of P ( dashed lines in Fig . 2b ) corresponds to a boundary that “ flips ” the status of its corresponding neuron . Similar to activation constraints , decision boundaries are piecewise-linear because each decision boundary corresponds to a constraint fi ( x ) ≥ fj ( x ) for two classes i , j ( Fromherz et al. , 2021 ; Jordan et al. , 2019 ) . Gradients might fail . Saliency maps , which we take to be simply the gradient of the model with respect to its input , can thus be seen as a way to project an input onto a decision boundary . That is , a saliency map is a vector that is normal to a nearby decision boundary segment . However , as others have noted , a saliency map is not always normal to any real boundary segment in the model ’ s geometry ( see the left plot of Fig . 2c ) , because when the closest boundary segment is not within the activation polytope containing x , the saliency map will instead be normal to the linear extension of some other hyperplane segment ( Fromherz et al. , 2021 ) . In fact , iterative gradient descent typically outperforms the Fast Gradient Sign Method ( Goodfellow et al. , 2015 ) as an attack demonstrates that this is often the case . When gradients succeed . While saliency maps may not be the best approach in general for capturing information about nearby segments of the model ’ s decision boundary , there are cases in which it serves as a good approximation . Recent work has proposed using the Lipschitz continuity of an attribution method to characterize the difference between the attributions of an input x and its neighbors within a ` p ball neighborhood ( Def . 4 ) ( Wang et al. , 2020c ) . This naturally leads to Proposition 1 , which states that the difference between the saliency map at an input and the correct normal to the closest boundary segment is bounded by the distance to that segment . Definition 4 ( Attribution Robustness ) An attribution method g ( x ) is ( λ , δ ) -locally robust at the evaluated point x if ∀x′ ∈ B ( x , δ ) , ||g ( x′ ) − g ( x ) || ≤ λ||x′ − x|| . Proposition 1 Suppose that f has a ( λ , δ ) -robust saliency map gS at x , x′ is the closest point on the closest decision boundary segment to x and ||x′ − x|| ≤ δ , and that n is the normal vector of that boundary segment . Then ||n− gS ( x ) || ≤ λ||x− x′|| . Proposition 1 therefore provides the following insight : for networks that admit robust attributions ( Chen et al. , 2019 ; Wang et al. , 2020c ) , the saliency map is a good approximation to the boundary vector . As prior work has demonstrated the close correspondence between robust prediction and robust attributions ( Wang et al. , 2020c ; Chalasani et al. , 2020 ) , this in turn suggests that explanations on robust models will more closely resemble boundary normals . As training robust models can be expensive , and may not come with guarantees of robustness , post-processing techniques like randomized smoothing ( Cohen et al. , 2019 ) , have been proposed as an alternative . Dombrowski et al . ( 2019 ) noted that models with softplus activations ( y = 1/β log ( 1+exp ( βx ) ) ) approximate smoothing , and in fact give an exact correspondence for singlelayer networks . Combining these insights , we arrive at Theorem 1 , which suggests that the saliency map on a smoothed model approximates the closest boundary normal vector well ; the similarity is inversely proportional to the standard deviation of the noise used to smooth the model . Theorem 1 Let m ( x ) = ReLU ( Wx ) be a one-layer network and when using randomized smoothing , we writemσ ( x ) . Let g ( x ) be the SM formσ ( x ) and suppose ∀x′′ ∈ B ( x , ||x−x′|| ) , ||g ( x′′ ) || ≥ c where x′ is the closest adversarial example , we have the following statement holds : ||g ( x ) − g ( x′ ) || ≤ λ where λ is monotonically decreasing w.r.t σ. Theorem 1 suggests that when randomized smoothing is used , the normal vector of the closest decision boundary segment and the saliency map are similar , and this similarity increases with the smoothness of the model ’ s boundaries . We think the analytical form for deeper networks exists but its expression might be unnecessarily complex due that we need to recursively apply ReLU before computing the integral ( i.e. , the expectation ) . The analytical result above for one-layer network and empirical validations for deeper nets in Figure 11 , if taken together , shows that attributions and boundary-based attributions are more similar in a smoothed model .

Summary: The idea is that as one smoothens the decision boundary of a piecewise linear function $f$ (e.g., from ReLU-Net) its saliency map ($g$) obtained by $g=\frac{df}{dx}$ gets closer ($||g - n||_2$) to the normal of the closest boundary hyperplane ($n$). The authors then propose two variants of explanation techniques based on the nearest decision boundary hyperplane and try it on explaining trained image deep classifiers. The results seem to corroborate as for a better alignment with the nearest boundary hyperplane's normal. Also, the proposed methods achieve better explanations as measured by locality and overlap with ground-truth bounding boxes.