Scaling Symbolic Methods using Gradients for Neural Model Explanation

1 INTRODUCTION . Satisfiability Modulo Theory ( SMT ) solvers ( Barrett & Tinelli , 2018 ) are routinely used for symbolic modeling and verifying correctness of software programs ( Srivastava et al. , 2009 ) , and more recently they have also been used for verifying properties of deep neural networks ( Katz et al. , 2017 ) . SMT Solvers in their current form are difficult to scale to large networks . Model explanation is one such domain where SMT solvers have been used but they are limited to very small sized networks ( Gopinath et al. , 2019 ; Ignatiev et al. , 2019 ) . The goal of our work is to address the issue of scalability of SMT solvers by using gradient information , thus enabling their use for different applications . In this work , we present a new application of SMT solvers for explaining neural network decisions . Model explanation can be viewed as identifying a minimal set of features in a given input that is critical to a model ’ s prediction ( Carter et al. , 2018 ; Macdonald et al. , 2019 ) . Such a problem formulation for identifying a minimal set lends itself to the use of SMT solvers for this task . We can encode a neural network using real arithmetic ( Katz et al. , 2017 ) and use an SMT solver to optimize over the constraints to identify a minimal set of inputs that can explain the prediction . However , there are two key challenges in this approach . First , we can not generate reliable explanations based on final model prediction as the minimal input is typically out of distribution . Second , solving such a formulation is challenging for SMT solvers as the decision procedures for solving these constraints have exponential complexity , and is further exacerbated by the large number of parameters in typical neural network models . Thus , previous approaches for SMT-based analysis of neural networks have been quite limited , and have only been able to scale to networks with few thousands of parameters . To solve these challenges , instead of doing minimization by encoding the entire network , our approach takes advantage of the gradient information , specifically Integrated Gradients ( IG ) ( Sundararajan et al. , 2017 ) , in lieu of encoding the deeper layers , and encodes a much simpler set of linear constraints pertaining to the layer closest to the input . We encode the mathematical equations of a neural network as SMT constraints using the theory of Linear Real Arithmetic ( LRA ) , and use z3 solver ( Bjørner et al. , 2015 ) as it additionally supports optimization constraints such as minimization . The SMT solver then finds a minimal subset ( of input features ) by performing minimization on these equations . Thus , our approach , which we refer to as SMUG , is able to scale Symbolic Methods Using Gradient information while still providing a faithful explanation of the neural network ’ s decision . SMUG is built on two properties . First , based on the target prediction , SMUG uses gradient information propagated from the deeper layers to identify neurons that are important in the first layer , and only encodes those . For this , we use IG ( Sundararajan et al. , 2017 ) instead of relying on gradients alone . Second , for the optimization , a set of input pixels are determined to be relevant for prediction if they are able to activate the neurons deemed important , and maintain a fraction of their activation as the original ( full ) input image . Empirically , we observe good performance on visual and text classification tasks . We evaluate SMUG on three datasets : MNIST ( LeCun et al. , 2010 ) , ImageNet ( Deng et al. , 2009 ) , and Beer Reviews ( McAuley et al. , 2012 ) . We show that we can fully encode the minimal feature identification problem for a small feedforward network ( without gradient-based neuron selection ) for MNIST , but this full SMT encoding scales poorly for even intermediate sized networks . On ImageNet , we observe that our method performs better than Integrated Gradients ( Sundararajan et al. , 2017 ) and several strong baselines . Additionally , we observe that our approach finds significantly sparser masks ( on average 17 % of the original image size ) . Finally , we also show that our technique is applicable to text models where it performs competitively with other methods including SIS ( Carter et al. , 2018 ) and Integrated Gradients ( Sundararajan et al. , 2017 ) . This paper makes the following key contributions : • We present a technique ( SMUG ) to encode the minimal input feature discovery problem for neural model explanation using SMT solvers . Our approach , which does masking on linear equations also overcomes the issue of handling out-of-distribution samples . • Our approach uses gradient information to scale SMT-based analysis of neural networks to larger models and input features . Further , it also overcomes the issue of choosing a “ baseline ” parameter for Integrated Gradients ( Kapishnikov et al. , 2019 ; Sturmfels et al. , 2020 ) . • We empirically evaluate SMUG on image and text datasets , and show that the minimal features identified by it are both quantitatively and qualitatively better than several baselines . • To improve our understanding on saliency map evaluation , we show how the popular and widely used LSC metric ( Dabkowski & Gal , 2017 ) can be gamed heuristically to generate explanations that are not necessarily faithful to the model ( Sec . 5.2 ) 2 RELATED WORK . SMT based symbolic techniques have been used for verifying neural network properties ( Huang et al. , 2017 ; Katz et al. , 2017 ) . Reluplex ( Katz et al. , 2017 ) extends the simplex method to handle ReLU functions by leveraging its piecewise linear property and presents an iterative procedure for gradual satisfaction of the constraints . ( Huang et al. , 2017 ) proposes a layer-wise analysis using a refinement-based approach with SMT solvers for verifying the absence of adversarial input perturbations . ( Zhang et al. , 2018 ) present a linear programming ( LP ) formulation again using the piecewise linear property of ReLU to find minimal changes to an input to change a network ’ s classification decisions . ( Gopinath et al. , 2019 ) uses Reluplex to learn input properties in the form of convex predicates over neuron activations , which in turn capture different behaviors of a neural network . While SMT based techniques from ( Gopinath et al. , 2019 ) , ( Ignatiev et al. , 2019 ) have shown promising results , they only scale to Neural Network with 5000 nodes . Thus , scaling these approaches for larger neural networks and performing richer analysis based on global input features still remains a challenge . We present and demonstrate an approach that works for larger and more complex image and text models . While most of the above SMT based techniques focus on verifying properties of deep networks , our work focuses on applying symbolic techniques to the related task of model explanation , i.e . to say where a model is “ looking ” , by solving for the input features responsible for a model ’ s prediction . Some explanation techniques are model agnostic ( i.e. , black-box ) while others are back-propagation based . Model agnostic ( black-box ) explanation techniques such as SIS , LIME ( Alvarez-Melis & Jaakkola , 2018 ; Carter et al. , 2018 ) have a similar formulation of the problem as ours in the sense that they perturb the input pixels by masking them and optimize to identify minimal regions affecting the performance of the model . This formulation can lead to evaluating the model on out of distribution samples ( Hooker et al. , 2019 ) with potential for adversarial attacks ( Slack et al. , 2020 ) . In contrast , back-propagation based methods ( Bach et al. , 2015 ; Sundararajan et al. , 2017 ; Selvaraju et al. , 2016 ) examine the gradients of the model with respect to an input instance to determine pixel attribution . Our work builds on the IG method ( Sundararajan et al. , 2017 ) . IG integrates gradients along the “ intensity ” path where the input ( image or text embedding ) is scaled from an information-less baseline ( all zeros input , e.g. , all black or random noise image ) to a specific instance . This helps the model determine attribution at the pixel level . In our work , we use IG to determine important nodes in the first layer ( closest to the input ) . The key improvement of our technique over IG is that , by using IG only on the first layer and then using SMT solver based solution to determine saliency on the input , we not only preserve faithfulness , but also overcome the issue of choosing an appropriate baseline image for IG ( Kapishnikov et al. , 2019 ; Sturmfels et al. , 2020 ; Xu et al. , 2020 ) . 3 METHOD : SCALING SYMBOLIC METHODS USING GRADIENTS ( SMUG ) . We describe our approach SMUG , which combines attribution based on gradient information with an SMT-based encoding of the minimal input feature identification problem . We then show how to generate saliency maps from the predicted boolean mask for image and text applications . 3.1 CHALLENGES WITH ENCODING NEURAL NETWORKS FOR EXPLANATION . Previous works ( Katz et al. , 2017 ) have shown it ’ s possible to use SMT solvers to encode the semantics of neural networks with specific activation functions , while others ( Carter et al. , 2018 ; Macdonald et al. , 2019 ) have used the idea of minimality to identify importance of inputs in the context of model explanation . In particular , given a neural network Nθ ( θ denotes the parameters ) , let X ∈ Rm×n denote an input image with m× n pixels , M ∈ { 0 , 1 } m×n an unknown binary mask , and Li ( · ) the output ( i.e. , activations ) of the ith layer . Also , let L|L|−1 ( Nθ ( X ) ) denote the logits of the final layer . Previous works ( Carter et al. , 2018 ; Macdonald et al. , 2019 ) have used the following formulation for explanation . min ( ∑ ij Mij ) : argmaxL|L|−1 ( Nθ ( X ) ) = argmaxL|L|−1 ( Nθ ( M X ) ) However , there are two key problems with this formulation . First , the masked inputs found by solving this formulation are typically out of training distribution , so explanation techniques based on model confidence over a forward pass of the masked inputs ( such as the above formulation ) no longer produce reliable explanations . Second , using such a formulation directly with SMT solvers is not tractable as the number of constraints in the encoding grows linearly with increasing network size and SMT decision procedure for Non-linear Real Arithmetic is doubly exponential . Thus , when we apply such an encoding even for a small feed-forward network on MNIST dataset , the SMT solver does not scale well ( Section 4 , Appendix A ) . 3.2 OUR SOLUTION : COMBINING GRADIENTS WITH SYMBOLIC ENCODING . Our proposal is to use gradient based attributions to overcome both these issues : 1 ) as a proxy for the deeper layers ( for scalability ) , and 2 ) to identify a subset of neurons in the first layer capturing information crucial for the model ’ s prediction , and encode them using SMT which avoids performing a forward pass on out-of-distribution inputs . Specifically , we use Integrated Gradients ( IG ) ( Sundararajan et al. , 2017 ) to score the neurons in order of relevance by treating the first layer activations ( L1 ) as an input to the subsequent network . This assigns an attribution score to each hidden node where a node with a positive score means that the information captured by that neuron is relevant to the prediction and a node with a negative / zero score is considered irrelevant . More specifically , suppose F : Ra×b×c → [ 0 , 1 ] d represents a deep network and x represents the input . F often refers to the full network ( from the input to the final softmax ) but in our case , it refers to the nodes in the network second through the final layer . a , b , and c refer to the dimensions of the tensor input to the network ( analogous to width , height , and channels ) and d is the number of output nodes in the final softmax layer ( number of classes ) . Integrated gradients are obtained by accumulating the gradients at all points along the straight line path from an “ information-less ” baseline x′ to the input x . The information-less baseline in this case is an all zeros tensor . The path can be parameterized as g ( α ) = x′ + α · ( x− x′ ) . IG is then given by Eq . 1 . IG ( x ) = ( x− x′ ) ∫ 1 α=0 ∂F ( g ( α ) ) ∂g ( α ) dα ( 1 ) Top-k Neurons . The second key idea of our approach is that we only consider activations with the highest positive attributions . Empirically , we still observe scaling issues with SMT when considering all first layer neurons ( L1 ) with +ve attributions , so a method for picking a subset of neurons ( top-k attributions ) is important for practical application . With these two ideas , we can now formulate our problem as follows . Let , Dk represent the set of k important nodes with the highest attributions in IG ( Nθ ( X ) ) , and γ be a parameter which regulates how “ active ” the neurons remain after masking1.The goal now is to learn a mask M such that : min ( ∑ ij Mij ) : L1 ( Nθ ( M X ) ) t > γ · L1 ( Nθ ( X ) ) t ∀t ∈ Dk ( 2 ) Inducing Sparsity . IG ensures that the information captured by the neurons in Dk are relevant for the model ’ s prediction . Thus , the set of features corresponding to these neurons contributes towards the relevant information . Let ’ s denote this set as S. To filter less relevant features from S we introduce a notion of minimization . By inducing sparsity we wish to remove any false positives from S. Thus , we define `` relevant features '' as a minimal subset of features in S that causes the nodes in Dk to remain active . The notion being - if for a masked input the `` important nodes '' ( w.r.t the original input ) are active then the information relevant for the prediction of the original image is contained in the mask . Thus , the mask highlights features relevant for the model ’ s prediction . Thus , this formalism allows us to come up with constraints which don ’ t involve a full forward pass of the masked image while at the same time keeping things scalable for the SMT solver .

Summary: This paper provides an interesting pos-hoc explanation method to identify relevant features in an input that may inform a trained neural model's prediction. The task is to identify a binary mask over input image/text such that the masked input yields almost similar prediction as original input. The author formulates this as an SMT solver task, but instead of making sure that the output prediction is similar (which involve multiple time consuming pass over potentially huge networks), they make sure that high influential neurons in first layer of the network are still activated. This provides a less time consuming way to evaluate invariance of masked input.