Interventional Black-Box Explanations

1 INTRODUCTION . The design of deep neural networks ( DNNs ) is built on complex structure of neurons , layers and operations ( e.g. , convolutions , non-linearity and back-propagation ) . These biologically-inspired designs are able to evolve by their own from training data . Their high dimensional parameter space allows learning meaningful semantics from large data distributions and perform well on many tasks . However , makes difficult to capture explanation of their behaviour . This is one of fundamental obstacles for using these models on critical systems . Most popular explanation methods focus on creating saliency maps from classification models to visualize important features ( Selvaraju et al . ( 2017b ) ; Binder et al . ( 2016 ) ; Simonyan et al . ( 2014 ) ) of a predicted class . However , saliency maps are not sufficient to reason on model behaviour and the explanations are not consistent between these methods . It is important to explain what are the mechanisms inside the hidden layers by which the model makes a prediction from an input . In this paper , we address this problem using causal inference . Understanding of cause-effect relations in the DNN architecture is one way to make such black box models transparent and to explain their behaviour when tested on new data . Structural causal models ( SCM ) ( Pearl ( 2009 ) ) and their causal diagrams use interventions in terms of the do-calculus to express these relations . We rely on this framework to explain black box DNNs . We are interested in explaining post-hoc models , i.e. , models after training . Recently , some work have addressed DNN explanations using causality ( Chattopadhyay et al . ( 2019 ) ; O ’ Shaughnessy et al . ( 2020 ) ; Narendra et al . ( 2018 ) ) . These methods concentrate on the data generation process and input-output relations in classification models . Their goal is to explain the effects of changing aspects in input data on model predictions . Our work is different because we focus on the model itself , more precisely , the pre-trained knowledge stored in its parameters . We consider deep convolution models which are widely applied to computer vision tasks and have some applications in speech recognition and natural language processing . A neural network architecture is a form of Directed Acyclic Graph ( DAG ) models , in which neurons ( nodes ) are connected by directed edges form one layer to the next . A causal diagram or SCM was used to summarize the complex structure of DNNs ( Chattopadhyay et al . ( 2019 ) ; Narendra et al . ( 2018 ) ) such that interventions can be applied to explain the effect of a variable of interest on model prediction . The construction of the causal diagram depends on the variables of interest that we would like to understand their effects . Some methods ( Chattopadhyay et al . ( 2019 ) ; O ’ Shaughnessy et al . ( 2020 ) ; Harradon et al . ( 2018 ) ) consider the latent features of a generative model as variables of interest . Narendra et al . ( 2018 ) focused on explaining the effect of the variance in convolution filters of a CNN . In this work , we provide a different view of the causal diagram ( see Fig.1 ) which allows to perform causal reasoning on the entire structure of the DNN . We summarize our contributions as follows . We build a causal graph of a post-hoc DNN . We develop an algorithm , termed interventional black box explanations , to find the causal mechanisms that explain the local and global DNN behaviour on individual samples and across samples , respectively . We show that our explanations can be used to correct or improve the probability of a prediction in test time . We consider , in this work , architectures for image classification . We capture explanations for LeNet and ResNet18 architectures using MNIST and ImageNet data inputs . We show that the explanations obtained by our method can be effectively used to remove noise in the model and improve its performance . Finally , like attribution-based methods , we provide visual explanations of classifier ’ s behaviour computing visual concepts ( semantics ) from the effect variables ( response of causal filters ) . We qualitatively demonstrate that our method captures more informative concepts strongly connected to model ’ s prediction and useful for human interpretability . 2 INTERVENTIONAL BLACK BOX EXPLANATION . We begin our method by paving the way to the construction of the proposed causal diagram for post-hoc DNN ( Section 2.1 ) . In Section 2.2 , we define the causal diagram followed describing of the proposed algorithm ( Section 2.3 ) . 2.1 FORMAL SETTING . We will use DNN and CNN interchangeably in this paper as we focus on classification networks comprising hidden convolution layers , but we keep in mind that our method is generic and can be applied on other architectures . In formal setting , a black-box CNN has two major components : a feature extraction module consisting of n convolution layers followed by a classifier with m fully connected layers . An input is an image X ∈ Rd1×d2×c0 , ( d1 , d2 are the spatial dimensions and c0 is the channels number ) . The subscribe number indicates that this is the input layer l0 . An output y ∈ RK ( K is the number of classes ) is a real-valued vector of predictions indicating the class logits of X . The feature extraction module can be a simple structure of sequential layers ( l1 , .... , ln ) where each layer li consists of ci convolution units ( we call them nodes ) and connected with all ci+1 units in layer li+1 . This structure can be more complex in state-of-the-art DNNs , such as ResNets , where skip connections are designed to jump over some layers ( see Fig 1 ( a ) ) . For simplicity , we assume non-linearity , batch normalization and pooling ( if they exist ) as parts of every node in layer li . A layer li in the classifier module consists of vi neurons , and every neuron is densely connected to all vi+1 neurons in layer li+1 . The neurons are naturally nodes in those densely connected layers . We will substitute ci by vi to unify the notations . Consequently , we summarize the graphical structure of the DNN by GM = ( { v0 , ... , vn , .... vN } ) , { E0,1 , ... , EN−1 , N } , with N = n + m and Ei , i+1 is edge vector connecting the nodes in layer li to li+1 as shown in Fig 1 ( b ) . In post-hoc DNNs , each path communicates the information from layer li to layer li+1 in a single direction starting from the input and passing forward to the output ( prediction ) . It is an Acyclic Directed Graph ( DAG ) . Note that this assumption is not valid on the network during training because of backpropagation . The pre-trained knowledge captured from training data is stored in the DNN ’ s parameters ( i.e. , weights ) . We define wki ∈ Rvi as the weight vector connected to the k-th node in layer li+1 . We show in the next section the motivation for this notation . 2.2 THE CAUSAL DIAGRAM OF POST-HOC DNNS . A causal diagram is a graphical model that summarizes an existing knowledge where the nodes represent the variables of interest and edges represent the causal relationships between variables ( Greenland & Pearl ( 2011 ) ) . A causal explanation consists of a causal diagram and symbolic queries defined by interventions , or do-calculus , to express cause-effect relations ( Pearl & Mackenzie ( 2018 ) ) . For post-hoc DNNs , knowledge is encapsulated in its weights during the training phase . For CNNs , the filters of convolution layers express the pieces of knowledge that construct the entangled feature space and concepts of an input data . There are too many parameters to explore , and the goal of the causal graph is to uncover the important ones that explain the behaviour of the model . To explain the effects of these parameters on predictions , we propose a causal diagram as the example shown in Fig . 1 ( c ) . In this graph , we distinguish between two types of variables : parameters and features ( network nodes ) . Our variables of interest are : parameters W = ( W0 , ... , WN−1 ) , test input X and the output ( prediction logits ) y . In between , there are mediator variables ( features ) which transmit the effect of interventions on the parameters of intermediate layers to the output . As we can see , there is no direct effect of parameters variables in intermediate layers ; there is only one direct effect on y which is the effect of WN−1 ( N is the logits layer ) . Also , there is no effect of input data X on parameters variables as they are independent in the case of post-hoc DNNs1 . The parameters variables , wi , and features ui form a collider at ui+1 in layer li+1 . This brings us to the following assumption which is the basic block of our proposed method . Assumption 1 A de-confounded ( robust ) explanation of a black-box DNN can be defined by the measure of changes between the effect distributions P ( y|do ( wki ) , X ) obtained by interventions on wki and the observed distribution ( outcome ) P ( y|X ) , for any i ∈ { 1 , ... , N } and k ∈ { 1 , ... , vi+1 } . It is easy to check the validity of Assumption 1 from the causal graph shown in Fig . 1 ( c ) . First , the graph allows us isolating the parameters of interest and reason on the effect of every parameter on model output . Second , model parameters are not confounded by the skip connections existing in complex models , like ResNet . This is not the case when considering the features space where some variables may have common effects on multiple variables . We show such case in Fig . 1 ( c ) where u1 is a confounder ( u2 ← u1 → y ) . Confounders require adjustment criterion ( Pearl ( 2009 ) ) to block any superiors correlations that give wrong effect estimates . This becomes more difficult in complex DNNs because of many confounders ( Chattopadhyay et al . ( 2019 ) ) . 2.3 FINDING CAUSAL MECHANISMS IN POST-HOC DNN . We are interested in finding the mechanisms ( causal pathways ) along which changes flow from causal variables to the effects . These mechanisms will form explanations of the DNN ’ s behaviour for every data input ( local explanations ) and across inputs ( global explanations ) . For simplicity , let us first consider the example in Fig 1 ( c ) , where we have a single parameter w in each layer . The causal model shows that the effect of a causal parameter wki , or filter for convolution layers , on model output ( y ) is mediated by the effect of changes on variables ( called mediators ) along the paths from layers li+1 , ... , lN−1 after intervening on wki . These changes will also transmit to the output variable in lN . Generally , each layer has multiple neurons and we may apply interventions on a set of selected parameters ( k ∈ I ) to analyze their combined effect on model ’ s output . 1The parameters update their values in the training phase using the training data inputs , which creates a causal path between data inputs and model parameters . The interventions do ( wki ) imply changing the values of w k i . In convolution layers , each w k i is a tiny set of pixels ( e.g. , 9 in case of 3 × 3 filter ) . Changing one single value would not carry out significant changes on the output . Therefore , we consider changing all the values of the filter . In fully connected layers , a parameter is a scalar value . The effect of changing one single parameter in the penultimate layer of the classifier would be highly significant to the output . To find the causal mechanisms , we start by quantifying the direct effect on the prediction variable yj ( for class j ) , which are in this case the parameters wjN−1 . We intervene on every single parameter and analyze its effect . Finding the direct effects enables identifying the causal paths in the network . However , for intermediate layers we can not do interventions on every single parameter . This is computationally very expensive and not practical in inference time . Moreover , deep networks are highly parametrized , which would lead to a high variance in cause-effect relations thus making difficult to capture robust explanations . We propose a robust selection criterion to select the variables that we want to intervene on . This leads to the following proposition . Proposition 1 ( Intervention variables in intermediate layers ) For wKi , the set of indices K ∈ { 1 , ... , vi+1 } of the mediators in intermediate layer li+1 and I ∈ { 1 , ... , vi } a subset of indices corresponding to the causal parameters wKI , i . Then there exist causal paths between uI , i and uK , i+1 and I will identify the intervention variables in li−1 We proof Proposition 1 on simple example in Appendix A . As we notice , we measure changes in retrospective way , so the selection criterion is defined for layer li−1 after finding the causal parameters in layer li . To find the indices I for layer li−1 , we focus on changes which are statistically significant using a threshold δi which is defined as δi = ( µỹ − µj ) /σỹ ( 1 ) where µj is the original prediction ( reference ) of the true class j , and µỹ is the average value of changed prediction and σỹ is the standard deviation . To capture the informativeness in causal parameters , we introduce in Proposition 2 the formula for the causal effect . Proposition 2 ( Causal effect ) Given an input X and let WKi be the set of causal variables ( or paths ) to the K neurons in li+1 , the causal effect ( CSi ) of do ( WKi ) is a measure of the information flow from WKi to y which is defined as : CSi ( n ) = Ey [ log p ( y|wKn , i = αn , X ) log p ( y|xs ) ] ( 2 ) where p ( y|X ) is the Softmax of the prediction logits of an input X , n = ( 0 , ... , vi − 1 ) , and αn are the change values for each single parameter . We put details of Proposition 2 in Appendix A . The interventions that we used in this work are αn = 0 . Propositions 1 and 2 uncover the causal mechanism of a hidden layer i in the DNN ’ s architecture . Algorithm 1 describes how we apply them to capture the causal mechanisms in all layers .

Summary: The authors aim to give post hoc explanations of Neural Network classifier decisions. They do so by finding causal relationships between model parameters and classifier outputs. To this end model parameters are set to zero and the change in the models prediction is calculated. If the change in prediction exceeds a threshold the parameter is deemed relevant and used for construction of an attribution map.