Interventional Black-Box Explanations

1 INTRODUCTION . The design of deep neural networks ( DNNs ) is built on complex structure of neurons , layers and operations ( e.g. , convolutions , non-linearity and back-propagation ) . These biologically-inspired designs are able to evolve by their own from training data . Their high dimensional parameter space allows learning meaningful semantics from large data distributions and perform well on many tasks . However , makes difficult to capture explanation of their behaviour . This is one of fundamental obstacles for using these models on critical systems . Most popular explanation methods focus on creating saliency maps from classification models to visualize important features ( Selvaraju et al . ( 2017b ) ; Binder et al . ( 2016 ) ; Simonyan et al . ( 2014 ) ) of a predicted class . However , saliency maps are not sufficient to reason on model behaviour and the explanations are not consistent between these methods . It is important to explain what are the mechanisms inside the hidden layers by which the model makes a prediction from an input . In this paper , we address this problem using causal inference . Understanding of cause-effect relations in the DNN architecture is one way to make such black box models transparent and to explain their behaviour when tested on new data . Structural causal models ( SCM ) ( Pearl ( 2009 ) ) and their causal diagrams use interventions in terms of the do-calculus to express these relations . We rely on this framework to explain black box DNNs . We are interested in explaining post-hoc models , i.e. , models after training . Recently , some work have addressed DNN explanations using causality ( Chattopadhyay et al . ( 2019 ) ; O ’ Shaughnessy et al . ( 2020 ) ; Narendra et al . ( 2018 ) ) . These methods concentrate on the data generation process and input-output relations in classification models . Their goal is to explain the effects of changing aspects in input data on model predictions . Our work is different because we focus on the model itself , more precisely , the pre-trained knowledge stored in its parameters . We consider deep convolution models which are widely applied to computer vision tasks and have some applications in speech recognition and natural language processing . A neural network architecture is a form of Directed Acyclic Graph ( DAG ) models , in which neurons ( nodes ) are connected by directed edges form one layer to the next . A causal diagram or SCM was used to summarize the complex structure of DNNs ( Chattopadhyay et al . ( 2019 ) ; Narendra et al . ( 2018 ) ) such that interventions can be applied to explain the effect of a variable of interest on model prediction . The construction of the causal diagram depends on the variables of interest that we would like to understand their effects . Some methods ( Chattopadhyay et al . ( 2019 ) ; O ’ Shaughnessy et al . ( 2020 ) ; Harradon et al . ( 2018 ) ) consider the latent features of a generative model as variables of interest . Narendra et al . ( 2018 ) focused on explaining the effect of the variance in convolution filters of a CNN . In this work , we provide a different view of the causal diagram ( see Fig.1 ) which allows to perform causal reasoning on the entire structure of the DNN . We summarize our contributions as follows . We build a causal graph of a post-hoc DNN . We develop an algorithm , termed interventional black box explanations , to find the causal mechanisms that explain the local and global DNN behaviour on individual samples and across samples , respectively . We show that our explanations can be used to correct or improve the probability of a prediction in test time . We consider , in this work , architectures for image classification . We capture explanations for LeNet and ResNet18 architectures using MNIST and ImageNet data inputs . We show that the explanations obtained by our method can be effectively used to remove noise in the model and improve its performance . Finally , like attribution-based methods , we provide visual explanations of classifier ’ s behaviour computing visual concepts ( semantics ) from the effect variables ( response of causal filters ) . We qualitatively demonstrate that our method captures more informative concepts strongly connected to model ’ s prediction and useful for human interpretability . 2 INTERVENTIONAL BLACK BOX EXPLANATION . We begin our method by paving the way to the construction of the proposed causal diagram for post-hoc DNN ( Section 2.1 ) . In Section 2.2 , we define the causal diagram followed describing of the proposed algorithm ( Section 2.3 ) . 2.1 FORMAL SETTING . We will use DNN and CNN interchangeably in this paper as we focus on classification networks comprising hidden convolution layers , but we keep in mind that our method is generic and can be applied on other architectures . In formal setting , a black-box CNN has two major components : a feature extraction module consisting of n convolution layers followed by a classifier with m fully connected layers . An input is an image X ∈ Rd1×d2×c0 , ( d1 , d2 are the spatial dimensions and c0 is the channels number ) . The subscribe number indicates that this is the input layer l0 . An output y ∈ RK ( K is the number of classes ) is a real-valued vector of predictions indicating the class logits of X . The feature extraction module can be a simple structure of sequential layers ( l1 , .... , ln ) where each layer li consists of ci convolution units ( we call them nodes ) and connected with all ci+1 units in layer li+1 . This structure can be more complex in state-of-the-art DNNs , such as ResNets , where skip connections are designed to jump over some layers ( see Fig 1 ( a ) ) . For simplicity , we assume non-linearity , batch normalization and pooling ( if they exist ) as parts of every node in layer li . A layer li in the classifier module consists of vi neurons , and every neuron is densely connected to all vi+1 neurons in layer li+1 . The neurons are naturally nodes in those densely connected layers . We will substitute ci by vi to unify the notations . Consequently , we summarize the graphical structure of the DNN by GM = ( { v0 , ... , vn , .... vN } ) , { E0,1 , ... , EN−1 , N } , with N = n + m and Ei , i+1 is edge vector connecting the nodes in layer li to li+1 as shown in Fig 1 ( b ) . In post-hoc DNNs , each path communicates the information from layer li to layer li+1 in a single direction starting from the input and passing forward to the output ( prediction ) . It is an Acyclic Directed Graph ( DAG ) . Note that this assumption is not valid on the network during training because of backpropagation . The pre-trained knowledge captured from training data is stored in the DNN ’ s parameters ( i.e. , weights ) . We define wki ∈ Rvi as the weight vector connected to the k-th node in layer li+1 . We show in the next section the motivation for this notation . 2.2 THE CAUSAL DIAGRAM OF POST-HOC DNNS . A causal diagram is a graphical model that summarizes an existing knowledge where the nodes represent the variables of interest and edges represent the causal relationships between variables ( Greenland & Pearl ( 2011 ) ) . A causal explanation consists of a causal diagram and symbolic queries defined by interventions , or do-calculus , to express cause-effect relations ( Pearl & Mackenzie ( 2018 ) ) . For post-hoc DNNs , knowledge is encapsulated in its weights during the training phase . For CNNs , the filters of convolution layers express the pieces of knowledge that construct the entangled feature space and concepts of an input data . There are too many parameters to explore , and the goal of the causal graph is to uncover the important ones that explain the behaviour of the model . To explain the effects of these parameters on predictions , we propose a causal diagram as the example shown in Fig . 1 ( c ) . In this graph , we distinguish between two types of variables : parameters and features ( network nodes ) . Our variables of interest are : parameters W = ( W0 , ... , WN−1 ) , test input X and the output ( prediction logits ) y . In between , there are mediator variables ( features ) which transmit the effect of interventions on the parameters of intermediate layers to the output . As we can see , there is no direct effect of parameters variables in intermediate layers ; there is only one direct effect on y which is the effect of WN−1 ( N is the logits layer ) . Also , there is no effect of input data X on parameters variables as they are independent in the case of post-hoc DNNs1 . The parameters variables , wi , and features ui form a collider at ui+1 in layer li+1 . This brings us to the following assumption which is the basic block of our proposed method . Assumption 1 A de-confounded ( robust ) explanation of a black-box DNN can be defined by the measure of changes between the effect distributions P ( y|do ( wki ) , X ) obtained by interventions on wki and the observed distribution ( outcome ) P ( y|X ) , for any i ∈ { 1 , ... , N } and k ∈ { 1 , ... , vi+1 } . It is easy to check the validity of Assumption 1 from the causal graph shown in Fig . 1 ( c ) . First , the graph allows us isolating the parameters of interest and reason on the effect of every parameter on model output . Second , model parameters are not confounded by the skip connections existing in complex models , like ResNet . This is not the case when considering the features space where some variables may have common effects on multiple variables . We show such case in Fig . 1 ( c ) where u1 is a confounder ( u2 ← u1 → y ) . Confounders require adjustment criterion ( Pearl ( 2009 ) ) to block any superiors correlations that give wrong effect estimates . This becomes more difficult in complex DNNs because of many confounders ( Chattopadhyay et al . ( 2019 ) ) . 2.3 FINDING CAUSAL MECHANISMS IN POST-HOC DNN . We are interested in finding the mechanisms ( causal pathways ) along which changes flow from causal variables to the effects . These mechanisms will form explanations of the DNN ’ s behaviour for every data input ( local explanations ) and across inputs ( global explanations ) . For simplicity , let us first consider the example in Fig 1 ( c ) , where we have a single parameter w in each layer . The causal model shows that the effect of a causal parameter wki , or filter for convolution layers , on model output ( y ) is mediated by the effect of changes on variables ( called mediators ) along the paths from layers li+1 , ... , lN−1 after intervening on wki . These changes will also transmit to the output variable in lN . Generally , each layer has multiple neurons and we may apply interventions on a set of selected parameters ( k ∈ I ) to analyze their combined effect on model ’ s output . 1The parameters update their values in the training phase using the training data inputs , which creates a causal path between data inputs and model parameters . The interventions do ( wki ) imply changing the values of w k i . In convolution layers , each w k i is a tiny set of pixels ( e.g. , 9 in case of 3 × 3 filter ) . Changing one single value would not carry out significant changes on the output . Therefore , we consider changing all the values of the filter . In fully connected layers , a parameter is a scalar value . The effect of changing one single parameter in the penultimate layer of the classifier would be highly significant to the output . To find the causal mechanisms , we start by quantifying the direct effect on the prediction variable yj ( for class j ) , which are in this case the parameters wjN−1 . We intervene on every single parameter and analyze its effect . Finding the direct effects enables identifying the causal paths in the network . However , for intermediate layers we can not do interventions on every single parameter . This is computationally very expensive and not practical in inference time . Moreover , deep networks are highly parametrized , which would lead to a high variance in cause-effect relations thus making difficult to capture robust explanations . We propose a robust selection criterion to select the variables that we want to intervene on . This leads to the following proposition . Proposition 1 ( Intervention variables in intermediate layers ) For wKi , the set of indices K ∈ { 1 , ... , vi+1 } of the mediators in intermediate layer li+1 and I ∈ { 1 , ... , vi } a subset of indices corresponding to the causal parameters wKI , i . Then there exist causal paths between uI , i and uK , i+1 and I will identify the intervention variables in li−1 We proof Proposition 1 on simple example in Appendix A . As we notice , we measure changes in retrospective way , so the selection criterion is defined for layer li−1 after finding the causal parameters in layer li . To find the indices I for layer li−1 , we focus on changes which are statistically significant using a threshold δi which is defined as δi = ( µỹ − µj ) /σỹ ( 1 ) where µj is the original prediction ( reference ) of the true class j , and µỹ is the average value of changed prediction and σỹ is the standard deviation . To capture the informativeness in causal parameters , we introduce in Proposition 2 the formula for the causal effect . Proposition 2 ( Causal effect ) Given an input X and let WKi be the set of causal variables ( or paths ) to the K neurons in li+1 , the causal effect ( CSi ) of do ( WKi ) is a measure of the information flow from WKi to y which is defined as : CSi ( n ) = Ey [ log p ( y|wKn , i = αn , X ) log p ( y|xs ) ] ( 2 ) where p ( y|X ) is the Softmax of the prediction logits of an input X , n = ( 0 , ... , vi − 1 ) , and αn are the change values for each single parameter . We put details of Proposition 2 in Appendix A . The interventions that we used in this work are αn = 0 . Propositions 1 and 2 uncover the causal mechanism of a hidden layer i in the DNN ’ s architecture . Algorithm 1 describes how we apply them to capture the causal mechanisms in all layers .

Summary: This paper proposed a causal-driven method aiming to resolve the black-box issue of DNNs. The proposed interventional black-box explanations method tends to be model-agnostic and can apply a variety of DNN models. In the experiments, the authors examined the proposed method on two well-known DNN architectures LeNet and ResNet18.