Interventional Black-Box Explanations
Authors: Ola Ahmad
Simon Corbeil
Vahid Hashemi
Freddy Lecue
Venue: ICLR 2022 Submitted (2022)
Keywords: Causal Inference
Interventions
black-Box Models
Explanations
Deep Neural Networks

Abstract: Deep Neural Networks (DNNs) are powerful systems able to freely evolve on their own from training data. However, like any highly parametrized mathematical model, capturing the explanation of any prediction of such models is rather difficult. We believe that there exist relevant mechanisms inside the structure of post-hoc DNNs that supports transparency and interpretability. To capture these mechanisms, we quantify the effects of parameters (pieces of knowledge) on models' predictions using the framework of causality. We introduce a general formalism of the causal diagram to express cause-effect relations inside the DNN's architecture. Then, we develop a novel algorithm to construct explanations of DNN's predictions using the $do$-operator. We call our method, Interventional Black-Box Explanations. On image classification tasks, we explain the behaviour of the model and extract visual explanations from the effects of the causal filters in convolution layers. We qualitatively demonstrate that our method captures more informative concepts compared to traditional attribution-based methods.         
Finally, we believe that our method is orthogonal to logic-based explanation methods and can be leveraged to improve their explanations.

Under review as a conference paper at ICLR 2022
INTERVENTIONAL BLACK-BOX EXPLANATIONS
Anonymous authors
Paper under double-blind review
ABSTRACT
Deep Neural Networks (DNNs) are powerful systems able to freely evolve on their from training data. However, like any highly parametrized mathemati- cal model, capturing the explanation of any prediction of such models is rather difficult. We believe that there exist relevant mechanisms inside the structure of post-hoe DNNs that supports transparency and interpretability. To capture these mechanisms, we quantify the effects of parameters (pieces of knowledge) on models' predictions using the framework of causality. We introduce a gen- eral formalism of the causal diagram to express cause-effect relations inside the DNN's architecture. Then, we develop novel algorithm to construct explanations DNN's predictions using the do-operator. We call our method, Interventional Black-Box Explanations. On image classification tasks, explain the behaviour of the model and extract visual explanations from the effects of the causal filters convolution layers. We qualitatively demonstrate that our method captures more informative concepts compared to traditional attribution- based methods. Finally, we believe that our method is orthogonal to logic-based explanation methods and can be leveraged to improve their explanations.
INTRODUCTION
The design of deep neural networks (DNNs) is built on complex structure of neurons, layers and operations (e.g.. convolutions, non-linearity and back-propagation). These piologically-insiree de- signs are able to evolve by their own from training data. Their high dimensional parameter space allows learning meaningful semantics from large data distributions and perform well on many tasks. However, makes difficult to capture explanation of their behaviour. This is one of fundamental obstacles for using these models on critical systems. Most popular explanation methods focus on creating saliency maps from classification models to visualize important features (Selvaraju et al. (2017b): [Binder et al.) (2016); [Simonyan et al. (2014) of a predicted class. However, saliency maps are not sufficient to reason on model behaviour and the explanations are not consistent between these methods. is important to explain what are the mechanisms inside the hidden layers by which the model makes prediction from an input. In this paper, address this problem using causal inference.
Understanding of cause-effecct relations in the DNN architecture is way to make such black box models transparent and to explain their behaviour when tested on new data. Structural causal models (SCM) (Pearl ) and their causal diagrams use interventions in terms of the do-calculus to express these relations. We rely on this framework to explain black box DNNs. We are interested in explaining post-hoc models, i.e., models after training. Recently, some work have addressed DNN explanations using causality (Chattopadhyay et al. (2019): O Shaughnessy et al. (2020); Narendra al. (2018)). These methods concentrate on the data generation process and input-output relations in classification models. Their goal is to explain the effects of changing aspects in input data on model predictions. Our work is different because we focus on the model itself, more precisely, the pre-trained knowledge stored in its parameters. We consider deep convolution models which are widely applied to computer vision tasks and have some applications in speech recognition and natural language processing. A neural network architecture is a form of Directed Acyclic Graph (DAG) models, in which neurons (nodes) are connected by directed edges form one layer to the next. A causal diagram SCM was used to summarize the complex structure of DNNs (Chattopadhyay jtt al. (2019); Narendra et al. (2018)) such that interventions can be applied to explain the effect of variable of interest model prediction. The construction of the causal diagram depends the

---

Under review as a conference paper at ICLR 2022
LeNetS
conv unit in
Input
neeiduel Block neeheel
Wn
Output
affecting
Un in
[b)
(C)
Figure 1: The proposed causal diagram of DNN model. (a) Examples of DNNs with convolution
layers. (b) A graphical abstraction of the DNNs architectures to a simple DAG graph. Each node u; represents one channel in hidden layer. (c) Illustrates the proposed causal diagram on an example of a CNN with 2 hidden layers, each composed of one convolution unit.
variables of interest that we would like to understand their effects. Some methods (Chattopadhyay et al. (2019): O""Shuugnnessy et al. (2020); Harradon et al. (2018)) consider the latent features of generative model as variables of interest. Narendra et al. (2018) 18) focused on explaining the effect of the variance in convolution filters of CNN. In this work, we provide different view of the causal diagram (see [ which allows to perform causal reasoning on the entire structure of the DNN.
We summarize contributions follows. We build causal graph of post-hoc DNN. We de- velop an algorithm, termed interventional black box explanations, to find the causal mechanisms that explain the local and global DNN behaviour individual samples and across samples, respectively. We show that our explanations can be used to correct or improve the probability of a prediction in test time. We consider, in this work, architectures for image classification. We capture explanations for LeNet and ResNet18 8 architectures using MNIST and ImageNet data inputs. We show that the explanations obtained by our method can be effectively used to remove noise in the model and im- prove its performance. Finally, like attribution-bart methods, we provide visual explanations of classifier's 's behaviour computing visual concepts (semantics) from the effect variables (response of causal filters). We qualitatively demonstrate that our method captures more informative concepts strongly connected to model's prediction and useful for human interpretability.
INTERVENTIONAL BLACK BOX EXPLANATION
We begin our method by paving the way the construction of the proposed causal diagram for post-hoc DNN (Section 2.1. In Section 2.2 we define the causal diagram followed describing of the proposed algorithm (Section 2.3
2.1 FORMAL SETTING
We will use DNN and CNN interchangeably in this paper focus on classification networks comprising hidden convolution layers, but we keep in mind that our method is generic and can be applied other architectures. In formal setting, black-box CNN has two major components: feature extraction module consisting of convolution layers followed by a classifier with mn fully connected layers. An input an image X E Rdh xd2x66 (di, d2 the spatial dimensions and co the channels number). The subscribe number indicates that this is the input layer lo- An output y K (K the number of classes) is real-valued vector of predictions indicating the class logits of X. The feature extraction module can be simple structure of sequential layers (4, 4x) where each layer 1, consists of 6; convolution units (we call them nodes) and connected with all c++1 units in layer 4+1. This structure can be more complex in state-of-ther tfftth--art DNNs, such as ResNets, where skip connections are designed to jump over some layers (see Fig M aa). For simplicity, we assume non-linearity, batch normalization and pooling (if they exist) as parts of every node in layer 4, A layer 4, in the classifier module consists of v; neurons, and every neuron is densely connected to all U441 neurons in layer 441. The neurons naturally nodes in those densely connected layers. We will substitute o; by v, to unify the notations. Consequently, we summarize the graphical structure of the DNN by GM ((vo, -. Eny UN) }); (Fo,1,... EN-I,N), with N
and E is
2

---

Under review as a conference paper at ICLR 2022
edge vector connecting the nodes in layer 1, to 441 shown in Fig [b). In post-hoc DNNs, each path communicates the information from layer 4, to layer 4+1 in a single direction starting from the input and passing forward to the output (prediction). It is Acyelic Directed Graph (DAG). Note that this assumption is not valid on the network during training because of backpropagation. The pre-trained knowledge captured from training data is stored in the DNN's parameters (i.e., weights). We define w E _t as the weight vector connected to the k-th node in layer 4,41. We show in the next section the motivation for this notation.
2.2 THE CAUSAL DIAGRAM OF POST-HOC DNNs
A causal diagram is a graphical model that summarizes an existing knowledge where the nodes rep- resent the variables of interest and edges represent the causal relationships between variables (Green- land & Pearl (2011). A causal explanation consists of a causal diagram and symbolic queries de- fined by interventions, or do-calculus, to express cause-effect relations (Pearl & Mackenzie (2018) ). For post-hoe DNNs, knowledge is encapsulated in its weights during the training phase. For CNNs, the filters of convolution layers express the pieces knowledge that construct the entangled feature space and concepts of an input data. There are too many parameters to explore, and the goal of the causal graph is to uncover the important ones that explain the behaviour of the model. To explain the effects of these parameters on predictions, we propose a causal diagram as the example shown in Fig. [c). In this graph, we distinguish between two types of variables: parameters and features (network nodes). Our variables of interest are: parameters W = (Wo- .... WN-1), test input X and the output (prediction logits) y. In between, there are mediator variables (features) which transmit the parameters of intermediate layers the output. As we can see,
the effect of interventions there is no direct effect of parameters variables in intermediate layers; there is only one direct effect on y which is the effect of WN (N is the logits layer). Also, there is no effect of input data X on parameters variables as they are independent in the case of post-hoc [NN The parameters variables. wP and features u, form a collider at u in layer 4+1 This brings to the following assumption which is the basic block of our proposed method.
Assumption A de-confounded (robust) explanation of black-box DNN can be defined by the measure of changes between the effect distributions P(y]do(w)) ), X) obtained by interventions on : and the observed distribution (outcome) P(y for any E {I.....N} and E {1,... 1141).
It easy to check the validity of Assumption from the causal graph shown in Fig. Tc). First, the graph allows us isolating the parameters of interest and reason on the effect of every parameter on model output. Second, model parameters are not confounded by the skip connections existing in complex models, like ResNet. This is not the case when considering the features space where some variables may have common effects on multiple variables. We show such case in Fig, Tc) where u1 is confounder (u2 d1 y). Confounders require adjustment criterion (Pearl (2009)) to block any superiors correlations that give wrong effect estimates. This becomes more difficult in complex DNNs because of many confounders (Chattopadhyay et al. 2019))
2.3 FINDING CAUSAL MECHANISMS POST-HOC DNN
We are interested in finding the mechanisms (causal pathways) along which changes flow from causal variables to the effects. These mechanisms will form explanations of the DNN's behaviour for every data input (local explanations) and across inputs (global explanations).
For simplicity, let us first consider the example in Fig (c), where we have single parameter w in each layer. The causal model shows that the effect of a causal parameter w/" or filter for convolution layers, on model output (y) is mediated by the effect of changes on variables (called mediators) along the paths from layers 1,+1,..., IN-1 after intervening on we. These changes will also transmit to the output variable in Ix. Generally, each layer has multiple neurons and we may apply interventions on a set of selected parameters (k I) to analyze their combined effect on model's output.
'The parameters update their values in the training phase using the training data inputs, which creates causal path between data inputs and model parameters.
3

---

Under review as a conference paper at ICLR 2022
The interventions do(wf") imply changing the values of w.. In convolution layers, each w is tiny set pixels (e.g., 9 in case x 3 filter). Changing one single value would not carry out significant changes on the output. Therefore,
consider changing all the values of the filter. In fully connected layers, a parameter is a scalar value. The effect of changing one single parameter in the penultimate layer of the classifier would be highly significant to the output.
To find the causal mechanisms, we start by quantifying the direct effect on the prediction variable yj (for class j). which are in this case the parameters -- We intervene every single parameter and analyze its effect. Finding the direct effects enables identifying the causal paths in the network. However, for intermediate layers we cannot do interventions computationally very expensive and not practical in inference time. Moreover, deep networks are highly parametrized, which would lead to high variance in cause-effect relations thus making difficult to capture robust explanations. We propose robust selection criterion to select the variables that we want to intervene on. This leads to the following proposition.
every single parameter. This is
Proposition 1 (Intervention variables in intermediate layers) For w. the set of indices K (L...., 2 of the mediators in intermediate layer 1,+1 and I E {l...., v.., subset of indices cor- responding to the causal parameters w Then there exist causal paths between u1, and uk,++1 and I will identify the intervention variables in 1i-1
We proof Proposition T on simple example in Appendix A. As we notice, we measure changes in retrospective way, so the selection criterion is defined for layer 4-1 after finding the causal param- eters in layer 4, To find the indices / for layer 4-1, we focus on changes which are statistically significant using a threshold 6, which is defined as
6, (ug ( p2)/og
(1)
where M4 is the original prediction (reference) of the true class j. and jég is the average value of changed prediction and og the standard deviation. To capture the informativeness in causal pa- rameters, we introduce in Proposition 2 the formula for the causal effect.
Proposition 2 (Causal effect) Given an input X and let W.K be the set of causal variables (or paths) to the K neurons in 444, the causal effect (CS,) of do(W?) is a measure of the information flow from W.K. K to which is defined as:
log W, = an X)
CS,(n) Ey
logp(y((ze)
where plylX) is the Softmax of the prediction logits of an input X, (0, 1), and on, are the change values for each single parameter.
We put details of Proposition in Appendix A. The interventions that we used in this work are an = 0. Propositions [ and uncover the causal mechanism of hidden layer i in the DNN's architecture. Algorithm describes how we apply them to capture the causal mechanisms in all layers.
2.4 IMPLEMENTATION DETAILS
The implementation of our Algorithm requires an architecture (Gx) of the DNN, that can be simple MLP or CNN, a pre-trained knowledge (W), an input X and its prediction y. The output of the algorithm is two dictionaries G and G which we use to explain the behaviour of the model given X and y. The algorithm starts in a retrospective way from the penultimate layer (I = N - 1) and the the output (prediction
target class K j. First, it computes the direct effect of the parameters layer Ix). It searches two sets indices: 14 indicating the causal parameters, {wf] ). which have positive effect, and I_ that refers the causal parameters, {wf)) (n * k Vk, n), that have negative effect on the class of interest yj. then updates G_,, G_ respectively. Next, we use I, to update K
and select the intervention variables (w for the next iteration (where becomes 1). We repeat the steps 3 to 10 until end up with the input layer lo
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Causal mechanisms of post-hoc DNN
Input: K.y.J.M. N, W
Output: G4, G_
1: Initialize 1 N 1, GG= =G= {1}
2: repeat
3: 1 0,,
4: for each w E w/ do
5: w 0 and compute y
6:
Add the index of w to 4 if ÿ < 6: else: Add the index of w to -- if ÿ y > 61 Compute causal effects (S((n) in layer
7:
8:
update G+ and G_ by cause-effect variables corresponding to 1+ and I_ end for
9:
10:
< 1 -11-1 1, K + T+.
11:
until lo
3 EXPERIMENTS
We implement interventional black box explanations on deep neural networks trained for image classification tasks. We use two convolution neural network architectures: LeNet (Leeun et al. (1988). simple architecture composed of two convolution layers followed by two fully connected layers, and R (He et al. (2016)). We capture post-hoc explanations for the model LeNet using MNIST data with 10 digits; and for ResNet18 using MNIST and a subset of ImageNet data (Deng al. (2009)). For explainability, we selected the digits and from MNIST data, and some arbitrary classes from ImageNet.
3.1 CAUSAL EXPLANATIONS
Per-sample causal mechanisms. We implemented Algorithm [ on single inputs to extract local explanations model's prediction. For this experiment, we chose an example where the model pro- vided correct predictions and compute the causal effects CS of interventions from each layer. Fig. 2 shows the causal effect computed over all layers (convolution and fully connected). This figure shows the differences between the causal mechanisms that the model use reason on each different class. As we can see, the causal effects uncover different and common causal parameters between classes. To illustrate the behaviour V3 and Us and compare it to the prediction obtained using the original parameters of the model. For these examples, the original logit values are 5/3 = 8.19 and y/a 11.29 and the probabilities ob- tained by the Softmax function are 93% and 97% respectively. Fig. 2 demonstrates the presence causal mechanisms that have either positive or negative effects on model prediction. Moreover, there are many parameters that don't have any effect model behaviour, i.e. removing or keeping these parameters didn't ' change the output. As we mentioned in Section 2.3] removing these parameters is equivalent to removing the effect of the mediators (features) in the same layer.
model's 's prediction, we show Fig. 3 the effect distributions
evidence
Cross-sampler causal mechanisms. We applied our algorithm on group of samples (100 positive example) capture explanations about global behaviour of the model. Fig. shows results on MNIST digits predicted by LeNet. It illustrates the frequency of causal parameters in convolution and fully connected layers in terms of number of samples (colour and size of blobs are relative to the frequency). We also illustrate the causal filters which appeared in above 80% of the samples. These filters explain the global patterns in each digit. The causal filters the first convolution layer are identical across the samples of two classes. They are edge detectors and capture global shape of the object. Note that their effect is not identical as we show later in Fig. 6 The causal filters in the second convolution layer are more interesting. They explain what makes prediction to be class or as their responses describe meaningful concepts. In fully connected layers, our method highlights different levels of importance of causal parameters. Those which persist for more than 80% of samples are the most important and explain a global behaviour of the model. The causal parameters which have lower frequency explain local behaviour in small group of samples.
5

---

Under review as a conference paper at ICLR 2022
(6) convt layer
H comd byer
151 961 layer
(6) 902 layer
Figure 2: Causal effect of interventions on LeNet parameters over each layer for 3 and 8 digits using one test input
En) convt bayer
boyer
layer
Figure 3: Distribution of effects of intervention variables over LeNet layers.
3.2 USE EXPLANATIONS TO CORRECT PREDICTIONS
In previous experiments, we provided explanations for positive predictions. In this experiment, we focus on negative samples to evaluate if our method can capture meaningful explanations correct the predictions of wrong classes. We selected samples with digits 3 on which LeNet predicted the wrong digit 8. The model made wrong predictions for 11 samples from the set. We implemented our algorithm on all negative samples belonging to this subset and extracted cause-effect relations corresponding to the true class. Fig 5 shows the causal parameters for the last convolution layer and the fully connected layers. Comparing the results to the group of 100 positive samples shown Fig. 4 for digit 3, the algorithm consistently captures similar explanations corresponding to strongest causal parameters. Other causal parameters mostly related to local differences specific to each sample. This demonstrates that the extracted causes can used to correct the model. To do so, we turned off (put to zero) all the non-causal parameters and the ones in G_ and only kept the causal parameters G4. Those which are expected to have positive effect on the model. Then, we tested the model again the data. As show in Fig the model is 100% certain of its prediction with significantly high logit value compared to the original predictions where the average probability was 15% for the true class. We show in Appendix additional results on ResNet18.
3.3 COMPARISON TO EXPLANATION METHODS
Besides that our method captures the causal parameters and filters that affect DNN's prediction on each class, can visualize the features in the mediators located in the paths of the causal graphs. These mediators are simply the neurons or channels (in case of convolution layers) that mediate the effect of causal parameters on model predictions. Based on Assumption ??, we can find the mediators from the index set of the causal weights. For instance, the mediator in the last convolution layer of ResNet18, layer4,1.cone I4. Remaining channels are omitted. We obtain the
is a 2D feature vector dimension
512
with the effective channels where attribution map, by computing the average at each pixel. This allows us comparing our method with other explanation methods, where the goal is explore if the causal mechanisms lead to extract better explanations. We compared the visual explanations generated by our method with the ones generated by other explanation methods such Saliency (Simonyan al. 2014)). Occlus (Zeiler

---

Under review as a conference paper at ICLR 2022
8
3
(bl corm layer
JX conee layer
No: layer
for bayer
Figure 4: Global explanations and comparison between digits 3 and 8 using 100 samples. (a) and (b) show the frequency of causal filters causal filters in conv1 and conv2 layers, respectively. (c) and (d) show the frequency of causal parameters in fully connected layers. The x-axis indicate the indices of causal parameters.
Figure 5: Causal parameters corresponding to the true class for LeNet when tested negative sam- ples. The figure on the right shows the original predictions of the model (before) and the corrected one
& Fergus (2014)) IG (Sundararajan al. (017)) and DeepLift (??). We also wanted include the popular LRP (Binder et al. (2016)) method, however, this method doesn't work on complex architectures such ResNet because the method doesn't handle the skip connections. Figure shows results on MNIST digits using LeNet model. Comparing the explanations generated for digits and 8, we can see that other methods don't 't provide informative explanations recognize the digit 3 from 8. Our method shows the effects of the causal filters which explain the difference between digits through their shapes. In Fig- 7 we show causal explanations ResNet18 model on arbitrary classes of ImageNet. Our method captures visual explanations from all layers. The causal filters capture different meaningful concepts (semantics) at multiple levels of hierarchy. We show here results from some layers and illustrate further details in Appendix C.
RELATED WORK
The work explaining deep neural networks keep rising by the machine learning community be- cause of the difficulties of computing robust explanations from these black box models. Many methods have been proposed by the community. In this paper, focus on explanation methods of post-hoc models which are related to our research problem.
Local and Global Explanations. The main forms of explanations split into these two categories Velez & Kim (2017); Das & Rad (2020)). Global explanations aim at explaining and sum- marizing the behaviour the model, for instance the mechanism underlying a classification model, over group of instances (data points). Local explanations focus local data point under scrutiny. They capture explanations from a single instance and search the parts of input data that are most relevant for model's prediction. Rule-based methods and decision trees have been used to cap- ture global explanations approximating the DNN models (Letham et al. (2015); [Caven & Shavlik (995)). These techniques are also used to capture local explanations by approximating the de-

---

Under review as a conference paper at ICLR 2022
Sallency methods
Effects of causal fitters conv layers
Input
Oocks
Searryy
Cocka
ma0
cnn
Figure 6: Visual attributions from different methods. Comparison visual explanations between and 8 digits. Our method generates visual explanations by integrating the response of the mediators corresponding to the causal filters in each conv layer.
Delie
Meric
Inyeul zevo
Lcaecs
canv
lnve
Rozzlin
Uypett Apattarawat
Butteriy
Myaenia sard
candl
Orles
10)
03
Figure 7: Visualizations and qualitative comparisons. (a) Examples of ImageNet classes. (b) Visual effects (explanations) of causal parameters of some hidden layers of ResNet18. (c) Saliency maps obtained by attribution-based methods.
cision boundary around local data input with linear classifiers (Ribeiro al. (2016); Montavon let al. (2017)). Saliency maps (Simonyan et al. [Binder et al. (2016); /Selaraju et al. (2017)); Shrikumar al. (2017)) are attribution-based methods that describe the relevance of local feature input (e.g., image pixels) on model's 's output. The explanations of input-output relation for a post-hoc model are generated by of two different techniques. A backproagation, which rely on class gra- dients (Selvaraju et al. (2077)); Sundararajan et al. (2017)). and perturbations technique, which use random counterfactual instances (Burns al. (2020) or partial substitution local features (Zeiler & Fergus (2014)) to capture the relevance of such variations prediction. Although, these meth- ods are significant contributions in DNN interpretability, the explanations can be summerized by finding correlation between input features concepts and model output. We believe that robust explanations for complex DNN models require the language of causal reasoning and cause-cffect relations.
Causality for Black Box Explanations. Structural causal models (SCMs) Pearl 2009); Peters et al. (2017)) have been recently proposed for extracting local explanations from post-hoc models. These methods focus on local input-output relations by extracting causal explanations from inter- ventions applied on feature space of input data (Chattopadhyay et al. (019); Schwab & Karlen), or
8

---

Under review as a conference paper at ICLR 2022
on the latent space of generative models (Harradon et al. (2018); Goyal et al. (2019): O' Shaughnessy jet al. (2020)). Our work is different as we focus on the model itself as a mechanism to capture global and local explanations from the pieces of knowledge encoded in its parameter space during the phase of training.A prior work (Narendra et al. (2018) has used features importance as a way to extract most and least important filters in CNN. They used linear regression to transform the vector-valued features to real-valued ones on which interventions where applied to estimate the effect of variance of filters response. Compared to them, our method is fundamentally different.
not limited to
CNN and does not rely on features transformations to extract causal filters as this is highly sensitive to noise and confounding errors. We focus on interventions on the parameter space, and provide fundamental framework on searching global and local mechanisms to explain the behaviour of post-hoc DNN models.
CONCLUSION AND DISCUSSIONS
Causality has become an important player in the deep learning field and more particularly, in aspect related to explainability, robustness fairness and bias (Kusner et al. (2017): Kilbertus et al. (2017); Zhang al. (2021)) In this work, focused DNN explanations. We introduced novel method to capture explanations for post-hoc DNN models which quantify cause-effect relations between model prediction and its parameters. We started our method by constructing the causal diagram that generally describes a post-hoe model. The proposed causal graph is generic as it can be used to cap- ture the effects of interventions on features space, or on parameters space. We then proposed new algorithm (interventional black box explanations) to capture the effect of parameters on output both locally and globally. Our method is model-agnostic because it does not require specific architecture. We demonstrated that it works on simple architectures, such as LeNet, and complex one including skip connections between structured layers (blocks). Although, in this work we focused on classifi- cation architectures, our method can be easily generalized to other architectures like object detection and NLP models. This will be one of the next steps of our future work.
Recently, work on DNN explanations considered logic forms of explanations Muu & Andreas (020 Logic explanations search relations between concept (semantic) neurons in the DNN to generate natural language expressions for model interpretability. These methods limit their explana- tions to the last hidden layers of the network because of their expensive computational cost. Other limitations that might affect the robustness of explanations in these methods is the presence of some low quality neurons in the network. Our framework can be used as a basic block for these methods. We can approach to first capture the causal filters and the corresponding responses (features), then generate logic explanations that expand to all hidden layers in the network.
REFERENCES
Nihat Ay and Daniel Polani. Information flows in causal networks. Advances in Complex Systems, 11001):1741, 2008
Alexander Binder, Sebastian Bach, Gregoire Montavon, Klaus-Robert Müller, and Wajeiech Samek. Layer-Wise Relevance Propagation for Deep Neural Network Architectures. In Kuinam J. Kim and Nikolai Joukov (eds.), Information Science and Applications (ICISA) 2016, pp. 913-922. Springer Singapore, 2016.
Collin Burns, Jesse Thomason, and Wesley Tansey. Interpreting Black Box Models via Hypothesis Testing. In Proceedings of the 2020 ACM-IMS on Foundations of Data Science Conference, pp- 47-57. Association for Computing Machinery, 2020.
Aditya Chattopadhyay, Piyushi Manupriya. Anirban Sarkar, and Vineeth N Balasubramaniaa. Neu- ral network attributions: A causal perspective. In Proceedings of the 36th International Confer- ence on Machine Learning, volume 97, pp. 981-990, 2019.
Mark W. Craven and Jude W. Shavlik. Extracting tree-structured representations of trained networks. In Proceedings of the 8th International Conference on Neural Information Processing Systems, pp. 24-30. MIT Press, 1995.
Arun Das and Paul Rad. Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey. arXiv:2006. 1371 [cs], 2020.

---

Under review as a conference paper at ICLR 2022
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Finale Doshi Doshi-Velez and Been Kim. Towards A Rigorous Science of Interpretable Machine Learning. 2017. arXiv: 1702.08608.
Yash Goyal, Amir Feder, U. Shalit, and Been Kim. Explaining Classifiers with Causal Concept Effect (CaCE). ArXiv, 2019.
Sander Greenland and Judea Pearl. Causal Diagrams. In International Encyclopedia of Statistical Science, pp. 208 216. Springer, 2011.
Michael Harradon, Jeff Druce, and Brian Ruttenberg. Causal Learning and Explanation of Deep [cs, stat), 2018.
Neural Networks via Autoencoded Activations.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Niki Kilbertus, Mateo Rojas-Carullae Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Schölkopf. Avoiding discrimination through causal reasoning. In Proceedings of the 31st International Conference Neural Information Processing Systems, pp. 656-666. Curran Associates Inc., 2017.
Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In Pro- ceedings the s1 International Conference Neural Information Processing Systems, pp- 4069-4079, Curran Associates Inc., 2017.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-basse learning applied to document recog- nition. Proceedings of the IEEE, 86(1 1):2278-2324, 1998.
Benjamin Letham, Cynthia Rudin, Tyler H. McCormick, and David Madigan. Interpretable clas- sifiers using rules and bayesian analysis: Building a better stroke prediction model. CoRR, abs/1511.01644,, 2015.
Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Robert Müller. Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recognition, 65:211-222. 2017.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. In Advances in Neural In- formation Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Tanmayee Narendra, Anush Sankaran, Deepak Vijaykeerthy, and Senthil Mani. Explaining Deep Learning Models using Causal Inference. arXiv: 1.04376 [cs, stat], 2018.
Matthew R. O' Shaughnessy, Gregory Canal, Marissa Connor, M. Davenport, and C. Rozell. Gener- ative causal explanations of black-box classifiers. NeurIPS, 2020.
Judea Pearl. Causality Cambridge University Press, 2009.
Judea Pearl and Dana Mackenzie. The Book of Why: The New Science of Cause and Effect. Basic Books, Inc., 2018
Jonas Peters, Dominik Janzing, and Bernhard Schlkopf. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should Trust You?": Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Con- ference Knowledge Discovery and Data Mining, pp. 1135-1144, Association for Computing Machinery, 2016.
Patrick Schwab and W. Karlen. CXPlain: Causal Explanations for Model Interpretation under Un- certainty. In Advances in Neural Information Processing Systems (NeurIPS), pp. 10220-10230.
10

---

Under review as a conference paper at ICLR 2022
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based lo- calization. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618-626, 2017a.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 618-626. IEEE Computer Society, 2017b.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Ma- chine Learning- Volume 70, pp. 3145-3153, JMLR.org, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Yoshua Bengio and Yann Le- Cun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 3319- 3328. JMLR.org. 2017.
Matthew D. Zeiler and R. Fergus. Visualizing and Understanding Convolutional Networks. In ECCV, 2014.
Cheng Zhang, Kun Zhang, and Yingzhen Li. A Causal View Robustness of Neural Networks. arXiv:2005.01095 [cs, stat], 2021.
A APPENDIX
A.1 DETAILS FOR PROPOSITION II
To proof Proposition I we show in Fig. Sa toy example of an MLP network with a hidden layer containing 2 units with variables u and u2, one output simple linear function) y and scalar input x. We start by the direct effect (wi, wh) - Since we have only one class, then K = 1. We consider that do(w;) has effect on y and changed the prediction. Then, a causal path exists between 21 and y. Since the intervention values = O, this leads to
y u1 wi
u2 wh,
u2
W2
(w) 0)
(3)
We also have
y w x w/ + // w
2
ÿ w/ w
(4)
From equation (4). and the fact that we know i| has a significant effect e, we then got
e y ÿ
wi x w/
(5)
The important information flows from input x to y through the mediator u1, and wi is the selected intervention variable for the effects of the parameters lo on y. It is easy to see this on input of two variables (x1, x2). In this case the parameters W1] (wg) w)") are the selected interventions variables.
A.2 DETAILS FOR PROPOSITION
In causal graphs (DAG) (Ay & Polani 2008)), the measure of information flow is defined by the Kullback-Leibler divergence which is related to mutual information, and used to quantify the causal effects between disjoint subsets of nodes. Our case is similar as we search to compare the prediction probability of the post-hoc DNN XX) with the probability resulted form interventions on W, P(y (d) X). The output is obtained by feedforward the effects of intervention variables to the output. To capture the amount of information related to the action do(K) in the hidden layer 4, we used Kullback-L Leibler divergence which provides the result in 2
11

---

Under review as a conference paper at ICLR 2022
U1
w11
1
X
y
w2o
U2
W12
Figure 8: toy example. MLP with hidden layer, input and linear output
B APPENDIX
Our method applied to ResNet18 trained on MNIST dataset. The model achieved almost perfect performance all digits of the MNIST test set. The average performance was 99%, it was rare to find negative samples on which the model wrongly predicted the class. We found only one negative sample corresponding digit 8, and negative samples for digit Fig. | illustrates explanations computed from two selected convolution layers. We show human interpretable way what are the causes for wrong prediction and what treatment can fix it.
p-39%
Label
O.covv
layarl1.comvl
kyyer 2.c.mol
7-78%
po-70%
explanations causal parameters (Negative offects)
(3) Visual explanations causal parameters G+ (Positive offects)
(d) Model predictions beforo and after treetment
Figure 9: Using causal explanations to correct wrong predictions of ResNet18 for MNIST exmaple.
C APPENDIX
More visualization details from all hidden convolution layers of ResNet18 on ImageNet classes mentioned in Section B.3
12

---

Under review as a conference paper at ICLR 2022
syon2,1.com2
layer4 1ynnv convt
layer4 conv2
layer4.0.com ..comv
ayer3.1.com2
1yee3.1.comv
layerS O conv2
layer 5.0 conv1
layer2.1.com
layer2. nyer2.1.cow1
layer2. 0.comv2
layer2 O convl
layer1.1 conv2
loyer2.1.covv
aycor_0.comv
layer1.0.com
conv2
Input
Figure 10: Visualization of causal effects in all convolution layers for the class object Dalmatien.
ccon
layerd
layerd.0.com
layer3.0.comv2
layera 6.conv1
bayer2.1.com/
1ayer2.1.comv
byer220.comv2 0.com2
Oyer2.0.comvi
layer2.1.com
layer1.1.com
layer1. Oyer1.0.convz
layer1.0.com
convi
Input
Figure 11: Visualization causal effects in convolution layers for the class object Merle.
13

---

Under review as a conference paper at ICLR 2022
ayeer.1.conv2
layer4.1.covv
layará, O comv2
ayer4.0.convl
laye331.comv
byera.1.comvl
layer3.0.con2
layer3.0.com/
laye221.comv2
layer2.1.cavv
byyor2.0.omv2
0.covv
1yyer11.comv2
byyer11.comvi
byerl.0.comv2
Ayyer..0.onv1
conv1
input
Figure 12: Visualization of causal effects in all convolution layers for the class object Roslein.
bycovv1.
LcyorA.1.covv
layer4,
layer4.0 2.com.0.oomv
b.cov
layer3.0.com
byer3.0.comv
layer2.1.com22
layer2
0.co2.0.oonu2
layer2 a 0.conw3
layer1. L.conu
layer1.1.com Lcoon
layer1.0.comv 0.conw2
Figure 13: Visualization of causal effects in all convolution layers for the class object Butterfly.
14


Reviews:
[{"review_id": "eVU8ipiXKb1", "reviewer": "Reviewer_Acmr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation for the paper is clear and well written and the idea is good. There is also relatively little work on explanations from model manipulation, so the direction of the paper is promising. The paper is well structured and the setup is described clearly.\n\n**I do have a few concrete questions and remarks regarding Clarity:**\n\n**In algorithm 1: **\nThe index of weight $w_l^k$ is added to $I_+$ if $w_l^k$ has a positive effect on the correct class, that is when when $\\tilde{y}>y$, then if $\\delta_l$ is positive shouldn't it say \"Add the index of $w$ to $I_+$ if $\\tilde{y} − y > \\delta_l$\"?\n\n**Figure 2:**\nThe text in the legend and on the axis is way to small!\nAre these averages over several samples of 3 and 8 or just one?\n\nHere it would be interesting to \n1. get quantitative graphs\n2. compare digits that are similar (3 and 8) AND very different (1 and 5)\n\nThe authors state \"Fig. 2 demonstrates an evidence on the presence of causal mechanisms that have either positive or negative effects on model prediction.\"\nHow does the figure show both negative and positive effects?\n\n**Figure 3:**\nThe text in the legend and on the axis is way to small!\nthere is no x-label.\nI haven't completely understood what Figure 3 shows and how it relates to figure 1, it would be good to explain this a bit more.\n\n**Figure 4:**\nThe text in the legend and on the axis is way to small!\nHere a scatter plot with 8 on one and 3 on the other axis makes more sense as the colour and shape of the circles already represent frequency.\n\n**Cross-samples causal mechanisms:**\nAre the 100 positive samples taken from the trainings set or the test set?\n\n**3.2 use explanations to correct predictions:**\nSo here the model was pruned and then tested again?\nHere it is very important what data was used for pruning. The authors need to explain the process more rigorously.\nAlso, what does it mean to say \"the model is 100% certain of its prediction\"? What is the test accuracy of the pruned model?\n\n**3.3 Comparison to explanation methods**\nThe authors state \"We obtain the attribution map, by computing the average at each pixel.\" It would be advantageous to explain in a bit more detail how they get from a collection of indices of relevant weights to an attribution map.\nI think, LRP does exist for ResNet, see \"INNvestigate Neural Networks\" (Alber et al.).\nIt is not obvious to me how the authors conclude \"Our method shows the effects of the causal filters which explain the difference between digits through their shapes.\"\n\n**A.1**\nThe proposed MLP seems to have no activation functions and is therefor just a one layer linear mapping which can be reduced to y=x*w. It is unclear what the authors intend to show with this.\nThe notation is not clear. What are $w_0^{11}$ and $w_0^{21}$?\n\n**B and C**\nHow can we interpret the attribution maps?\n\n\n**I also have a few concerns regarding quality of the paper:**\n\n- The attribution maps shown are not very convincing and it's unclear in which ways they provide better insights than existing explanation methods.\n\n- There is no quantitative evaluation of explanations for example with ROAR (2019, Hooker et al.).\n\n- Figure legends and axis labels are too tiny to be read.\n\n- The paper \"Interpreting CNN Knowledge Via An Explanatory Graph\" by Zhang et al. seems very closely related but is not discussed in the current paper.\n\n- The authors do hint on time constraints when applying the do- operation to all parameters of a model but do not give further insight into how many parameters need to be tested on average and how long this taked for bigger models such as ResNet", "summary_of_the_paper": "The authors aim to give post hoc explanations of Neural Network classifier decisions. They do so by finding causal relationships between model parameters and classifier outputs. To this end model parameters are set to zero and the change in the models prediction is calculated. If the change in prediction exceeds a threshold the parameter is deemed relevant and used for construction of an attribution map.", "summary_of_the_review": "There is relatively little research on Explanation by manipulating weights. So the direction of the paper is interesting and relevant.\nUnfortunately the paper is lacking clarity in many aspects, for example how attribution maps are extracted or the description of experiments, especially the \"model treatment\".\nIt is unclear how the attribution maps give better insight in model behaviour than other explanation methods.\nThere is no quantitative evaluation of results.\n\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KbjmYVLDoFk", "reviewer": "Reviewer_NpLg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, I found the paper hard to read, with notation and terminology rather confusing. Below are a few examples:\n- The definition of w_i^k could be a lot clearer, for example, explaining that the intervention involves setting multiple *edges* that lead to a neuron to zero, and perhaps drawing a connection to dropout where neuron weights are zeroed out. \n- I think the terminology “A de-confounded (robust) explanation of a black-box”. As far as I can tell, this is the only use of the term de-confounded in the paper; it seems unnecessary. What is being proposed here also seems less of an assumption and more like something you are defining in this paper. \n- The \\tilde{y} notation is a little confusing. On page 4 it is mentioned that mu_{\\tilde{y}} is the average value of the changed prediction, and \\tilde{y} next appears in Algorithm 1 where it is used in a difference term: \\tilde{y} - y. \n- The terminology “post-hoc DNN”” is unusual. For example, “We are interested in explaining post-hoc models, i.e., models after training”, “We build a causal graph of a post-hoc DNN”, “In post-hoc DNNs, each path communicates the information”. IMO, explanations can be called post-hoc explanations but the DNN model should not be called post-hoc.\n\nI think it should be clarified that the interventions are not independent (i.e. different interventions w_i^k could be operating on some common set of edges), and that the earlier layers are ancestors of the later layers. I am not sure how to reconcile this with Algorithm 1 running from the penultimate layer to the initial layers.\n\nI have a few questions about the experimental results:\n- In Figure 7, with the exception of Occlus, Saliency, IG, Deep_Lift all look very blue. Could this be due to a scaling issue? Otherwise, these results suggest that IG and DeepLift methods found nothing at all important in these images, which is seems unusual. \n- Figures 2-5 are hard to read — x and y axes labels could be larger — and I found it hard to interpret Figure 2 and 3. \n\nCitations:\n- (Page 7) Citation for DeepLift is missing\n- (Page 7) Letham 2015 is cited in the context of global approximations approximating DNNs but this paper proposes a ground-up interpretable model, bayesian rule lists, and doesn’t have DNNs\n\nTypos:\n- (Page 3) “The graph allows us isolating…” —> The graph allows us to isolate\n- (Page 6) “Based on Assumption ??”\n- (Page 8) “A backproagation”\n- (Page 9) “and more particularly, in aspect related to” —> in aspects related to", "summary_of_the_paper": "The paper observes that many DNNs are DAGs and proposes to run interventions on parts of the network then compute the effect of the interventions, which the paper calls causal explanations of the DNNs. \n", "summary_of_the_review": "The idea is interesting but the terminology and experiment results are confusing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F3wS7-1tb7U", "reviewer": "Reviewer_TDuY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n1. The problem raised by the paper is well-motivated.\n2. The visualization of the layer-wise weights is quite intuitive, helps readers to better understand the behaviors of each layer.\n\nWeakness:\nOverall, the paper lacks technical rigorous and verbal accuracy which makes the contribution delivered by the authors less convincing.\n1. It appears to me that the paper used intervention and do-calculus in an interchangeable manner without clearly defining what 'intervention' means. Under such circumstances, I can only conclude that such language utility is not precise, please check for instance (Peters et al. 2017) for a more precise defintion.  \n\n2. In  Assumption 1, the authors claim '... by the measure of changes ... ', what does 'measure of changes' mean? This definition sounds confusing to me.\n\n3. In Proposition 2, the authors name CS_i as the causal effect.  Be aware that there has been a well-established terminology so-called average causal effect used in the causal inference community (See Hernán MA, Robins JM (2020)). The authors should make a clear discussion about how the causal effect defined in the paper related to one that is well-known to the community.\n\n4.  The paper states 'Structural causal models (SCMs) (Pearl (2009); Peters et al. (2017)) have been recently proposed for extracting local explanations from post-hoc models.' Could you be more specific? Correct me if I am wrong, as far as I know, the SCMs proposed in the two books were not meant for trained DNN models. They might be very good candidates for the author's tasks, but that is unclear based on the contents of the two books and remains to be investigated.   \n\n5. LeNet and ResNet 18 are less representative compared to the rapid development of the DNN model architectures.\n", "summary_of_the_paper": "This paper proposed a causal-driven method aiming to resolve the black-box issue of DNNs. The proposed interventional black-box explanations method tends to be model-agnostic and can apply a variety of DNN models. In the experiments, the authors examined the proposed method on two well-known DNN architectures LeNet and ResNet18. ", "summary_of_the_review": "In weighing the pros and cons of the paper, I believe that the current version is not ready for formal publication. \nAs well-motivated as the paper can be, a significant revision needs to be done before the paper can bring useful insights to the community.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AdXVEqRNkZ", "reviewer": "Reviewer_GBET", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find the overall idea of the paper, i.e. connecting explanations with causal diagrams, quite interesting and promising. This is because it opens an approach to study the underlying mechanisms of the decision making process of the neural network in a principled manner. \nI also find the application to use these causal explanations to correct predictions quite intriguing (although I do not find the technical design of the experiments convincing - as discussed below).\n\nUnfortunately, my overall impression is that the manuscript would still require a significant amount of work in order to reach a state in which it could be considered for acceptance. I will try to summarize this in detail below, but before doing so I do want to encourage the authors to invest more time into this work as it is underlying idea is quite interesting.\n\n- I found Section 2.1 a bit chaotic. For example, it contains the sentence \"The subscribe number indicates that this is the input layer l_0\". It is unclear what this sentence refers to. Another example is that c_i is being defined as the convolution units. This is not standard terminology and one has to guess from the context what is meant. E_{i,i+1} is defined as \"the edge vector connecting the nodes in layer l_i to l_{i+1}\" but it is again not clearly defined/stated what is meant by this. \n\n- All Figures in the manuscript are very hard to read as they use extremely small font sizes. \n\n- Section 2.2. states that parameters and features form a \"collider\" but never defines this term. If this is standard terminology in the causal diagram literature then one should add a background section on causal diagrams which contains all relevant definitions. This is particularly important as the main contribution of this work is to introduce these novel techniques to the field of explainability.\n\n- Proposition 1 is only proven for a simple example as far as I can see? This is obviously not enough. If the stated existence of causal paths is assumed axiomatically, this should be made clear. Similarly, Proposition 2 is a definition not a proposition as far as I can tell.\n\n- As intervention, only \\alpha_n=0 is used. One may expect that this choice is of the utmost importance for the results. It should thus be motivated and/or compared with other choices.\n\n- Algorithm 1 should clearly state what the input symbols mean, i.e. X input data, y predictions, and so on. \n\n- Per-sample experiments: as far as I understand these experiments were only performed for two selected data samples. If so, it needs to be discussed if they demonstrate generic behavior. It would be preferable to design experiments that are not sensitive to the choice of data sample, for example by reporting averaged values over many data samples.\n\n- Figure 3 does not have a label for the x-axis. Is this the index number? Both Figure 2 and 3 have legends and labels which are significantly too small. It should also be clearly explained what is meant by \"conv1 layer\". Is this the first convolutional layer (if counted from the input layer upwards)?\n\n- \"use explanations for correcting predictions\". As far as I understand, the 100% certainty of the model is measured with respect to the samples from which the type of correction is inferred? What happens with the other samples if the same parameters are set to zero?\n\n- Section 3.3 contains broken citation references for DeepLift and Assumption 1. \n\n- Section 3.3 incorrectly states that LRP does not work on ResNet, see e.g. 2003.07631. There are several open source implementations for LRP on various architectures including ResNets, see e.g. https://github.com/chr5tphr/zennit.\n\n\n\n", "summary_of_the_paper": "The manuscript proposes an explanation method for feedforward neural networks which is based on causal diagrams. \n\nIt evaluates the proposed method using two architectures (LeNet and Resnet18) and two datasets (MNIST and a subset of ImageNet) and performs a comparison to existing explanation methods. ", "summary_of_the_review": "Overall, I think that the underlying idea of the manuscript is promising but that the draft is immature and requires a significant rewrite to more clearly explain the underlying notation, background, and theoretical assumptions. Furthermore, the experiments need to be extended and more clearly explained. Please refer to the list of feedback above for details. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are in consensus. I recommend that the authors take their recommendations into consideration in revising their manuscript.
Decision: Reject