Causal Explanations of Structural Causal Models
Authors: Matej Zečević
Devendra Singh Dhami
Constantin A. Rothkopf
Kristian Kersting
Venue: Submitted to ICLR 2023 (2023)
Keywords: explanatory interactive learning
explainable artificial intelligence
causal explanations
structural causal models
user study

Abstract: In explanatory interactive learning (XIL) the user queries the learner, then the learner explains its answer to the user and finally the loop repeats. XIL is attractive for two reasons, (1) the learner becomes better and (2) the user's trust increases. For both reasons to hold, the learner's explanations must be useful to the user and the user must be allowed to ask useful questions. Ideally, both questions and explanations should be grounded in a causal model since they avoid spurious fallacies. Ultimately, we seem to seek a causal variant of XIL. The question part on the user's end we believe to be solved since the user's mental model can provide the causal model. But how would the learner provide causal explanations? In this work we show that existing explanation methods are not guaranteed to be causal even when provided with a Structural Causal Model (SCM). Specifically, we use the popular, proclaimed causal explanation method CXPlain to illustrate how the generated explanations leave open the question of truly causal explanations. Thus as a step towards causal XIL, we propose a solution to the lack of causal explanations. We solve this problem by deriving from first principles an explanation method that makes full use of a given SCM, which we refer to as SC$\textbf{E}$ ($\textbf{E}$ standing for explanation). Since SCEs make use of structural information, any causal graph learner can now provide human-readable explanations. We conduct several experiments including a user study with 22 participants to investigate the virtue of SCE as causal explanations of SCMs.

Under review as a conference paper at ICLR 2023
CAUSAL EXPLANATIONS OF STRUCTURAL CAUSAL MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
In explanatory interactive learning (XIL) the user queries the learner, then the learner explains its answer to the user and finally the loop repeats. XIL is attrac- tive for two reasons, (1) the learner becomes better and (2) the user's trust in- creases. For both reasons to hold, the learner's explanations must be useful to the user and the user must be allowed to ask useful questions. Ideally, both questions and explanations should be grounded in a causal model since they avoid spurious fallacies. Ultimatcly, we seem to seek a causal variant of XIL. The question part the user's end we believe to be solved since the user's mental model can pro- vide the causal model. But how would the learner provide causal explanations? In this work show that existing explanation methods not guaranteed be causal even when provided with a Structural Causal Model (SCM). Specifically, we use the popular, proclaimed causal explanation method CXPlain to illustrate how the generated explanations leave open the question of truly causal explana- tions. Thus as a step towards causal XIL, we propose a solution to the lack of causal explanations. We solve this problem by deriving from first principles an explanation method that makes full use of a given SCM, which we refer to as SCE (E standing for explanation). Since SCEs make use of structural information, any causal graph learner can now provide numan-readale several experiments including user study with 22 participants to investigate the virtue of SCE as causal explanations of SCMs.
explanations. We conduct
INTRODUCTION
There has been an exponential rise in the use of machine learning, especially deep learning in several real-world applications such as medical image analysis (Ker et al., 2017), particle physics (Bourilkov, 2019), drug discovery (Chen al., 2018) and cybersecurity (Xin et al., 2018) to name few. While there have been several arguments that claim deep models are interpretable, the practical reality is much to the contrary. The very reason for the extraordinary discriminatory power of deep models (namely, their depth) is also the reason for their lack of interpretability. To al- leviate this shortcoming, interpretable and explainable AI/ML (Chen et al., 2019; Molnar, 2020) has gained traction to explain algorithm predictions and thereby increase the trust in the deployed mod- els. However, providing explanations to increase user trust is only part of the problem. Ultimately, explanations or interpretations (however one defines these otherwise ill-posed terms) are means for humans to understand something-in in this case the deployed AI model. Therefore, a closed feedback loop between user and model necessary for both boosting trust through understanding/- transparency and improving models robustness by exposing and correcting their shortcomings. The new paradigm of XIL (Teso & Kersting, 2019) offers exactly the described where a model can be "right wrong for the right or wrong reasons" and depending on the specific scenario the user- model interaction will adapt (e-g. giving the right answer and correction when being "wrong for the wrong reasons").
Now the question arises, what would constitute a good explanation inline with human reasoning? In their seminal book, Pearl & Mackenzie (2018) argue that causal reasoning is the most important fac- tor for machines to achieve true human-level intelligence and ultimately constitutes the way humans reason. Several works in cognitive science are indeed in support of Pearl's 's counterfactual theory of causation as a great tool to capture important aspects of human reasoning (Gerstenberg et al., 2015; 2017) and thereby also how humans provide explanations (Lagnado et al., 2013). The authors in

---

Under review as a conference paper at ICLR 2023
Hofman et al. (2021) even argue that systems that are efficient in both causality and explanations are need of the hour. Questions of the form "What if?" and "Why?" have been shown to be used by children to learn and explore their external environment (Gopnik, 2012; Buchsbaum et al., 2012) and are essential for human survival (Byrne, 2016). These humane forms of causal inferences are part of the human mental model which can be defined as the illustration of one's thought process regarding the understanding of world dynamics (see also discussions in Simon (1961); Nersessian (1992); Chakraborti et al. (2017)). All these views make understanding and reasoning about causal- ity an inherently important problem and suggest that what we truly seek is a causal variant of XIL in which the explanations of the model are grounded in causal model and the user is also allowed to give feedback about causal facts.
While acknowledging the difficulty of the problem we address it pragmatically by leveraging qual- itative (partial) knowledge on the SCM (Pearl, 2009) justifying the naming of our truly causal ex- planations (by construction) that we will be referring to as Structural Causal Explanation (SCE). The motivation behind this work is the observation that spurious associations in the training data inevitably leads to the failure of non-causal models and explanation methods. For example, an im- age classifier that learns watermarked images will have high accuracy on the test data from the same distribution but it will be "right for the wrong reasons" and furthermore the "right" part is brittle it will fail when moving -uitt--distribution (Lapuschkin et al., 2019). In psychology this phenomenon of spurious association fallacy is known as "Clever Hans" behavior named after the 20th century Orlov Trotter horse Hans that was wrongly believed to be able to perform arithmetic (Pfungst, 1911). Some works such as (Stammer et al., 2021) moved beyond basic methods (like heat maps for image data) using a XIL setup to move beyond "Clever Hans" fallacies. Some other works purely on the explanation part made an effort in devising a "causal" explanation algorithm to avoid spuriousness (Schwab & Karlen, 2019), however, as we show in this work they still leave open the question of truly causal explanations-a gap that we fit. We provide a new, natural language expressible (thus, human understandable) explanation algorithm with SCE.
Overall, we make several contributions: (I) we devise from first principles a new algorithm (SCH) for computing explanations from SCM making them truly causal explanations by construction, (II) we showcase how SCE fixes several of the shortcomings of previous explainers, (III) we apply the SCE algorithm to several popular causal inference methods, (IV) we discuss using a synthetic toy data set how one could use SCE for improving model learning, and finally (V) we perform survey
with 22 participants to investigate the difference between user and algorithmic SCE.
We make code repository publically available at: https//aanoyymouss4ppenn.ccceee Manctural-Causal -aw.saa-Explaaatiionon Explanations DOR7/
BACKGROUND AND RELATED WORK
We briefly review key concepts from previous and related work to establish a high-level understand- ing of the basies needed for the discussion in this paper.
Causality. Following the Pearlian notion of Causality (Pearl, 2009), an SCM is defined as a 4- tuple M := (U, V F(F,)(()) where the so-called structural equations (which are deterministic functions) U < [pa, u4) E F assign values (denoted by lowercase letters) to the respective endogenous/system variables Vi V based on the values of their parents Pa, C \ V; and the values of some exogenous variables U, C U (sometimes also referred to as unmodelled or nature terms), and P(U) denotes the probability function defined over U. The SCM formalism comes with several interesting properties. They induce a causal graph G, they induce an observationallaasooia tional distribution over V (typical question "What is?". example "What does the symptoms tell us distributions
about the disease?"), and they can generate infinitely many interventiona//yyotteeie (typical question "What if?", example "What if take an aspirin, will my headache be cured?") and counteffactal//errspectii distributions (typical question "Why?", example "Was it the aspirin that cured my headache?") by using the do-operator which "overwrites" structural equations. Note that, opposed to the Markovian SCM discussed in for instance (Peters et al., 2017), the definition of M is thus allowing for shared U between the different V4. Such a shared U is also called hidden confounder since it is common cause of at least two V2, Vū # j). Opposite to that, "common" confounder would be a common cause from within V. In the case of linear SCM, where the structural equations fi are linear in their arguments, we call the coefficients "dependency terms"
2

---

Under review as a conference paper at ICLR 2023
and they fully capture the causal effect of V2 onto V, denoted as 0z++j, that is, how changing V2 will change V,, The depedency terms of non-linear SCM are generally more involved, so azz+j cannot be obtained by simply reading off coefficients in f.. In most settings the causal effect, e.g. of medical treatment onto the patient's recovery, is the sought quantity of interest. If interventions are admissi- ble, then the average causal (or treatment) effect (ACE/ATE) within binary system is defined as difference in interventional distributions, that is ae == BVV, do(V; = 1)] LEVV, do = 0)]. A great deal of research in causality (especially for ML) concerned with leveraging observational data to reason about causal relationships (also known as identification), which often times overlaps with the broader study of identifying the graphical structure underlying the data distribution. For example, Ke et al. (2019) made use of data from the first two levels of Pearl's Causal Hierarchy (PCH), namely observational and interventional, to update their graph estimate while using masked neural networks to mimic the structural equations. Most of the time in causality we seek a Directed Acyclic Graph (DAG) that is consistent with our available data and assumptions, on that end Zheng et al. (2018) proposed continuous approach to learning DAGs efficiently. A followup work by Brouillard et al. (2020) then combined ideas from both previous works to provide an efficient causal DAG learner.
Explainable and Interpretable AI/ML. A great body of work within deep learning has provided visual means for explanations of how neural model came up with its decision i.e., importance es- timates for a model's 's prediction are being mapped back to the original input space e.g. raw pixels in the arguably standard use-case of computer vision (Selvaraju al., 2017: Schulz et al., 2020). Formally defined in (Sundararajan et al.), we simply have that Ax = (a1...., an) E R" is an attribution of predictive model I' when ag the contribution of æ; for prediction F(x) (with = (I1,.... F.,)) E IR"). Recently, Stammer et al. (2021) argued that such explanations are in- sufficient for any task that requires symbolic-level knowledge while comparing the existing state of explanations to "children that are only able to point fingers but lack articulation". (Stammer et al., 2021) 1) therefore proposed neuro-symbolic explanation scheme to revise and ultimately circumvent "Clever Hans" like behavior from learned models in XIL user-model loop (Teso & Kersting, 2019). On the causal end, (Schwab & Karlen, 2019) proposed a model-agnostic approach that can generate explanations following the idea of Granger causality (which is very different from Pearlian causality as it captures "temporal relatedness" which holds in their setting as input precedes output). Specif- ically, they train surrogate model to capture model to be explained. They achieve this by simply comparing the prediction loss of the model for the original input C(v, úxx (where X is the input) with the alternate prediction loss when a certain feature i is being removed [ly. (x)(g). On the Pearlian side of explanations, the arguably closest works on explainable AI/ML can be found in research around fairness (Kusner et al., 2017: Plecko & Bareinboim, 2022). For instance, Karimi et al. (2020) investigated how to best find a counter- factual that flips a decision of interest e.g. an applicant for a credit rejected and the question now which counterfactual setting (changes to the applicant) would have resulted in a credit approval. From a purely causal viewpoint, our work might be compared to the definitions of Halpern (2016) for "actual causation."
what degree certain inputs cause outputs in the
DERIVING A CAUSAL EXPLANATION FOR THE CAUSAL HANS' EXAMPLE
Causal Hans Example. Consider the setting from Fig.1 which highlights the Causal XIL loop that we envision. The users, for instance a medical doctor (he) and a ML developer (she). have access to some data sets, for instance a set of patients's 's medical records. The ML developer will train a model that tries its best in learning about the data distribution's 's underlying causal relations. The M.D. on the other hand might make an observation such as for the patient named Hans that his mobility is below average, thus raising the question "Why is Hans's 's Mobility bad?". Given causal explanation method, we could now derive an explanation that is grounded in the learned causal model to answer the question about the patient Hans in a causal manner thus the naming Causal Hans example. Not satisfied with the answer, the M.D. decides to tell the ML developer about the inconsistencies he ob- served between the generated model explanation and his structural intuition. Subsequently, the ML developer adapts the training procedure with the corrections, ultimately resulting in a new learned structure whose generated explanation satisfies the M.D. and moves the learned model closer the latent true SCM. This completes the Causal XIL setup as we envision, however, what is the causal
As analogue and wordplay the "Clever Hans" notion, see Introduction.
3

---

Under review as a conference paper at ICLR 2023
feedback user, loop cleces user saggests connection/ increseees, performance
after onfo))
Structural Intuition
User(s) n.g. M.D. Develsper
Question
Explanation using SCE Algorithm
e-g. patient
observes and asks question
posuldtion evarage 29.6 "Whn is Hand's Hoaility ty bad?'
Mobility is
spita his high Age (A)
Data Patsent Records
Learned Structure
mostly because of his
seek
FILE TOUR 2/R
learning algoritim
explan
bad mostly due his good P. Hobits (F).
kieully contains
Latent True SCM
"Hans" Mubbiiiy (M) is bad bocause of his bac Health which mostly due to his high Age (A), withough his Food Habits (F)
sosumptions
A ZalFa
tweeks leaming
explain
generates
M- (a19).7)))
through XIL iteraionn convering closer to true SCH
good."
Figure 1: Causal XIL. illustrated the Causal Hans Example. Refer to the first paragraph "Causal Hans Example" of Section 3 for a detailed description. We propose the SCE algorithm that allows for truly causal explanations in Causal XII.. (Best viewed in color.)
explanation method? the following we will derive from first principles new explanation scheme to answer an observation/qucstion (using our running example about a patient named Hans) that is consistent with given SCM thereby constituting causal explanation of a SCM by construction.
3.1 DERIVING A COMPUTABLE ALGORITHM FOR CAUSAL EXPLANATIONS
We will use the Causal Hans example that we just introduced as our workbench to derive a com- putable algorithm for truly causal explanations. For the sake of simplicity we assume that the struc- tural intuition that the user/expert can capture about the problem using their mental model can be described as an SCMP. In the following, consider an SCM that generates patients's 's medical records described by numerical representations for age, nutrition, overall health and mobility respectively
(V = {A, F, MY). Next, let's consider some samples from said SCM. E.g. we might observe the data set containing the individual named Hans H (azr. fn, hn m,) = (93.8, 58.8, 2.6, 26.2) where each value could be associated with a discrete label e.g. as = 93.8 would be 93 years old, whereas m 26.2 refers to rather immobile person. The latter label is actually implicitly the sessment m < μ where PM = 35.6 is the average mobility value for the population, that is, we observe Hans to be a rather immobile person when compared (or relative) to the whole population.
With this we are in the position to pose question like
"Why is Hans's 's Mobility bad?"
where the word "bad" refers to "bad relative to the population." " Formally, we can now define such an observation or question as:
Definition 1 (Why Question). Let x. E XXl(X) be an instance of X E V of SCM M. Further, let yxx be the empirical mean for a set of samples (μx* == E. x4)) and let R E {<,>} be a binary ordering relation. We call Qx == R(x,H1²) a (single) why question if x true.
Checking back with the definition, we see that Q1 defines a valid question for the Causal Hans example since Qa := m < μ M 26.2 < 35.6 evaluates to true.
Next, we will discuss the structural intuition of the user (e.g. the M.D. in Fig. Generally, the true SCM M is latent but we can realistically expect to have access to partial knowledge (or estimate)
of M Say, the user has intuition for SCM M that contains the relations A 2 F, A 2 H.
F 3 H. H 4 M where a, 3, 7, 8 denotes the respective causal effects² Further, a, a,7, > 0
"While several works in cognitive science support the fact that humans are capable of judging causal rela- tionships both qualitatively and quanitatively it is not generally true that the human mental model is in fact an SCM but rather that parts of can he represented by it.
'Let's assume that M is linear for the moment. The key property needed here is just some kind of value estimate for the causal effect, which for linear SCM is simply the coefficients, that is OF 7 etc. Thus w.l.o.g. we use it here make the example more clear.
4

---

Under review as a conference paper at ICLR 2023
while 3 < 0 meaning that for instance aging (higher value for A) will have a negative, decreasing effect on health (smaller value for H), and also 3 >> y meaning the causal effect of aging on health is greater in absolute terms than the one of food habits onto health. Now when we intend on answering Q1 it seems reasonable to start with the queried variable first, mobility in this case. We observe that M is effect of H with ? > 0 meaning that since Hans has also below average health (and not just below average mobility) and lower health translates to lower mobility that m is inline with fir. Traversing the chain further to the causes of H, which are A, F in M we observe two different scenarios. Since A is above average as Hans is an elderly person and / < O we can conclude that a definitely an explanation for hy whereas F with Y > is actually a countering factor since Hans has good diet beneficial to health. In summary, by exploiting the knowledge on M we have arrived at a causal explanation that can be pronounced in natural language as:
Explanation 1 (for Q1). "Hans's 's Mobility is bad because of his bad Health which is mostly due to his high Age although his Food Habits are good."
Explanation is a truly causal answer to the observation about Hans's s mobility deficiency based on SCM M. It captures both the existence and the "strength" " of a causal relation. In the following we will capture and formalize our intuition that allowed to derive Exp.1. This will allows us to move towards computing such causal explanations automatically.
We mainly used four ideas or pieces of knowledge in our argument above: (I) that there is relative notion in the why question Qs like "why bad?" that implicitly compares an individual (here, Hans) the remaining population, (II) note that by definition there can only exist causal effect from some variable to another if and only if one is the argument of the other in a structural equation of M. (III) the causal effect ax-y allows us to assert whether the observed values for (I. y) are "surprising" or not (e.g. it was not surprising that my < uu" after observing hµ > 0 and knowing that y > 0 since decreasing health means decreasing mobility in general and Hans old), and (IV) that some causal effects are more important or influental than others (e.g. age versus food habits w.r.t. health). We can neatly collect all information from (I-III) in a single tuple which we call causal scenario.
Definition 2. The tuple (xy:= 2, 2 (() is called causal scenario.
The (IV) point we capture separately as will be shown below. Now, we finally express our build up intuition and understanding into rules expressed in first-order logic that will then allow us to compute causal explanations like Exp.1 automatically.
Definition 3 (Explanation Rules). Let Cxy denote a causal scenario, let s(x) E (-1,1) be the sign of a scalar, let R2 E {<,>) } be a binary ordering relation and let 2x= = {1az-x1 ZE Pax) be the set of absolute parental causal effects onto X. We define FOL-based rule functions as
(ER1) If Roz then:
R1 (((s(ax-)):0) (R2 y R (x,,,x)))
(ER2) If RR2, then:
(((s(ax+r):0) (V, R2 (²),, and
(ER3) 2x > 1, then
Y = arg argmaxze Z
indicating for each rule ERi(-) E (-1,0,1) how the causal relation X X satisfies that rule.
One can easily verify that these rules form the building blocks for the generation of causal explana- tion like in E.p.1 We can now give a simple recursive algorithm that traverses all possible directed paths (or causal chains) to the queried variable checking each of rules ERi thus constructing unique code that maps to unique answer. We define the Structural Causal Explanation algorithm as:
Definition (SCE). Like before let Qx,M M be a valid why-question and some proxy SCM. Further, let D x [V] denote our data set. We define recursion
B(Qx,M, = =1 ER(Z + X), 4ZErs(X) E(Qz, M))
(1)
where Oi Us = (U1,..., vn) denotes concatenation and ER checks each rule ERi (Def.3), and the recursion's 's base case is being evaluated at the roots of the causal path to X, that is. for some ZEV
with path Z + + X we have
E(Q2,M, D) 0.
(2)
We call E(Qx. M, D) Structural Causal Explanation of M.
5

---

Under review as a conference paper at ICLR 2023
Causal Hans Example revisited (using SCE algorithm). To return one last time, we clearly see that for Qu (corresponding to Q1) we can compute using Eq.1
B(QM,M, = ((ERI = -),, + ZE(A,F) Ei D)) = (..., (((ER( = 1, ER3 = MI), E(P, =...., (...,,,,,,))) (..., ()).
So the recursion result is H + M (ER1 - -1, 0, ER3 0), -
(ER1 = ER2
0, ER3 1), F - H (ER1 = 0, ER2 = 1, ER3 = =)). This result uniquely identifies the human understandable pronunciation of our causal explanation in Exp.1. We provide a detailed explanation on the pronunciation scheme and also intuitive namings for the rules ER; in the appendix. It is worthwhile noting that the natural language choice of words to express the interpretation is not implied by the form of the SCE e.g., while Hans's 's mobility is said to be "bad", car's remaining fuel is rather considered to be "low".
3.2 THEORETICAL PROPERTIES OF ER AND SCE
The concepts of why-question, causal scenarios and ERi rulest hat we had develop for the in- troduction of SCE algorithm, alongside SCE itself, come with several mathematical consequences which we now discuss. All of the subsequent results are simple and can be proven easily, still, their importance needs to be stressed since they make implications about the wide applicability of SCE.
Proposition 1. For any causal scenario the rules ER1 and ER2 will be mutually exclusive.
Proof. First, we code the binary ordering relations <,> to represent 0 and 1 respectively. Second, we observe that ERi E {<,>}. (R(s(ax-r); 0), R(y. )),, R(x, Third, let T := {0, 1}3 be the set of all such triples as their code words, so T E T. Looking at the total number of possible scenarios [T| = 2 = 8, we easily see that ER1 covers codewords (010, 011, 100, 101, 000, 111} and ER2 covers the codewords (00T, 110), and together they cover all codewords ER1 U ER2 T. Since any single scenario Cxy is uniquely mapped to codeword, it will either trigger ER1 or ER2 but never both.
{1,2} always involves the triplet
Proposition 2. The SCE recursion always terminates.
Proof. The recursion's 's base case is reached when root node is reached i.e., node i with Pa, 0 An SCM implies a finite DAG, so root nodes are reached eventually.
Theorem 1. The output of any causal structure learning algorithm can be used to compute SCE.
Proof. The proof for this theorem is surprisingly simple in that the SCM M used in the SCE recur- sion is only required to provide some kind of numerical value 0z-j for the relation of any variable pair (i, that is, a matrix A E RVVIXIVI which represents linear SCM or SCM where each 0z-+ represents a causal effect description. If the matrix A is an adjacency matrix living in [0, 1] then we simply have no information about ER3 since all causal effects are assumed to be the same. Since any causal structure learning algoirthm will produce a causal graph represented by a matrix, we have that we can compute SCE.
The beauty of 1 can be fully appreciated when being put into the context of practical AI/ML research and application. It tells us that any causal graph learner ever invented and that will ever be invented can provide causal explanations thus reflecting the learnt. In practice this means that all prominent graph learning algorithms like NT (Zheng et al., 2018), CGNN (Goudet et al., 2018), DAG-GNN (Yu et al., 2019) and NCM (Ke et al., 2019) are all explainable'. On a concluding note to this section, we have a remark on SCM that allow for hidden confounder. SCE presented Def.4 do not cover hidden confounders and we leave this for future work. However, we can always modify the algorithm to talk about "unknown reasons" when giving knowledge on U. An extended discussion on this and also other noteworthy aspects of SCE can be found in the Appendix.
any query of interest consistent with the learned model
*The DAG learner in NT can be interpreted linear SCM but there no guarantce.

---

Under review as a conference paper at ICLR 2023
Query / Question "Why is Hans's Mobility bad?"
Cousal Explanation with SCE Algorithm
"Causal" Explanation with CXPlain
Gruphical
Age, Food Hablts, Health, Mobllity 93.8, 58.8, 2.6,
which mosty due Io right Ago (A), ethough is Pood Mutila (P) ure good!"
(M]
20
30
Hans
Tolter
10
Mobility Popu beps 35.6
Age
Habits
Health
Figure 2: Comparison of SCE with CXPlain. Refer to the text in Sec.4 for the discussion. Both approaches use the same ground truth SCM, however, only SCE is capable of leveraging necessary causal knowledge. (Best viewed in color.)
4 EMPIRICAL ANALYSIS
4.1 FAILURE OF EXISTING EXPLANATION METHODS
In the beginning we said that existing methods such as CXPlain (Schwab & Karlen, 2019) would fail to provide truly causal explanations useful for Causal XIL but did not argue or show how they would fail. Now after having introduced our solution to the problem in the previous section on SCE, we can compare both and make apparent the shortcomings that we have fixed with SCE, For illustration we make once again use of the Causal Hans example. F shows the results. We provided CXPlain with the same ground truth SCM as SCE to train the surrogate explanation model. We trained 10 bootstraped neural models using suitable parameters for the masking operation and loss function. What we observe is a distribution over importance scores where all factors are being deemed relevant and "causal" to the output (which in this case is the mobility of Hans). Also the highest attribution is given to age, then food habits and finally the lowest to health. This single observation makes apparent two important shortcomings: (I) from the output we do not know which is a direct (H) and which are indirect (A, F) causes while the ordering of presentation in SCE clearly distincts the former from the latter, and (II) we have no information on the causal effect, that is, we cannot tell in which way a variable with high attribution will affect the predicted variable, for example food habits received a high importance score like age but age will have detrimental effect on mobility whereas food habits will have a beneficial effect- again, SCE fixes this by discriminating the positive and negative cases. We also ran CXPlain for more queries and observed two further fallacies that we discuss in the Appendix. One the flip side, like SCE, the CXPlain attribution was able to identify the that the effect of aging is stronger that that of dieting.
4.2 QUALITY OF LEARNED EXPLANATIONS
The overarching question for this experiment was "If we apply a causal graph learning method to our data, then what do the generated explanations reveal about the learning method?" To investigate this question we looked at different data sets including different graph learners and for each combination their respective SCE. We considered several different why-questions for each of the four data sets: data set for the Causal Hans example, weather forecast (W, real world, Mooij et al. (2016)), mileage (M, synthetic), and recovery (R, real world, Charig et al. (1986)). To avoid cluttering in the main text we have moved the relevant tables and figures to the the Appendix where we also provide an extended account. Here we highlight what the SCE when applied to NT (Zheng al., 2020) would reveal about what was learned. We ran NT with suitable paramaters. In Tab.2 of the Appendix we see the computed SCE. We observe expected results on the W and M data sets, whereas differences on the R and H data sets. For R, the difference is only subtle as the model's explanation to the query "Why did Kurt not Recover?" is not "Kurt did not Recover because of his bad Pre-condition, although he got Treatment." but "[...]. which were bad although he got Treatment." which is on the second recursion in the reasoning process i.e., the treatment countering the state of condition and not affecting the condition itself. This difference becomes apparent in the graphical structure where the arrow from Pre-conditions to Treatment is inverted contrary to expectation. To illustrate one more example using the data set to our Causal Hans example where the difference was more drastic, here the discrepancy revolves around a totally different graph structure e.g. the learned model expects direct cause-effect relation between age and mobility while also wrongly assuming that food habits have a detrimental effect on health. Therefore the answer to the question "Why is Hans's 's Mobility bad?" suddenly becomes "Hans's 's Mobility, in spite his high Age, is bad mostly because of
7

---

Under review as a conference paper at ICLR 2023
With Without Median
Without
With
Less False Links
Teeqamay
D
C
1
Figure 3: Graph Learning Improves with Explanations. Left: error distributions when perform- ing graph learning with/-out SCE regularization (which is simply an added penalty term for incon- sistent explanations), next to is the ground truth graph. Right (boxes): the predicted graphs, showing a decreased number of false positives. (Best viewed in color.)
his bad Health which is bad mostly due to his good Food Habits." which sounds very absurd. The ground truth SCM for this data set contains non-linear causal relationships, while NT makes linearity assumptions, which explains the wrongly learned graph structure. The crucial observation, though, which answers our initial question about what SCE can reveal about the learning algorithm is the following: only by looking at the SCE, effectively using it as graph distance or metric, we were able to tell that the learned model is very different from what we expected. Put differently, it made apparent for the Causal Hans example that by simply adding an extra edge (here between A, M, see Tab.2) and flipping the sign of an edge (here between F, H) we already get a big difference in what these graphs express/explain.
4.3 USING EXPLANATIONS AS REGULARIZATION DURING LEARNING
Since SCE contain knowledge on (some) causal relationships underlying the data, we wondered to which extent they might be used for improving the sample efficiency of graph learners. In this ex- periment the overarching question therefore was "If we have explanations available provided, by say an expert, then could we use it for improving learning?" For ease of applicability and consistency with the previous experiment we take NT again and add a simple regularization term to its loss that penalizes inconsistent explanations. We generate 70 random linear SCMs with respective observa- tion distributions. Then we use graph learning to infer 70 more graphs, making 140 graphs in total. For each graph generate 50 random single-why questions to be answered, resulting in a data set of 7000 explanations. All the details regarding this learning setup, such as for instance how to make make SCE differentiable for it to function training signal, are being discussed in the Appendix. The graph learning is being performed in a data scarce setting with only 10 data samples per graph. Thus to infer the true causal structure the method ideally needs to perform sample-efficient learning. Fig.3 shows our results. The error distributions over all of the graphs are shown both with and with- out the SCE regularization. We also highlight the graph estimate upon which most improvement was observed. It can be observed that with the regularization the method can both identify more key structures while significantly reducing the number of false positives. For example many false links that pointed towards node H (like B to H or G to H) were removed while some key structures could now be recovered like the directed edge from node I to node A. While more experiments would be necessary to claim that indeed learning is (significantly) improved through explanations, naive learner already provides evidence in favor that initial hypothesis that sample efficiency is being improved by models that are under pressure to explain what they learned.
4.4 SURVEY WITH 22 PARTICIPANTS
Throughout this paper we provided several arguments in advocacy of Causal XIL as the key paradigm of interest for future research and application. We proposed SCE as a solution to the prob- lem of truly causal explanations, therefore, in this final experiment we investigate the overarching question of "What does SCE explain about the causal intuition that humans have that could pro- vide for Causal XIL?" We let N = 22 human subjects judge the qualitative causal structure of four "daily-life" examples using questionnaire specifically designed to provide with the data neces- sary for constructing causal graphs representative of what the participants think about the presented concepts. Please refer to the Appendix for the questionnaire and [Human Data] for the anonymized
8

---

Under review as a conference paper at ICLR 2023
Graph Distribution for Human Data
Edge Distribution for Human Data
Result
Result
Count
Count
"Mode"
"Greedy"
Graph
2
222
Edge
Figure 4: Measuring Agreement between Subjects. Refer to Sec.4.4 for details. Left, the graph estimate is the mode of the distribution. Right, greedily pick each edge. (Best viewed in color.)
answers that we used for evaluating the survey. Also prolonged discussion is provided in the Ap- pendix. The first question to answer is, how we construct the graph estimates from human data? In Fig.4 we show two ways that considered: "Mode" where we can simply look at the different graphs and take the most frequently occurring one as representative of the population or "Greedy" where we look at the frequency which edges are predicted and then simply construct graph from greedily taking the most probable edge each time. Greedy comes at the cost that the predicted graph not necessarily in the populaton. With the graphs at hand, For brevity, we will only highlight some key observations: (I) we observe a systematic approach and thereby non-random approach edge-/structure-scleeee
analyze our initial question.
by the subjects. Furthermore, there are only few clusters even with increasing hypothesis space. Both the systematic manner and the tendency to common ground are evidence support SCMs and mental models overlapping, (II) we observe that the increase in hypothesis/search space (i.e., more variables) comes with an increase variance. This variance increase can argued be due to the progressive difficulty of inference problems as well as decreased levels of attention and potential fatigue across the duration of the experiment, (III) some subjects implicitly assume notion of time for example that there must cyclic relationship between, say, treatment and recovery where the subject likely thought in terms of 'ncreasiig treatment increases the speed of recovery which subsequently feeds back into a decrease of treatment (since the individual is better off than before)', and (TV) answering initial question of looking at the SCE we can observe that they lie much closer to the expected explanation opposed what have observed in our second experiment when looking existing methods for graph learning (Sec.4.2), suggesting that the human subjects are better at judging causal relationships for the examples we considered.
CONCLUDING REMARKS
Our work made clear how existing explanation methods, when proclaimed causal, leave open the problem of truly causal explanations. We further argued that truly causal explanations are need of the hour since ultimately seek a causal variant XIL it can improve both model perfor- and increase user trust. To this end, we derived from first principles using our Causal Hans example a computable explanation algorithm grounded in SCM. We proved the wide applicability of our SCE algorithm and then went to corroborate our results with several experiments including a na've approach to learning with explanations and survey with 22 partici- pants to analye the human component in XIL. For future work, there several interesting routes to take such as proving that the proposed ERi rules are complete in that there is no other missing rules in the formalism also reasoning about "when you don't 't know" to capture for example uncertainty about the inputs like the current belief about the SCM.
mance
Societal Ethical Implications. While this work discussed for the first time XIL approach with truly causal explanations and therefore is to be considered still in its infancy, deploying a causal XIL. process with SCE could potentially help in many applications improve model performance and increase user trust by making transparent the "understanding" of the model in natural language.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Dimitri Bourilkov. Machine and deep learning applications in particle physics. International Journal of Modern Physics A, 2019.
Philippe Brouillard, Sébastien Lachapelle, Alexandre Lacoste, Simon Lacoste-Julien, and Alexan- dre Drouin. Differentiable causal discovery from interventional data. arXiv preprint arXiv:2007.01754. 2020.
Daphna Buchsbaum, Sophie Bridgers, Deena Skolnick Weisberg, and Alison Gopnik. The power of possibility: Causal learning, counterfactual reasoning, and pretend play. Philosophical Transac- tions of the Royal Society Biological Sciences, 2012.
Ruth MJ Byme. Counterfactual thought. Annual review of psychology, 67:135-157, 2016.
Tathagata Chakraborti, Sarath Sreedharan, Yu Zhang, and Subbarao Kambhampati. Plan explana- tions as model reconciliation: Moving beyond explanation as soliloquy. IJCAI, 2017.
Clive Charig, David R Webb, Stephen Richard Payne, and John E Wickham. Comparison of treatment renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy. Br Med J (Clin Res Ed), 1986.
Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin. This looks like that: deep learning for interpretable image recognition. NeurlPS, 2019.
Hongming Chen, Ola Engkvist, Yinhai Wang, Marcus Olivecrona, and Thomas Blaschke, The rise of deep learning in drug discovery. Drug discovery today, 2018.
Max Chickering, David Heckerman, and Chris Meck. Large-sample learning of bayesian networks is np-hard. JMLR, 2004.
Kahneman Daniel. Thinking, fast and slow, 2017.
Tobias Gerstenberg, Noah D Goodman, David A Lagnado, and Joshua Tenenbaum. How, whether, why: Causal judgments as counterfactual contrasts. CogSci, 2015.
Tobias Gerstenberg, Matthew F Peterson, Noah D Goodman, David A Lagnado, and Joshua B Tenenbaum. Eye-tracking causality. Psychological science, 28(12): 2017.
Alison Gopnik. Scientific thinking in young children: Theoretical advances, empirical research, and
policy implications. Science, 2012.
Olivier Goudet, Diviyan Kalainathan, Philippe Caillou, Isabelle Guyon, David Lopez-Paz, and Michele Sebag. Learning functional causal models with generative neural networks. In Explain- able and interpretable models in computer vision and machine learning, pp. 39-80. Springer, 2018.
Thomas L Griffiths and Joshua B Tenenbaum. Optimal predictions in everyday cognition. Psycho- logical science, 1799)76677773, 2006.
Joseph Y Halpern. Actual causality. MiT Press, 2016.
Masasi Hattori. Probabilistic representation in syllogistic reasoning: theory to integrate mental models and heuristics. Cognition, 557:296-320, 2016.
Jake M Hofman, Duncan J Watts, Susan Athey, Filiz Garip, Thomas L Griffiths, Jon Kleinberg, He- len Margetts, Sendhil Mullainathan, Matthew Salganik, Simine Vazire, Alessandro Vespignani, and Tal Yarkoni. Integrating explanation and prediction in computational social science. Nature, 2021.
Amir- HosseHo Karimi, Julius Von Kügelgen, Bernhard Schölkopf. and Isabel Valera. Algorithmic recourse under imperfect causal knowledge: a probabilistic approach. Advances in Neural Infor- mation Processing Systems, 33:265-2277 2020.
10

---

Under review as a conference paper at ICLR 2023
Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, Stefan Bauer, Hugo Larochelle, Bernhard Schölkopf, Michael C Mozer, Chris Pal, and Yoshua Bengio. Learning neural causal models from unknown interventions. arXiv preprint arXiv:1110.01075, 2019.
Justin Ker, Lipo Wang. Jai Rao, and Tchoyoson Lim. Deep learning applications in medical image analysis. IEEE Access, 2017.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual faimess. Advances in neural information processing systems, 30, 2017.
David A Lagnado, Tobias Gerstenberg, and Ro'i Zultan. Causal responsibility and counterfactuals. Cognitive science, 37(6): 103)):1036-177, 2013.
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 2019.
Christoph Molnar. Interpretable machine learning. Lulu. com, 2020.
Joris M Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf. Dis- tinguishing cause from effect using observational data: methods and benchmarks. The Journal of Machine Learning Research, 17(1): 103-1204, 2016.
Nancy Nersessian. In the theoretician's 's laboratory: Thought experimenting as mental modeling. In PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association, 1992.
Judea Pearl. Causality. Cambridge university press, 2009.
Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018.
Derek Penn and Daniel Povinelli. Causal cognition human and nonhuman animals: com- parative, critical review. Annu. Rev. Psychol., 58:97-118, 2007.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference. The MIT Press, 2017.
Oskar Pfungst. Clever Hans:(the horse of Mr. Von Osten.) contribution to experimental animal and human psychology. Holt, Rinehart and Winston, 1911.
Drago Plecko and Elias Bareinboim. Causal fairness analysis. arXiv preprint arXiv:2207.11385, 2022.
Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. Restricting the flow: Information bottlenecks for attribution. arXiv preprint arXiv:2001.0033 2020.
Patrick Schwab and Walter Karlen. Cxplain: Causal explanations for model interpretation under uncertainty. arXiv preprint arXiv:1910.12332. 2019.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In 2017.
Herbert A Simon. Modeling human mental processes. In Western joint IRE-AIEE-ACM computer conference, 1961.
Edward H Simpson. The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society: Series (Methodological), 13(2):238-22 1951.
Wolfgang Stammer. Patrick Schramowski, and Kristian Kersting, Right for the right concept: Re- vising neuro-symbolic concepts by interacting with their explanations. In CVPR, 2021.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML. PMLR.
11

---

Under review as a conference paper at ICLR 2023
Stefano Teso and Kristian Kersting. Explanatory interactive machine learning. In AIES, 2019.
Kevin Xia, Kai-Zhan Lee, Yoshua Bengio, and Elias Bareinboim. The causal-neural connection:
Expressiveness, learnability, and inference. 2021.
Yang Xin, Lingshuang Kong, Zhi Liu, Yuling Chen, Yanmiao Li, Hongliang Zhu, Mingcheng Gao, Haixia Hou, and Chunhua Wang. Machine learning and deep learning methods for cybersecurity. IEEE access, 2018.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. Dag-gnn: Dag structure learning with graph neural networks. In International Conference on Machine Learning, pp. 7154-7163, PMLR, 2019.
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric Xing. Dags with no tears: Continuous optimization for structure learning. In NeurIPS, 2018.
Xun Zheng, Chen Dan, Bryon Aragam, Pradeep Ravikumar, and Eric Xing. Learning sparse non- parametric dags. In International Conference on Artificial Intelligence and Statistics, pp. 3414- 3425. PMLR, 2020.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX FOR "CAUSAL EXPLANATIONS OF STRUCTURAL CAUSAL MODELS"
We make use of this appendix following the main paper provide extended discussions and details.
A.1 LEARNING DAGS AND CAUSAL GRAPHS
Induction of inter-variable relationships hased on available data lies at the core of most scientific endeavour (Penn & Povinelli, 2007). The sub-class of relation structures of DAGs plays a central role due to its representational role in causality (Pearl, 2009; Peters et al., 2017). Unfortunately, due to the combinatoric nature of the problem setting, learning DAGs from data is recognized to be an NP-hard problem (Chickering et al., 2004). In their seminal work, Zheng et al. (2018) were able to re- ro-formul the traditional view into continuous shape such that any non-convex optimiza- tion can be applied for the graph estimation problem. The authors propose the general formulation, minw 43*x4 ((W) subject ((W) O, where / is data-based score, e.g. in Zheng et al. (2018) a regularized least-squares loss is applied assuming a sparse linear model (possibly SCM) i.e., f(W) = || X WI? [| and is a smooth function with a kernel (or null space) that only contains acyclic graphs, /u( (W) = 0 += W is acyclic. Different variations of the same continuous counting mechanism using this acyclicity constraint have been proposed, e.g., Zheng al. (2020) proposed h (W) tr(e tr(cWow)- ) d while Yu et al. (2019) proposed h(W)= = tr( W W unfortunately, both suffer from cubic -caaabiicallbilitty in the number of graph nodes, O(d)) While the aforementioned works have focussed on data originating from (non-linear transforma- tion) of linear SCM, there exists yet another sub-class of DAG-learning methodologies that focuses on more general causal inference. Ke et al. (2019) made use of data from the first two levels of Pearl's Causal Hierarchy (PCH; Pearl (2009); ?), namely observational and interventional, to update their graph estimate while using masked neural networks to mimic the structural equations. Brouil- lard et al. (2020) follows the same idea of leveraging causal information, e.g. interventional data, for overcoming identifiability issues while staying close to the continuous optimization formalism introduced in (Zheng et al., 2018).
A.2 FAIL OF EXISTING METHIODS, EXAMPLE SHOWCASE WITH CXPLAIN
In the main text we illustrated the key aspects of how (even proclaimed causal) explanation methods like CXPlain (Schwab & Karlen, 2019) fail to do justice for a causal XIL framework. Here we highlight two more interesting shortcomings. Fig.5 shows the same setup as from the main text but for different queries (i.e., patients other than Hans). We make the following observations: (I) the attributions are deterministic. This might first be considered feature, however, the causal mech- anism of an SCM are only deterministic up to realization of the exogeneous terms. Therefore, can have the exact same patient record for different patients. This cannot be captured by these previous attribution methods. (II) when querying for random individuals we actually observe incon- sistencies between the attributions which is weird since the patient records are being generated by the same causal mechanisms. For example in the Hans case we had age, nutrition and then health ordered from highest lowest attribution. For rather similar patient we observe that nutrition and age swap in importance. For yet another patient we observe that suddenly age and nutrition, which previously played the most important role, are not important anymore. The shortcomings (I,II) cor- roborate the shortcomings previously mentioned in the main text, rendering the need for truly causal explanations as presented with SCE more so important.
A.3 HYPOTHESIS: MENTAL MODEL OF HUMANS REPRESENTABLE AS SCM?
has been argued that at the core of a human mental model (abbreviated MM in the following) the illustration of one's thought process (regarding the understanding of world dynamics) is to be found (Simon, 1961; Nersessian, 1992; Chakraborti et al., 2017). The difficulty of said thought process illustration is partly due to circular and abstract terms like explanation and interpretations for which we do not provide an explicit defimition as this is up to philosophical debates and ideally we keep the idea more general than what has been done previously in explainable AI/ML where "explanation equals pixel attributions" in many cases. Assuming the world dynamics to be governed by causality observe that humans capable of modelling both causal relationships between endogenous
13

---

Under review as a conference paper at ICLR 2023
Conancep PHELUE FiAeeiito 1830, 2.24 2A35
Cauatiatt aeau regice imporsace Oe
indicial Cland Feah ousrer inpertecc
Figure 5: Further Examples of CXPlain Shortcomings. Refer Sec.A.2 for details.
variables and additionally information the strength of said relationship. Put differently, MM model a causal graph and corresponding causal effects akin to the formal notions from the previous section. Consider the following real world example:
MM Example. At any given time human has state of overall health (relating to fat- muscle ratio, allergies and diseases, etc.) and mobility (relating to the general freedom and flexibility of movement, e.g., a gymmast is more mobile than the average person). Now, the MM allows inferring (1) that mobility is being (partially) caused by something else (for instance health, e.g., being overweight decreases one's 's mobility) and (2) that dif- ferent events can have different "strength" e.g., that an average car accident causes more harm to the individual's 's mobility than average workout session causes good.
natural candidate for capturing the two properties from the MM example formally are SCM, thereby we hypothesize the following:
Hypothesis 1 (MM Conversion, short MMC). The parts of the MM that are being used for encod- ing the causal relationships of the variables of interest can be formally captured by a corresponding SCM, in short this "equivalence" can be denoted as MM SCM.
While the MMC hypothesis leaves room for notions not captured by mathematical rigor, it suggests an equivalence to SCM regarding the causal aspects. The MM example has motivated the MMC hypothesis which itself suggests justification of using SCM in the first place.
A.4 IMPLICATIONS OF MM = SCM
If we accept that MM SCM, then we can SCMs an adequate proxy to the MM. Furthermore, any useful property of SCM implies corresponding aspects back in the MM. We immediately observe one such key property of SCM namely comparability. That if one is given say two different SCMs that are defined over the same endogenous and exogenous variables (so only differing in the actual parameterizations) then one can compare said SCMs i.e., there exists a notion of distance. For the linear case, we easily prove this by construcing example metric space.
Definition 5. We define function d(MI. = Enys (M, - M2(i,5) + q(P,, P2) where q is the square-root of the Sonsen-Shannon Divergence (JSD). M2 (U,, V., F., P((U.) for E [1 such that Vi=V2, U,=U, FR define linear functions in I, and in slight abuse of notation Mi(ii is the causal effect Oii+ij-
Proposition 3. Let d be in Def.5 and let M. denote the of all linear SCM defined over the same exogenous and endogenous variables, U, V. Then (M, d) is a metric space.
Proof. The absolute difference on the real numbers is metric (i.e., positive-definiteness symmetry, and inteulllytyalttyy hold) therefore holding for the "dependency" terms from F. Furthermore, was chosen as the Jensen-Shammn
Finally, metrics are closed under summation.
Prop.3 is just one example of what might be considered sensible metric space for subset of all SCMs. What it does compare each of the linear coefficients for any causally related tuple of vari- ables, aggregating the sum, and further adding divergence term between the defined distributions
14

---

Under review as a conference paper at ICLR 2023
HMM = SCM
MHans
Mgan
MKuu
(Agge
A=UA
(F)ood Habits
F= A+
d Mi Masel
(Hheahh (M (Moolilty
R
# A++
Kurt
1I Us
M
M Jonas
Maan
M
Hans
Jonias
Figure 6: MMC Hypothesis and Linear SCM Metric Space. Left: Accepting Hyp. means that the MM of Hans is SCM. Right: be compared, an example metric space for (M, d) is given by Prop.3. (Best viewed in color.)
Different linear SCM (from different individuals) can
over the exogenous variables. This comparability and the visual intuition behind MMC are illus- trated in Fig.6, 6. We now state our first key observation following Hyp.1 and Prop.3: the existence of a "true" SCM is in fact justified i.e., there exists an underlying data generating process for any data and the MM of any person might or might not coincide with that SCM.
On another note, consider the fact that while the "true" SCM represents the concept objectiveness, oppositely, the MMs are of subjective nature (that is, every human has their own subjective life expe- rience). Coming back to MM = SCM, we see that Prop.3 further implies that MMs are also capable of dis-/agreeing with each other. With this at hand, now state our second key observation: in most practical cases having access to many SCM-encodings of subjective MMs can ultimately lead in their overlap-agreemenn to (parts of) the objective "true" SCM. There is certainly since all available MM-SCM samples can in fact be wrong, however, note the emphasis on in most practical thees-ther,fore, identifying this overlap in MM (or SCM) for a specifie problem is highly valuable for AI/ML research.
guarantee
Our final, third key observation is concerned with explanations. Existing literature views planations as derivable from MMs and thus implicitly containing some information on the MM (Chakraborti et al., 2017) and since MM SCM, we argue that there must exist an equivalent of the human notion of explanation within SCM. This justifies our further investigation on -Csebased explanations, which eventually leads the formalism of SCE. The benefits of an approach using explanations derived from SCM are two-fold (1) that by construction they are human understandable allowing for explainable MI in which models can reason about the learnt and (2) that the models themselves become better, they need to account for consistency in explanations, which is benefi- cial to any downstream-tez
A.5 ELABORATION ON SCE PROPERTIES
The three basic logic rules (Prop. 1) dictate how the SCE (4) will look like for causal estimate of the system and any given query and data. Having the actual relation R as a return argument of each of the rules allows for a fine-grained explanation. because of X" to a more detailed one like "Y because of X being low". The general pronunciation scheme for the the three rules (excitation, inhibition, and preference), that allows for human- understandable natural language version of the SCE, are as follows: The pronunciation of the details
nutshell, it allows to extend a statement "Y
ER1
Excitation Inhibition
"Y because of X [being low/high]" "Y although X [is low/high]"
ER2
ER3
Preference
"mostly" ER1 or ER2 pronunciation
Table 1: Pronunciation Scheme. Right shows the natural language reading of a rule's activation.
to the relation e.g. "low"/high" is context-dependent that these words might need to replaced with adequate/correspondngg words suitable for the context ie.. "the Matterhorn is cold because of the high altitude" (*cold Temperature because of Altitude being high") is fine while "the remaining fuel is low because of the bad driving style" ("low Fuel because of Driving Style being bad")
15

---

Under review as a conference paper at ICLR 2023
requires the context-adaptation ("low" is converted to "bad"). Another noteworthy detail to the SCE properties is the property of non-repeating causes within explanations which reduces redundancy. Consider for instance our lead example Hans's 's mobility (Exp.1 or Fig. 1), the SCM suggests that F can also he explained by A, since A + F. However, the corresponding SCE does not give this reason because of the aforementioned property which ensures that redundancy is being avoided. Le., in the explanation step before we actually explain H using both A and F, since {A, F} - H, therefore, making irrelevant for the question explain the relation between the parents (A, F). While we provided intuition on the derivation of these basic FOL rules alongside the "Causal Hans" example, we now additionally motivate the namings "excitation", "inhibition" and "preference" i.e., what inspired us to name them in such way. We took inspiration from neuroscience, where the former two terms relate to the way neurons interface with each other using their synaptie-dendric connections. The last term is term to propose "relativity" and thus a preference for one cause over the other. All terms thereby adequately describe any causal path in qualitative terms while also providing an almost synonym-qualits to the pronunciations (Ta
A.6 DETAILS FOR EXPERIMENT 2
We select NT (Zheng et al., 2018) as a representative data-driven graph learner for the illustration in Tab.2 which considers the data sets and why questions illustrated in the appendix Fig.?? i.e., weather forecast (W, Mooij et al. (2016)), health (H), mileage (M), and recovery (R, Charig et al. (1986)). The SCE generated using the learned causal semantics are identical for the DW and M data sets, while differing only subtle for R and drastically for CH data sets. The former discrepancy occurs on the second-level of reasoning i.e., the right top-level explaining answer given to the question (i.e., "Kurt did not recover because of the problematic pre- prreconditions") but was contrasted wrongly (i.e., the treatment countering the state of condition and not affecting the condition). The latter discrepancy revolves around totally different structure e.g. the learned model expects direct cause-effect relation between age and mobility while also wrongly assuming that food habits have a detrimental effect on health. An explanation in the case of NOTEARS is clearly the violation of the linearity assumption for the CH data set generating SCM.
While in 1 we prove that graph learner generally explainable in the sense of SCE, for empirical illustration we also provide more examples of such graph learner-based SCE, as we did with our lead examples for NT, in this case additionally for CGNN (Goudet et al., 2018) and DAG- GNN (Yu et al., 2019). Tables 2 and 4 shows an application to NT with graph visualizations and of all methods to superset of questions (that is, same and more) as the data used for NT. It is crucial to note that the presented results have not been experparameter-opiinzze example CGNN, where candidate selection is exhaustive (brute force, and thus super-exponential in the number of nodes) and the model selection heavily relies on the neural approximation, thereby, HO is likely to be important. In a nutshell, the motivation behind Tab.4 is to present support for our theoretical proof on SCTE-interpeetabiiit of graph learner i.e., we also give empirical proof for several methods in practice (opposed to pure theory). To assess the quality of the SCE, is important to note the assumptions made by the original method. F.g., NT and DAG-GNN assume linear SCM. Thereby, we have no guarantees for running such a method in a non- non-linear data domain (which we do with the data sets DW and CHD). Interestingly, these assumptions can in fact be exposed by SCE. Consider the DW data set (Tab.4, first example), theory suggests that a linear model with Gaussian noise will exist in both directions X - Y and Y + X, thus being non-identifiable (Peters et al., 2017). Methods like NT and DAG-GNN therefore pose the assumption that the given data comes, in this case, from linear model with Gaussian noise i.e., the identifiability problem is being circumvented altogether. This is also the reason why different random seeds can lead to both modellings (A + T and T - A) for the DW data set (see in Tab.4 how the SCE flips for M3=DAG- GNN for the two opposing DW queries). Another important note is that the uninformed SCEs "No causal explanation _..' occur when the method's SCM estimate does not contain causal path to the variable that is being queried by the why question i.e., the SCM will actually contain a non-trivial estimate of the underlying causal structure, even though the SCE returns a trivial/empty explanation since the variable of interest can not be reached within the estimate's 's structure with a directed path (i.e.. the base case in Def.4 is trivially triggered). In fact, these negative "no answer" -type of cases important since the model need also be able to know when there is nothing to be known. For this case, we also pose why questions to which the ground truth is already a "no answer" explanation since there is no causal connection to the variable being queried by the why-question. The empirics
(HO). Take for
16

---

Under review as a conference paper at ICLR 2023
(Left)
(Right) Learned
identical
identical
0
(A) Altitudee (T) Temperature
"Why the Temperature at the Matterhom low?"
(Question) Ground Truch)
"The Temperature the Matterhorn is low because of the high Altitude."
Temperature at the Matterhorn low because of the high Altitude."
(Learned)
2
"Why is Hans's 's Mobility bad?"
"Hans's Mobility bad because of bad Health which is mostly duc his high Age. although Food Habits are good." "Hans's Mobility, in spite his high Age, is had mostly because of his bad Health which is bad mostly due to his good Food Habits."
"Why's your personal car's left Mileage low?" "Your left Milcage is low becmuse of your small Car and your bad Driving Style." "Your left Millage low because of your small Car and your had Driving Style."
"Why did Kurt not Recover?"
"Kurt did not Recover because of his had Pre condition, although he got Treatment."
"Kurt did Recover because his bad Pre-Condition, which were had although got Treatment."
Table 2: Quality of Learned Interpretations. We chose the simple, popular NT from Zheng et al. (2018) as our graph learner for generating the SCE. Subtle differences between explanations exist e.g., the explanation 4 is right on the top-level but for the wrong reasons, that is T + Z instead of - R. Variable letters are capitalized. (Best viewed in color.)
in Tab.4 suggest, as theoretically proven (Thm. 1), that the graph learner are explainable and also that all rules (excitation, inhibition and preference) are being used for the graph learner-based SCE. As a positive example, consider example #3 for the CH data set where M1 captures the complex explanation correctly up to preference and falsely assuming that food habits (F) have a negative causal effect on health (H), A more interesting example (#8 for the R data set) shows that the main reason being bad pre-conditions (Z) is being captured but the model falsely assumes that those are because of the received treatment (T). To consider negative example have look at example #4 again for CH where the actual answer is a "no causal explanation" since age (A) is a root node. However, M3 claims that the age is high because of the food habits and mobility (M), then again because of health. While the statement is wrong and also feels exaggerated, inspecting closely one can detect the correct existence of the causal edge between mobility and health (H + M). L.c., the model interprets wrongly, but its causal model is still partially valid.
A.7 DETAILS FOR EXPERIMENT 3
We again make use of the NT representative of graph learners for the subsequent experiment in which we investigate our conjecture from the final paragraph in Sec.3.2 i.e., whether SCEs them- selves can be used as supervision signal to improve the quality of the learned graph. To circumvent the non-differentiable nature of our recursive formulation of SCE (consider checking the formalism again in Def.4) we train a neural network on a set of legal SCE to mimic the interpreter while being fully differentiable. Following Zheng et al. (2018), we generate 70 random linear SCMs follow- ing Erdos-Reny' structures. We use graph induction to infer 70 more graphs, making 140 in total. For each graph generate 50 random single-why why questions to be answered, resulting in a data set of 7000 explanations. We extend the NT loss composition with this neural approximation using a SCE regularization penalty (to penalize SCE inconsistent graph estimates) and perform graph in- duction once with and once without the regularization (where between and 50 explanations are being observed). The graph induction is being performed in data-scarce setting with only 10 data samples per graph induction. Thus to infer the true causal structure the method ideally needs to perform sample-efficient Main paper Fig.3 shows our empirical results on the error distributions for all the graphs while presenting the qualitative difference in the estimated graphs for the most sig- nificantly improved example. It can be observed that with the regularization the induction method can both identify more key structures while significantly reducing the number of false links, thereby appearing to be overall more sample-efficient. An explanation would be that, as conjectured, the explanations contain valuable information about the underlying SCM if the explanations themselves were generated by a similar SCM, thereby striking structures that would lead to contradicting expla- nations.
17

---

Under review as a conference paper at ICLR 2023
A.8 DETAILS FOR EXPERIMENT 4
We instructed N- = 22 participants answer our questionnaire (see appendix Fig.7). The ques- tionnaire asked the following questions: "Given pair of variables, does causal relationship exist (existence)? If yes, then which is the cause and which the effect (direction)? If there are mul- riple causes for single variable, then how impactful is each of the causes (preference)?" All of these questions, alongside their responses, are of qualitative and subjective nature. It is important to note that the participants do not perform the actual induction from specific, provided data like the algorithms do i.e., the human subjects are not given the variable names nor concrete data points that would allow them to find the rules for the specific data sets. Instead, they were only given the vari- able names/depictions, thereby having to induct from personal experienec//understadnigg essentially. This approach to human induction is related to the experimental setups in (Griffiths & Tenenbaum, 2006; Hattori, 2016).
The motivating lead research questions we intended to answer, and in fact do answer successfully with this experiment, are: What are SCM that (some) human could model? How does overlap for human-based SCM occur? How do subsequent SCE (Def.4) between humans and algorithms differ? In a nutshell, we wanted to investigate the similarity of SCMs between subjects in addition to the similarity between subjects- and algorithm-based SCEs.
A caveat regarding the analysis and explanation of human judgements is that sample bias may distort conclusions. Sample bias has long been identified within the behavioral and social sciences as limiting the generalization of results obtained in a specific sample to the population. A common methodological fix to counteract such biases is to increase the sample size, see (Daniel, 2017) for a recent application and discussion. Certainly, the observed sample will affect the way the difference (to e.g. algorithm-baser SCE) turns out to be, but then again our research question is not concerned with all possible human explanations, but any. Furthermore, we chose data sets that model very general examples and thus offer accessibility to the general population since no single person might be an expert. Ultimately, this way of designing our experiment, while not removing sample bias of course, renders the bias's 's qualitative effect onto our subsequent investigation negligible.
In the following we provide discussion of several interesting and important insights discovered through the human user study. Nonetheless, it is important to note that our results like most modern day interpretations of human behavior of conjectural nature- sensible, educated guesses essen- tially. During this discussion, we will point to specific aspects of the descriptive statisties displayed in appendix Fig.8. The actual human data is also being appended for the sake of completion (click on the following link to access the anonymized human data: https://anonymour www.ooonmoooooooooooooooooo 4open.scienceefi science/r/ Structural- CanallCaaall-aae Caasall-Eplanatttnn popplanations-DDE//Suuuet Haancyy---------- Datn--Dtta-aa Anonymized. pdf). The questionnaire contains four examples with two, three, three, and four variables (or concepts) respectively that are being visually depicted in addition concise textual description. We randomized the textual de- scription of up to three variables across all examples for any randomly selected participant. Doing so, we allow for the randomized concept to reverse causal influence directions, thus, diminishing the bias of chance-selecting said causal direction in a nutshell, this randomization scheme helps us in controlling for explanation variance (or leeway) of the subjects. Nonetheless, we still ob- served that for any variable pair (X, YY the meanings of X and Y themselves could be interpreted differently, which ultimately resulted in False Negatives regarding agreement i.e., people will dis- agree technically although they actually agree. To give concrete example, consider the following: pre-condition in Example 2 can be interpreted as "the length of the medical history of a patient" (negative; increasing implies lower chance of recovery) opposed "the state of well-being of pa- tient" (positive; increasing implies higher chance of recovery), thereby some subjects might choose Z1 + R while others will choose Zz + R where Z, are the different explanations of the "pre- condition" concept (and R denotes recovery). yet all subjects agree on an existing relation between the two variables: Z, <> R. Also, some variahles/concepts were more stable in their explanation variance. To give yet another specific example, altitude and temperature in Example 1 (appendix Fig.7) are stable concepts while the aforementioned pre-condition in Example 2 is unstable (due to its explanation variance/lewway). More importantly these different explanations due to the ambiguity inherent in language become visible within the statistics. To stay inline with the previous example, consider the medical example within appendix Fig.8 (second row, middle) and specifically consider the edges T + R and Z + R. For the former relation the agreement between subjects is evident ie., the majority of human subjects will select this edge. For the latter relation, we clearly the
18

---

Under review as a conference paper at ICLR 2023
two previously discussed explanations that subjects employ during edge decision. Le., for some subjects the edge between Z and R is positive and for some others it is negative, while naturally all agree upon there being relation between the variable pair (Z t+ R) opposed to there being no relation (Z 4/r R).
We observe systematic approach and thereby non-random approach edge-/sttucctreeseleciio by the human operators, see any of the subplots within appendix Fig.8. Furthermore, there are only few clusters even with increasing hypothesis space. Both the systematic manner and the tendency to common ground are evidence in support of the MMC hypothesis (MM = SCM, Hyp.1) and its implied argument on "true" SCM information reachable from the overlapping SCMs.
SCMs or
Although we randomize the order of variables in addition to consistently presenting them in simple line with the intention of not inducing any specific sorting/structure to avoid bias, we still observed apparent, unintended subject behavior. For instance, subject number 5 only considered pairs pre- sented next to each other as being questioned although the other combinations are meant to be queried as well. While additional research needs to corroborate these observations, our data sug- gests that attention might have decreased over the course of the experiment for a subset of subjects as suggested by c.g. subject number 7 where overall agreement with the subject majority is to be found but eventually at the very last example "mistakes" occur (specifically. the subject highlighted that "increasing age increases mobility", in stark disagreement with the majority of participants). We also observe that the increase in hypothesis/search space (i.e., more variables) comes with an increase in variance. This variance increase can be argued to be due to the progressive difficulty of inference problems as well as decreased levels of attention and potential fatigue across the dura- tion of the experiment (e.g. consider the duplicate plots, third column, in appendix Fig.8 where the number of unique structures that are being identified increases significantly). Yet another interesting observation concerns the aspect of time, consider subject number 17 where there is cycle between treatment and recovery where the subject likely thought terms of "increasing treatment increases speed of recovery which subsequently feeds back into a decrease of treatment (since the individual is better off than before)" which seems like a valid inference but clearly considers the arrow of time. Yet another observation, some subjects faced questions of variable scope e.g. if there is causal connection between food habits and mobility, then some subjects considered energy as the mediator and since energy is not part of the variable scope, confusion might arise whether to place edge between food habits and mobility or not. In fact, for such a scenario the correct answer is to place an edge, since there exists a causal path from food habits mobility, via energy, even if energy is not displayed. Lee., in causality, an edge can/will talk implicitly about all the more fine-grained variables that are part of the causal edge/path.
The second data set is an instance of the famous Kidney Stone example (Peters et al., 2017), where Z is a confounder that indicates the pre-conditions in terms of e.g. the size of the kidney stone, and it also illustrates the famous Simpson paradox (Simpson, 1951; Pearl, 2009; Peters et al., 2017) where the recovery will favor one treatment in the overall statistics while being better for all of the non-consolidated views for the other treatment. We observe that not single subject places the edge pre-condition to treatment (Z - T) which arguably at the core of Simpson's 's paradox. This observation gives an additional cue on why the phenomenon is called paradox because no human subject expects the existence of this connection and even actively neglect the existence.
We observe that the human-based SCE match the Ground Truth SCE perfectly up the R data set SCE, which is also the "Result" in Fig.4 i.e., the "Mode" approach returns the correct SCE while the "Greedy" approach chooses the wrong edge type for Z and R. After further investigation, believe to have found several explanations for this "human" " mistake that we discuss extensively in the appendix. On another note, we observe that the overall flawless performance of human-based SCE speaks for superiority over algorithmic graph learner-based SCE, To conclude this paragraph, let us appreciate one such drastic difference in explanations, which in fact occurred on our lead example "Causal Hans":
Humans:
"Hans's 's Mobility is bad because of his bad Health which is mostly due to his high Age, although his Food Habits are good." Machines: "Hans's 's Mobility, in spite his high Age, is bad mostly because of his bad Health which is bad mostly due to his good Food Habits." "
19

---

Under review as a conference paper at ICLR 2023
A.9 OTHER NOTEWORTHY ASPECTS
statement on the role of the MMC hypothesis. An algorithm for SCE-based graph learning- Technical details the experiments. A reference to the remaining appendix figures that make the bulk of the remaining appendix.
The Importance of MM = SCM for SCE. While the MMC is a fundamental question that cuts to the core of human thinking and remains to be proven right or wrong (although we believe it to be true to the extent of representability through SCM), and while we used it to ultimately justify the usage of SCM to then derive the causal explanations we call SCE, still, to the actual existence and formalism of SCE the MMC's truth value is invariant. Put blantly, if the MMC were to be wrong, then the formalism of SCE and all proven properties remain the same. However, if MMC were to be true, then SCE in fact become "stronger" formalism for causal explanations since they' 'd have direct link to the MM. More importantly, one could make the case that they'd d represent a "natural" formal pendant to the vague human explanations.
Simpson's 's Paradox Example. Consider the well-known Simpson's 's paradox example for the med- ical setting of Kidney stone treatments from (Charig et al., 1986). The setting is given by T, K, R which are Treatment, Kidney Stone Size, and Recovery respectively, and further the graph given by T R, K - (T. R}. It is known that T 0 denotes open surgery and T = denotes Percutancous nephrolithotomy (being a more involved procedure) and in the overall statistics for recovery of the patient (denoted by R 1) observe 78% versus 83% respectively, suggesting that T= 1 is the better option. Yet, when looking at the confounder K values of patient recovery, observe 93% versus 87% for small kidney stone K = 0 and 73% versus 69% for a large kidney stone K = respectively, suggesting that in fact 7 = 0 is better instead. This is the "paradoxical" situation, which is sensible from the causal perspective. If we now ask the single why-question for patient i with say values T = 1. R = 0, K = 1 on why i did not recover r; < ft (where μ is the mean recovery of the data set), then we obtain an SCE that reads as follows: "Patient i did not recover because of the large kidney stone, although (sh had Percutaneous nephrolitottomy"
Hidden Confounders in Semi-Markovian Models. As we pointed out in the main text, SCE can naturally handle/extend to semi-Markovian models. For illustration, consider the non-Markovian alternative to the example from the paragraph above on Simpson's is paradox, where K is a hidden confounder i.e., we only observe T + R as the graph. In common settings found in for example (Xia al., 2021), we might at least be aware of the fact that there is hidden confounding present between the two variables and thus have an additional (dashed) bi-directed edge between T and R (case 1) and in the arguably worst case, said variable is fully undetected (case 2, in this case it not necessarily a hidden confounder but simply hidden cause, since we don't 't know if it is confounding or not confoundiig meaning the same thing as common cause). Let's consider both cases, in case 1, the SCE for the same question as before would read as: "Patient i did not recover although (s)he had Percutaneous nephrolithotomy. We note that simply the reasoning on K is not being delivered, naturally, since K is not in the SCM/CEM that the SCE process observes. For case 2, we'd observe the same reading due to the definition of the SCE construction. Here, however, we note that this case allows for natural extension of SCE in which the reading could change to possibly, "Patient i did not recover because of an unknown reason, although (s)he had Percutaneous nephrolithotomy:: Note that this semi-Markovian SCE now allows for reasoning with "unknown reasons" since the hidden cause K will certainly have a causal relation to R (since K is a cause) but the name of K will not be revealed (since K is hidden). With this example, we thus conclude that Markovianity can be leveraged by SCE.
Algorithm for SCE Regularization. For sake of completion. we provide an explicit example algorithm for the simple penalty term we added in our setup for improving graph learning with available SCEs for the respective data set, consider the following:
20

---

Under review as a conference paper at ICLR 2023
Algorithm Graph Learning Using Arbitrary graph learner and SCE Regularization Penalty
Input: Data D, Graph learner Ma, Optimizer O
Output: SCM H, Causal Graph G
1: while & [I] do
2:
H.I <- MM(DD) {perform learning using available )ata}
3:
Qx I' +- I, facquire why question and corresponding SCE)
4:
I E(Qx, H, D) {estimate's SCE of graph estimate]
5:
+ || 2I (compare estimate and ground truth)
6:
0- (1,,0) (parameter update using penalty)
7:
end while
8: H1 MM (D), and G tanh(HH) (return arguments}
9: return H,
Technical Details and Code. All experiments being performed on MacBook Pro (13-inch, 2020, Four Thunderbolt 3 ports) laptop running 2,3 Gllz Quad-Cord Core Intel Core i7 CPU with 16 GB 3733 MHZ LPDDR4X RAM on time scales ranging from few seconds (e.g. evaluating SCE in Exp.2) up to approximately an hour (e.g. SCE- based learning in Exp.3). Our code is available at: https:// anonymous. scpensccec r/ rSStructural- Causal- Explanations D0E7/README.nd.
21

---

Under review as a conference paper at ICLR 2023
3
A-+
T-A
T T-2 Rea 2-42
241 24R
FeD F-85 DeeF 0-55 SisF
5-0
mobiliy
mostly
high age, allbough nutrition good."
H2
-HI
"Hans's mobility, spite high age, montly becouse of
mostly due his good nutrition.
2
111
"Kurt thd not recover because of his had conditions, although he gos treatment.
although pre condition "Kur did not recover becouse of his had pre- cond., which were had although he got treatmentt"
did
fact got
A
Table 3: "Humans Machines". Top: Edge plots per example where the bars denote the average value of given relation and the errors confidence intervals. Bottom: The SCE generated for the two human variants (from main paper 4)g.4) against graph learner representative (NT, Zheng et al. (2018)). Human explanations are (ncar-)identical to the ground truth from Tab.2. (Best viewed in color.)
eea, then is pouline or regaltve?
Tempenture
regative ralatis, caunss
wowssnng Oreddess
negative miation cauess
moce importeet cauioe important
Treatmant
Speed Becovery
Pre-CondRonn
"dopendant on contast might change Improsing
ritatons,
proude any 1ime grohh ersuttu candoioe the the
Pacl hameeiin
with cddiional botrutorn.
The perteppoot we seand pertomm indurtin baned
arange
Ground Truth:
Figure 7: Experiment Setup for the Human Case Study. The participants are being asked two questions: whether there is directed relation between some variable pair A and B, and when there are multiple causes how they behave relatively i.c., the order of strength in relations. We avoid bias in drawing relations by randomizing the order and presenting the variables in a sequence. Induction is being performed from personal "data"/experi" "lata"/experience, rather than by looking at a matrix of data points. (Best viewed in color.)
22

---

Under review as a conference paper at ICLR 2023
Structural Causal Interpretation (Thm.1)
#1
Dataset: DW, Query: "Why is the temperature at the Matterhorn low?"
GT
"The temperature at the Matterhorn is low because of the high altitude."
M1
"The temperature at the Matterhorn is low because of the high altitude."
M₂
"The temperature the Matterhomn low because of the high altitude."
Ma
"No causal explanation for Matterhorn's 's temperature."
#2
Dataset: DW, Query: "Why is the Matterhorn so high?"
GT
"No causal explanation for Matterhorn's 's altitude."
M
"No causal explanation for Matterhorn's 's altitude." "
M₂
"No causal explanation for Matterhorn's 's altitude."
M3
"The altitude of the Matterhorn is high because of the low temperature."
#3
Dataset: CH, Query: "Why is Hans's 's mobility bad?"
GT
"Hans's 's mobility is bad because of his bad health which is mostly due to his high age, although his nutrition is good."
M
"Hans's mobility is bad because of his bad health which bad because of high age and mostly due to his good nutrition."
M2
"Hans's mobility bad because of his good nutrition."
M3
"No causal explanation for Hans's 's bad mobility."
#4
Dataset: CH, Query: "Why is Hans old?"
GT
"No causal explanation for Hans being old."
M1
"No causal explanation for Hans being old."
M 2
"No causal explanation for Hans being old."
Ma
"Hans is old because of his good nutrition and bad mobility, which is because of his bad health."
#5
Dataset: CH. Query: "Why is Hans's ' nutrition good?"
GT
"Hans's nutrition is good because of being older."
MI
"Hans's nutrition is good because of being older."
M2
"No causal explanation for Hans's nutrition."
M3
"Hans's 's nutrition is good because of his bad health and mobility."
#6
Dataset: M. Query: "Why is your personal car's left mileage low?"
GT
"Your left mileage is low because of your small car and your bad driving style."
MI
"Your left milcage is low mostly because of your small car and because of your bad driving style."
M2
"No causal explanation for the left mileage."
M3
"Your left mileage is low because of your small car and your bad driving style."
#7
Dataset: M, Query: "Why is your personal car small?"
GT
"No causal explanation for the car size."
M1
"No causal explanation for the car size."
M2
"Your personal car's size is small because of your good driving style and fuel savings."
M3
"No causal explanation for the car size."
#8
Dataset: R, Query: "Why did Kurt not recover?"
GT
"Kurt did mostly not recover because of his had pre-conditions, although he got treatment."
M1
"Kurt did not recover because of his bad pre-conditions which is because of the treatment he got."
M₂
2
"No causal explanation for Kurt's recovery."
M
"No causal explanation for Kurt's recovery."
#9
Dataset: R, Query: "Why did Kurt get treatment?"'
GT
"Kurt got treatment because of his bad pre-conditions."
M₁
"No causal explanation for Kurt's 's received treatment." "
M2
"Kurt got treatment because of his bad pre-conditions."
Ma
"No causal explanation for Kurt's received treatment."
Table 4: More NaMebased SCE. We prove Thm. for general NIM while pointing to some example methods from the existing literature NIM. Here we show the results of running the methods Mi, I:NT (Zheng et al., 2018), 2:CGNN (Goudet et al., 2018), 3:DAG-GNN (Yu et al., 2019) on the four data sets weather forecast (W, Mooij al. (2016)), health (H), mileage (M), and recovery (R, Charig et al. (1986)) for the respective queries. As suggested, the methods are explainable and reveal insights onto the learned causal semantics, while varying significantly in quality in terms of accuracy relative to the ground truth (GT). Independent of accuracy, "No causal explanation
occur when the SCM estimate of M; contains no causal path to the queried variable X i.c., pax = 0 (supported through GT sparsity). We also show GT explanations that require a negative "no answer" response by M,
23

---

Under review as a conference paper at ICLR 2023
-05
-05
Figure 8: Human Data Analysis: Qualitative, Quantitative, and Uniqueness. Statistics collected from the human data (N 22). The rows denote the four data sets; weather forecast (W, Mooij al. (2016)), health (H), mileage (M), and recovery (R, Charig et al. (1986)). The columns: qualitative edge distributions that show for each of the different edge type how often it was chosen respectively (left), quantitative edge distribution for each edge where the error bars denote confidence intervals (middle), and the unique structure counts where each bar depicts the frequency of a qualitative structure discovered by the human subjects (right). Extensive elaboration on the setup, execution and results of this human study are to found the corresponding appendix section. (Best viewed in color.)
24


Reviews:
[{"review_id": "tPssZPj4ws", "reviewer": "Reviewer_vYZo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors list 5 technical contributions:\n1. the structural causal explanations algorithm - \"a new algorithm (SCE) for computing explanations from SCM making them truly causal explanations by construction\"\n2. an illustration of \"how SCE fixes several of the shortcomings of previous explainers\"\n3. an application of \"SCE algorithm to several popular causal inference methods\" (I assume this is referring to the statement in the abstract, \"Since SCEs make use of structural information, any causal graph learner can now provide human-readable explanations.\")\n4. a discussion \"using a synthetic toy data set [of] how one could use SCE for improving model learning\"\n5. \"a survey with 22 participants to investigate the difference between user and algorithmic SCE\" (I assume this is referring to the statement in the abstract, \"We conduct several experiments including a user study with 22 participants to investigate the virtue of SCE as causal explanations of SCMs.\")\n\nThe authors also make the following claims in their abstract and introduction (and in the main body of the paper), which appear to be distinct from the claims listed in their technical contributions section:\n\n6. SCE is a step forwards in terms of (causal) explanatory interactive learning (XIL). \"In explanatory interactive learning (XIL) the user queries the learner, then the learner explains its answer to the user and finally the loop repeats. [...] Thus as a step towards causal XIL, we propose a solution to the lack of causal explanations.\"\n7. CXPlain does not output \"truly causal\" explanations. \"[...] we propose a solution to the lack of causal explanations. Specifically, we use the popular, proclaimed causal explanation method CXPlain to illustrate how the generated explanations leave open the question of truly causal explanations. [...] as we show in this work [CXPlain] still leave[s] open the question of truly causal explanations\" \n8. The explanations generated by SCE are human-understandable because they can be expressed using natural language. \"We provide a new, natural language expressible (thus, human understandable) explanation algorithm with SCE.\"\n9. \"Throughout this paper we provided several arguments in advocacy of Causal XIL as the key paradigm of interest for future research and application.\"\n10. By using SCE, any causal graph learner can provide human-readable explanations on any query of interest, and this was not possible before. \"Since SCEs make use of structural information, any causal graph learner can now provide human-readable explanations. [...] [Theorem 1] tells us that any causal graph learner ever invented and that will ever be invented can provide causal explanations on any query of interest consistent with the learned model thus reflecting the learnt. [sic]\"", "summary_of_the_review": "While the paper is targeted towards an important field and introduces an interesting general concept of how to construct explanations, the content of the paper does not support its claimed contributions. The paper also needs reorganization to make its contributions clear, as it states stronger and distinct contributions in its introduction/abstract when compared to those that are listed in its contributions section. The paper also needs to clearly state the assumptions and limitations underlying its method, as described in the weaknesses section (my comments from 1.1.-1.4). Therefore, I recommend that the paper be rejected.", "strength_and_weaknesses": "# Strengths\n\nThe paper's topic is very relevant to the field of explainable AI and the particular subfield of explanatory interactive learning (XIL).\n\nThe concept of an algorithm which recursively traces the ancestors of a variable which the algorithm is explaining to construct an explanation, as in Definition 4, is interesting and novel to my knowledge. The concept exists in Hall's notion of production but it hasn't been used to algorithmically construct explanations to my knowledge.\n\n# Weaknesses\n\nOverall, it seems that very strong claims are made in the introduction and abstract, and the claims are significantly weakened when the technical contributions are listed. Because these strong claims are not supported by the paper, they should not be present. I address these claims first, as they seem to be the strongest.\n\n6. **SCE leads to improvements in (causal) XIL.** XIL is mentioned heavily in the abstract/introduction and in the discussion of Figure 1, which illustrates SCE being used in an XIL setup (a setup where users repeatedly query a system for explanations to better understand it). However, it is not shown in the paper that SCE actually leads to a better user experience in XIL. XIL and the setup in Figure 1 is not mentioned again in the body of the paper until section 4.1, where the authors argue CXPLain (the prior method) has shortcomings with respect to SCE, and in section 4.4, where the authors attempt to answer the question \"What does SCE explain about the causal intuition that humans have that could provide for Causal XIL?\". I do not see any mention of XIL in the arguments for SCE over CXPlain in section 4.1 or the analysis of the study in section 4.4. A human study would be useful to demonstrate this is the case.\n7. **CXPlain does not output \"truly causal\" explanations.** Already, \"truly causal\" is a vague term which is not defined in the paper. Section 4.1 argues that SCE is superior to CXPlain because it provides additional information, but it is not shown that CXPlain's output is *wrong* or *not* \"truly causal\". In the example given (Figure 2), it seems reasonable to me to state that Hans' age is the main contributing factor to his (lack of) mobility, the explanation CXPlain seems to give, and the paper doesn't explain why this might be wrong. Thus, the claim that CXPlain does not output \"truly causal\" explanations doesn't seem to be supported in the paper.\n8. **SCE explanations are human-understandable because they can be expressed using natural language.** First, I examine the claim that SCE explanations can be expressed using natural language. It seems that this is true for rules 1 and 2 (excitation and inhibition). However, rule 3 doesn't actually seem to correspond to the phrase \"Y is *mostly* because of [...]\"; it seems that it would be more accurate to say \"[...] contributes the most to Y\", and this may not constitute an explanation. For instance, it seems wrong to say \"The amount of money a charity received was high mostly because of Alfred, who donated 2 dollars\" while 500 other people each donated 1 dollar. So I conclude that SCE explanations can indeed be expressed using natural language, but these expressions may be misleading and may not align with what we intuitively consider an explanation. Further discussion of this point in the paper is warranted. Next, I examine the claim that SCE explanations are human-understandable due to their ability to be expressed using natural language. I could not find any content in the paper supporting this claim (including the human study in section 4.4, which did not show SCE explanations to humans); a human study should be used to support this claim.\n9. **The paper argues in advocacy of Causal XIL.** I am unable to find any such arguments after reading the paper and searching for the term \"XIL\" in the paper in or before section 4.4, where the claim is made.\n10. **Any causal graph learner can now provide human-readable explanations on any query of interest.** First, I examine the claim that this was not possible before (implied by the usage of the word \"now\"). I agree that causal graph learners' purpose is not to provide explanations. Next, I examine the claim that this is possible now. First, as discussed in point (8), there are issues with claiming that the generated explanations are now human-readable due to misalignment with human intuition. Second, it is not shown that explanations on larger graphs (for instance, 50-100 nodes) are human-readable. Third, causal graphs learners generally do not output a single graph but an equivalence class of graphs, and SCE doesn't operate on equivalence classes of graphs. Fourth, SCE is limited to queries comparing variables' values to their means (Def 1); this seems far from \"any query of interest\". Thus, there is still a large gap between causal discovery algorithms and human-readable explanations on any query of interest, which SCE does not bridge.\n\nNext, I address the listed technical contributions. Overall, contributions 3-5 are not clearly stated, and their significance is unclear even with the context of the introduction and abstract. Contribution 1 requires more engagement with the prior literature on explanations and a clear statement + discussion of its limitations and assumptions. Contribution 2 relies on two qualitative arguments in section 4.1 which I do not think are sound.\n\n1. **The structural causal explanations (SCE) algorithm + the claim that SCE explanations are \"truly causal\"** Regarding the construction of the SCE framework, there seems to be very little justification for Definitions 1-3 when taking into account prior literature.\n\n    1.1. Regarding Def 1, why are \"Why\" questions constrained to variables' relationships with their means (Def 1)? Why aren't all kinds of why questions allowed, as in (Halpern, 2016)? It is stated after Explanation 1 that our \"Why\" questions are asked about *individuals*; if this is the case, are the unobserved variables in the SCM fixed?\n\n    1.2. Regarding Def 2, how is each causal effect $\\alpha_{X\\to Y}$ specifically defined? Is this a unit-level causal effect specific to a setting of unobserved variables $\\mathbf U$, or an average causal effect (a.k.a. average treatment effect)? If it is a unit-level causal effect, how is this supposed to be obtained, given that it is not possible to observe each variable's counterfactuals without the SCM? Is the assumption that we have the full SCM and the specific unobserved variable setting? If so, this should be stated explicitly in the contributions and the abstract, as it is even stronger than the assumption of just knowing the full SCM. And is the full SCM required? Or just the causal graph? If so, a causal graph is not sufficient to deduce all interventional effects from observational data, unless the graph is Markovian; if Markovianity is assumed, it should be explicitly stated as an assumption in the abstract/contribution section.\n\n    1.3. Regarding Def 3., what is the intuitive justification for each rule? Examples in cases of linear SCMs are given. As mentioned in my comments on claim (8), we can obtain unintuitive behavior with rule 3 even in linear cases. I think we can also obtain unintuitive behavior with rules 1 and 2 if the causal effect of increasing & decreasing is on average 0, but there are specific regions of increase/decrease where there is an effect of increasing/decreasing the variable (e.g., a sine curve). How does this change the justification for each rule? If the assumption is that each function is linear and/or monotonic, this should be stated explicitly in the contributions and the abstract; in addition, justification for each rule should be given, as well as discussion of the rules' limitations.\n\n    1.4. Regarding Def 4., it seems like a causal scenario $(\\alpha_{X' \\to Y'}, x', y', \\mu_{x'}, \\mu_{y'})$ is required for *every* pair of variables among the ancestors of the explained variable $X$. Is this correct? This isn't made clear in Def 2, and the causal scenario is referred to as a \"single tuple\", which makes it seem that there is only one causal scenario inputted to the SCE algorithm. In addition, why is the dataset $\\mathbf D$ required for the SCE algorithm if the causal effects $\\alpha_{Z \\to X}$ are already known from the causal scenario? Is there some additional step of transforming a dataset to a causal effect which is not included among the definitions? And from what distribution (observational, interventional, etc.) is the data in the dataset drawn?\n\n    1.5. Finally, it's not clear what \"truly causal\" means (as opposed to \"causal\" or \"purportedly causal\"), as this is not defined in the paper. The paper argues that SCE's outputs are \"truly causal\" by construction; it is not clear to me why this is the case.\n\n2. **SCE fixes several of the shortcomings of previous explainers.** Section 4.1 argues that SCE is superior to CXPlain because 1) some aspects of the causal graphical structure (variables with direct vs indirect effects) cannot be distinguished using CXPlain's outputs and 2) CXPlain's explanation does not give information on how changing each input variable will change the outcome.\n\n    2.1. The observation seems to be true, as CXPlain does not output any kind of recursive or graphical structure. But why is this a shortcoming? Is the explanation given by CXPlain wrong? Can't users look at the causal graph (which is required as an input to SCE) to obtain this information?\n\n    2.2. This seems to be a valid point about CXPlain. But is it true that SCE does give this information in general? It is certainly true in linear cases, where the causal effect can be modelled with a single value. What about in non-linear cases, such as the sine wave case mentioned in my comments in (1.3)?\n\n3. This seems to be an action the authors took rather than a result. Why are the applications mentioned in (3) novel/significant? \n\n4. **Improving model learning using SCE.** It's not quite clear what \"model learning\" means here; this point should be clarified. If this refers to section 4.3, then there are some points to clarify. What is the exact definition of SCE regularization? How is error computed? Is the difference in Fig 3 statistically significant? Doesn't SCE already take in the true causal graph, so isn't including an SCE-based regularization equivalent to passing the graph learning method additional information on the graph (this of course depends on what the definition of SCE regularization is)? All of these question would need to be answered to assess the benefit of SCE regularization to causal discovery.\n\n5. This seems to be an action the authors took rather than a result. Do the results of the study mentioned in (5) study support any novel/significant claims about SCE?\n\nActionable feedback:\n\n1. Either remove claims 6-10 from the paper or include them in the listed contributions section and include supporting evidence for them in the body of the paper as suggested above. For instance, XIL seems interesting, but the connection to SCE doesn't seem to be made; it would strengthen the paper to make this connection. In general, please do not include strong claims in the introduction/abstract which are then weakened in the listed contributions section.\n2. Remove claims 3-5 from the paper, as they do not seem to be contributions in themselves, and optionally replace them with novel/noteworthy conclusions from the listed application, discussion, and survey. Claim 5 could possibly be replaced with \"analysis of a user survey of 22 participants on 4 explanation settings that explores challenges for explanation methods to tackle in the future\".\n3. Regarding claim 2, highlight specific cases where CXPlain fails at its stated goal. For instance, construct two examples where CXPlain outputs the same set of scores but SCE outputs different explanations. Conduct a human study to show that humans find explanations generated by SCE more useful for performing a task (or higher-rated according to some other criteria) than explanations generated by CXPlain.\n4. Regarding claim 1, connect with the literature in greater depth to clearly state the assumptions/limitations of Definitions 1-3 in the abstract and contributions sections (or to change Def. 1-3 to encompass explanations as explored in prior work); some pointers include Halpern's approach to defining explanations and causation, which is cited in the paper (Halpern, 2016), and (MIller, 2017 - https://arxiv.org/abs/1706.07269). Clarify answers to the questions above in 1.1-1.4. Remove usage of the phrase \"truly causal\" or define it specifically.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "nZ69nJxZQHs", "reviewer": "Reviewer_j3te", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The focus of this paper is on Explanatory Interactive Learning (XIL), which consists of a loop in which the user queries the system, the system explains its predictions and the loop restarts. \n\nAn assumption here is that this interactive loop is interesting to both the system (improving its prediction accuracy) and the user (increasing their trust in the system).\n\nThe authors seek to causal XIL. Shortly the research question is how the ML system would provide causal explanations.\n\nAdditionally, they argue that previous explanation methods are not causal, even when they assume a Structural Causal Model (SCM). Their strong baseline is the CXPlain method, and they show that CXPlain doesn't provide causal explanations.\n\nHence, the authors propose an explanation method founded in a given SCM. They refer to their method as Structural Causal Explanation (SCE). Additionally, they conducted a user study to investigate the causal explanations based on SCMs.\n\nAiming to be self-contained, the paper should add some explanations on its relatedness to Actual Causality (mentioned at the end of Sec. 2). \n\nIt is a bit confusing when the authors mention M to be simultaneously (i) a valid why question and (ii) some proxy SCM (assuming that the hidden SCM is M^*).", "summary_of_the_review": "This paper address a relevant topic, i.e. explanations of learned models, and do this by exploiting concepts and techniques from the Causal Inference literature. The authors provide a toy example that guides the readers throughout the paper. Additionally, they validate several hypothesis and provide detailed information in Appendix.", "strength_and_weaknesses": "Strengths:\n\n- This paper presents a new causal-based explanatory method. Since many learning algorithms produce black-box models (e.g., Deep Neural Networks), explanatory methods are needed, and the causality aspect of the proposed method turns its application more viable in real-world settings (e.g., decision-making in healthcare).\n- When guiding the readers through a toy example, the authors provide the necessary intuition about their contribution.\n- The tentative validation  (see weaknesses below) of the failure of previous methods to explain ML predictions is relevant to highlight the contribution of this paper.\n\nWeaknesses:\n\n- As far as I understood, the proposed method assumes the learned (from data) causal graph, not the SCM. For instance, in Sec. 4.2, the authors investigate the possibility of generating causal explanations from the learned causal graph.\n\t- Or is it the case that after learning the causal graph, their method learns the coefficients of a linear model to infer the equations of the SCM? (assuming linear structural causal equations)\nIn any of these possibilities (or other understanding), the authors should clarify this issue in the text.\n- There is a lot of additional information in the Appendix, including important discussions, such as:\n\t- The relationship between the proposed method and Actual Causality.\n\t- Possible shortcuts to overcome the hidden confounder limitation of the proposed method.\nThere is no doubt that the authors provided a detailed Appendix for key points, and there is no sufficient space to include this information in the main text. My concern is that, at least, the above bullets should be in the main text.\n- As discussed in Sec. 3.2, a limitation of the proposed method is that it doesn't consider hidden confounders. This assumption is unrealistic in several real-world scenarios.\n- In Sec. 4.1., I did not understand the two shortcomings of CXPain. Since the SCM is given to both (i.e., SCE and CXPlain), it contains the information to derive the causal relations among the variables. Isn't it possible to use the input SCM to specify (i) direct/indirect causes among CXPlain's output? and  (ii) infer the causal effect among the output of the same explanation?\n- In terms of limitations, I believe that would not be easy to learn the causal graph for high-dimension data  (e.g., images/videos) that maintain semantic information in the graph. Hence, if that is the case, the authors should highlight the discreteness requirement of datasets for which their method is suitable.\n- Besides the cases of queries and explanations, I would like to see some general methodology to evaluate the quality of several queries. If that is not possible yet (i.e., it is an open research problem), the paper deserves some discussion on this.\n- Given definitions and the analysis of theoretical parts of ER and SCE, it is still unclear how there are a loop works and how the input from decision-makers, for instance, may alter the final explanations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bR9HRYXVm2O", "reviewer": "Reviewer_9oUq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this submission the authors present an end-to-end procedure for explanatory interactive learning. Most prominently, this submission focusses on a logical approach to constructing causal explanations of binary ordering relations, i.e. answers to \"why\" questions. They go on to test their approach in a survey of 22 participants.", "summary_of_the_review": "The paper presents an interesting and important idea in end-to-end interactive causal explanations. However, I have significant concerns about the theoretical and empirical claims.", "strength_and_weaknesses": "Strengths:\n\nThis paper addresses an important problem, and is clearly written. I found the Causal Hans example to be particularly helpful in understanding the paper's motivations, the problem scope, and the approach taken. \n\nOpportunties for Improvement:\n\nDespite the fact the paper addresses an important problem and communicates the approach clearly, unfortunately I have many signficant concerns about this work.\n\n(1) Theorem 1 is vague to the point of being unfalsifiable. To see this, consider an absurd structure learning algorithm that simply selects a random graph without looking at the data. While the statement is literally true that we could compute SCE, it says nothing about the quality of the derived explanation. Should a reader interpret the Theorem literally, in which case it is uninformative about explanation quality?\n\n(2) If my interpretation of Theorem 1 is correct, and it is claiming that the output of a sufficiently well behaved structure learning algorithm (e.g. a sound and complete algorithm that returns the entire markov equivalence class of graphs), then the Theorem is incorrect. Graph structure learning algorithms can not return unique structural causal models, as many structural causal models map to the same causal graph. In other words, there is a type mismatch between the output of a structure learning algorithm (equivalence class of causal graphs) and the inputs of the SCE procedure (a unique SCM). See Bareinboim et al for discussion of the relationship between SCMs and graphs.\n\n(3) The paper describes \"why\" queries in english language that map to formally to search procedures over counterfactuals. However, the authors do not take into account that counterfactuals involve holding exogenous noise fixed between factual and counterfactual worlds, nor do they consider the large literature on causal explanations with SCM. This is a bit perplexing, given that the authors cite Halpern 2016 as an analogous set of definitions.\n\n(4) The empirical results are presented in a way that is extremely uninterpretable. For example, the rightmost column of Figure 8 has no labels or numeric values at all. It is very important to have (at least) a summary of the empirical findings in the main body. In fact, I do not see anywhere in the appendix a table or graph comparing the proposed approach with CXPlain on a well defined metric.\n\nMinor concerns\n\nPage 3 - \"A great deal of research in causality (especially for ML) is concerned with leveraging observational data to reason about causal relatinoships (also known as identification), ...\" This is not exactly right. The task would be better described as \"effect estimation\". Identification, or identifiability, is when one wants to assess whether their causal assumptions are necessary to yield unique causal conclusions.\n\nPage 4 - \"We call Q_X := R(x, \\mu^X) a single why question if Q_X is true\". This seems like a type mismatch. If Q_X is the relation, then it is always a question. Is the intention here to say that the why question Q_X is a function that maps from R to explanations or answers?\n\nBareinboim, Elias, et al. \"On pearl’s hierarchy and the foundations of causal inference.\" Probabilistic and Causal Inference: The Works of Judea Pearl. 2022. 507-556.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "33gcXjwhIn", "reviewer": "Reviewer_Upiw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of deriving causal explanations for the decisions of opaque Machine Learning models. \nUnlike current state-of-the-art methods, this work proposes an explanation method that aspires to consider causal interactions by exploiting the knowledge of a given SCM. In particular, the novelty of the approach relies on the capacity to distinguish between direct and indirect effects and the possibility of identifying information on the causal effect with positive and negative cases.", "summary_of_the_review": "A well motivated paper with extensive and comprehensive experimentation. The evaluation of the paper would improve if the authors could be able to address the weak points highlighted above.", "strength_and_weaknesses": "Strengths: \n+ The paper is self-contained, and the contribution is well structured at the conceptual level. Thanks to straightforward examples and infographics, it is also clear for the reader to understand.\n+ Related work is condensed but rather complete, and background discussion places the current work in a relevant and impactful study area. \n+ Due to the approach's user-oriented nature, using a user study to evaluate the understanding of explanations is an engaging strategy. \n\nWeaknesses: \n- The availability of the full SCM is a strong assumption. In general, it is not testable and may thus not hold in practice. \n- Authors should introduce a discussion on possible limitations of the algorithm\n- All datasets used for the experimental evaluation contain a maximum of 4 variables. I would suggest to test the proposed approach w.r.t. dataset with higher dimensionality.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject