Structural Causal Interpretation Theorem
Authors: Matej Zecevic
Devendra Singh Dhami
Constantin A. Rothkopf
Kristian Kersting
Venue: ICLR 2022 Submitted (2022)
Keywords: causality
interpretations
neural causal models
induction

Abstract: Human mental processes allow for qualitative reasoning about causality in terms of mechanistic relations of the variables of interest, which we argue are naturally described by structural causal model (SCM). Since interpretations are being derived from mental models, the same applies for SCM. By defining a metric space on SCM, we provide a theoretical perspective on the comparison of mental models and thereby conclude that interpretations can be used for guiding a learning system towards true causality. To this effect, we present a theoretical analysis from first principles that results in a human-readable interpretation scheme consistent with the provided causality that we name structural causal interpretations (SCI). Going further, we prove that any existing neural induction method (NIM) is in fact interpretable. Our first experiment (E1) assesses the quality of such NIM-based SCI. In (E2) we observe evidence for our conjecture on improved sample-efficiency for SCI-based learning. After conducting a small user study, in (E3) we observe superiority in human-based over NIM-based SCI in support of our initial hypothesis.

Under review as a conference paper at ICLR 2022
STRUCTURAL CAUSAL INTERPRETATION THEOREM
Anonymous authors
Paper under double-blind review
ABSTRACT
Human mental processes allow for qualitative reasoning about causality in terms of mechanistic relations of the variables of interest, which we argue are naturally described by structural causal model (SCM). Since interpretations rived from mental models, the same applies for SCM. By defining a metric space SCM, we provide a theoretical perspective on the comparison of mental models and thereby conclude that interpretations can be used for guiding a learning system towards true causality. To this effect, we present a theoretical analysis from first principles that results in a human-readable interpretation scheme consistent with the provided causality that we name structural causal interpretations (SCT). Going further, we prove that any existing neural induction method (NIM) is in fact inter- pretable. Our first experiment (EI) assesses the quality of such NIM-based SCI. In (E2) we observe evidence for our conjecture on improved sample-efficiency for SCI-based learning. After conducting a small user study, in (E3) we observe supe- riority in numan-based over MiM-based SCI in support of our initial hypothesis.
being de-
INTRODUCTION
There has been exponential rise in the use of machine learning, especially deep learning in several real-world applications such as medical image analysis (Ker et al.) 2017). particle physics (Bourilkov) 2019), drug discovery (Chen et al.! (018)) and cybersecurity (Xin et al. 2018) to name a few. While there have been several arguments that claim deep models are interpretable, the practical reality is much to the contrary. The very reason for the extraordinary discriminatory power of deep models (namely, their depth) is also the reason for their lack of interpretability. To allevi- ate this shorteoming, interpretable machine learning (Chen al., 2019 Molnar| (020) has gained traction to explain algorithm predictions and thereby increase the trust in these learned models.
In their seminal book, Pearl Mackenzie (2018) argue that causal reasoning is the most important factor for machines to achieve true human-level intelligence. The same has been pointed out recently by Hooman et al. (202T) who argue that systems that are efficient in both causality and interpretations need of the hour. Questions of the form "What if?" and "Why?" have been shown to be used by children to learn and explore their external environment (Gopnik 2012 Buchsbaum et al.] al.]|2011 2012)) and are essential for human survival (Byrne 2016). This makes understanding and reasoning about causality an inherently important problem.
While acknowledging the difficulty of the problem, we push in this direction pragmatically by pre- senting the first work on causal grounding it in its causal semanties, which we call structural causal interpretations (SCI). We show how neural causal induction models due to their causal semantics are interpretable, but then go an- other step. by also suggesting how these interpretations, if available, can be used in the first place to improve on our models understanding, thereby also establishing their importance in the process. The motivation behind this work to move beyond black-box heat map based methods in explain- able artificial intelligence. Although there exist some works such as explanatory interactive learning (XIL) (Teso & Kersting 2019) and Clever-Hans methods (Lapuschkin et al., 2019 Stammer et al., 2021 that do move beyond such heat maps, we can go step further and provide structured causal interpretations. XIL methods can fix Clever- Hans like moments but are overly dependent on the expert provided explanations. SCIs provide elegant way to circumvent the fiever-Hans moments while also avoiding external factors such as expert explanations. Thus, we set the foundation for further research into making neural models, without initial causal semantics, possibly more causal.
a (deep) causal induction method by

---

Under review as a conference paper at ICLR 2022
Overall, we make the following contributions: (1) Starting from first principles involving men- tal models, causality and interpretations, we establish SCI theoretically to provide human- understandable interpretations from causal models, (2) we prove that SCI are readily available and that in fact any neural induction model (NIM) is SCI-interpretabll,, (3) we show empirically for three example members of NIM (Zheng et al. (0188) Goudet al. (2088) Yu et al. al. (2019)) multiple exam- ple SCIs and assess their quality relative to the true underlying causality, (4) we show that existing SCIs can help in improving the learning- barning-basedd causal induction, (5) we conduct a small human study to judge the qualitative causal structure for some graphs and show that gamann-aseed SCIs are close to the ground truth interpretations. We make our code publicly available
2 BACKGROUND AND RELATED WORK
Let us briefly review the background on a recent stride in learning that proposes a continuous formu- lation of data-driven estimation of (causal) graph structures, then the formalism of structural causal models and how estimation of respective dependency terms is conducted, and finally consider estab- lished works on the abstract high level ideas of explanations and interpretations.
Learning Directed Acyclic Graphs. Induction of inter-variable relationships based on available data lies at the core of most scientific endeavour (Penn & Povinelli 2007). The sub-class of relation structures known as Directed Acyclic Graphs (DAG) is being focussed here due to its representa- tional role in causality (Pearl) 2009; Peters et al.) 2017). Unfortunately, due to the combinatoric nature of the problem setting, learning DAGS from data is recognized to be an NP-hard problem (Chickering et al. 2004). In their seminal work, ZZeng et al.] (2018) were able to re-formulate the traditional view into continuous shape such that any non-convex optimization module can be applied to tackle the graph estimation problem. The authors propose the general formulation,
min WeRdxd
J(W)
subject to
h h(W) = 0,
(1)
where f is dasa- score, e.g. in Zheng al. 2018 regularized least-squares loss is applied
W + Wll. and h is a smooth function W is acyclic. For the acyclicity constraint, different variations of the same continuous counting mechanism have been proposed, c.g., Zheng et al. (2000) proposed h (W) = tr(e W. -d while Yu et al. (2019) proposed m. unfortunately, both suffer from cubic mantime-scalabilie in the
assuming a sparse linear SCM i.e. / [W)
with kernel (or null space) that only contains acyclic graphs, h (W)= =0
h(W)= = tr((T + W o oW) number of graph nodes ((d3). While the aforementioned works have focussed on data originating from (non-linea) transformation) of linear SCM, there exists yet another sub-class of DAG-learning methodologies that focuses on more general causal inference. Ke et al. (2019) made use of data from the first two levels of Pearl's Causal Hierarchy (PCH) (Pearl 2009: Bareinboim et al. 2020), namely observational and interventional, to update their graph estimate W = G(e) while using masked neural networks to mimic the structural equations f. MLP (pa(X,) in order to maximize the likelihood of the data under given parameterization. [Brouillard et al. (2020) follows the same idea of leveraging causal information, e.g. interventional data, for overcoming identifiability issues while staying close to the continuous optimization formalism introduced in
Causal Models and Effect Estimation. Following Peters et al. (017)) a Structural Causal Model (SCM) is defined as C = (S, PN) where PN is product distribution over noise variables and S is defined to be set of d structural equations
X, = ((a, No), where i 1,...,d
with pa(X: Xx) representing the parents of X; in graph G(C). An SCM C induces DAG G with edges (i,5) E V x V meaning i causes j. induces an observationallascciaaioon distribution pº, it can be intervened upon using the do-operator and thus generate interventional distributions p yCido(..)) and furthermore given some observations v can also be queried for interventions within a system with fixed noise terms amounting to counterfactual distributions pe/V=v,do(-. that is, C = PCH. To query for samples of a given SCM, the structural equations are being simulated sequentially following the underlying causal structure starting from the independent, exogenous variables. The dependency terms provided by the structural equations J; defines the translation between domains
ht nttpa://anonmmoii saapen.science///tttrroo -aw.a:la-aa-taaaa-a---a-----e--e www.-rrrrrrrrtttto-oo-eerr------
oon The--eherrem--
2

---

Under review as a conference paper at ICLR 2022
of the cause and the effect i.e., the causal effect g(X + Y). In most settings, this causal effect, e.g., of a medical treatment onto the patient's recovery. is the sought quantity of interest. If interventions admissible, then the average causal (or treatment) effect (ACE/ATE) within binary system is defined as a difference in interventional distributions: g(X + Y) == ACE(X, ,Xx, y) = TEl do(
1)] TE[Y do(X -)}- The ACE overcomes the fundamental problem of causal inference, which states the restriction that individual-level causal effects cannot be estimated as we cannot observe counterfactuals, but still suffers from the problem of identifiability i.e., that we usually don't t even have access to interventions and thus need to resort to other causal knowledge (e.g. structural knowledge on confounders).
Explanations and Interpretations in Deep Learning. A great body of work within deep learning has provided visual means for explanations of how a neural model came up with its decision i.e., importance estimates for a model's 's prediction are being mapped back to the original input space e.g. raw pixels (Selvaraju et al. 2017, Sundararajan et al., [Schwab Karlen 2019, Schulz et al.) 2020). Stammer al. (2021) argue that such explanations are insufficient for any task that requires symbolic-level knowledge while comparing the existing state of explanations to children that are only able to point fingers but lack articulation. While agreeing with the aspect that simple heat- map explanations are indeed insufficient for reasoning tasks, we consider the higher order task of understanding the meaning of matter (why) be akin what is meant by the word interpretation, while an explanation (invariant to its instantiation) refers to identifying the matter of interest (what). Seen from this perspective, explanations are crucial sub-process within interpretation. Thereby what the authors in (Stammer et al. (202I) seek is in fact an interpretation i.c., the articulation by the child which also takes into account the causal dynamics of the scene.
3 STRUCTURAL CAUSAL INTERPRETATIONS
We first provide big picture view the way human mental models can be expressed in terms of SCMs and how only the true underlying causality is of interest during optimization. Then we provide intuition with our leading example on how causal semantics can be queried for answers that interpret their response based on the underlying causality before formalizing our main results.
3.1 HUMAN MENTAL MODELS AND THE IMPORTANCE OF TRUE CAUSALITY
To illustrate one's thought process about the understanding of the world dynamics is argued to lie at the core of a human mental model (Simon) 1961, Nersessian) 1992, Chakraborti et al. 2017). Although the concept of mental model might contain circular and abstract terms like explanation and interpretations of its own, assuming the world dynamics to be governed by causality we ob- serve that humans capable of modelling both causal relationships between system variables and additionally information on the strength of said relationship. Consider the following real world ex- ample, at any given time human has state of overall health (relating fat-muscle ratio, allergies and diseases, etc.) and mobility (relating to the general freedom and ficxibility of movement, c.g., a gymnast is more mobile than the average person). Now, human can argue the following (1) that mobility is being (partially) directly caused by something, in this case for instance health, e.g., overweight will decrease the range of motion of any individual (2) that different events (even if me- diated via the causal variable health) can have a different severity of impact e.g., that average car accident causes more harm to the mobility of an individual than does an average workout session at the gym does good. A natural candidate for capturing such properties formally are Structural Causal Model (SCM), thereby we hypothesize the following:
Hypothesis 1 (Causal Mental Model Conversion (CMMC)). The parts of the human mental model that being used for encoding the causal relationships of reality's variables can be for- mally captured by a corresponding Structural Causal Model (see (2)))
The CMMC hypothesis suggests that we can make use of SCM for expressing portions of human understanding and intuition within mathematical language. Given such formulation with SCM, the implications allow for some interesting observations to be made with the key observation being that SCM live in metric space:
For an extended treatise why this is sensible assumption consider (Pearl 2009
Peters et al. 2017
3

---

Under review as a conference paper at ICLR 2022
(Alm
SCM farasase y-me- en Jeasia
Clesss ouann
Kurt A FHNNSA
(Plood Hulei (M)ahhh (kdjobiliy
M M-NNA M Ri N 24
C
C
Hans
Jones
Figure 1: Structural Causal Model Perspective onto Human Mental Models. The CMMC
hypothesis (see Hyp states that human mental model, which are capable of both modelling causal relations (overall health affects mobility, H - M) and the strength of such a relation (an average car accident has more negative impact, g(Hl + M). than an average workout has positive), can be naturally formalized within corresponding SCM. Furthermore, SCM live on a metric space (C, d) (see Thm D and can overlap (e.g. in pink, all humans agree on H + M), therefore, most causal estimates will in fact deviate from the underlying truth, E * E*. (Best viewed in color.)
Theorem 1 (n-SCM Metric Space). Let S be a set of structural equations X3 == Is(pa,, N₂) and PN product distribution, then Cn ((S, PN) |S| =n) defines the set of all constructable
n-SCMs. Furthermore, define d : (C), C2) + Eiz, lo(s)()) - a(S +
PR) where
on the product distributions and g(j + i) E R defines the (expected) causal effect of j on i where Sz(j) refers to the isolated dependency term within the struct.eq. i.e., [pa,, N:) = [N( + Ejepa, [in(i). Then (Cn, d) defines a metric space.
q == VJSD is the Jensen-Shun Seettt-Shannnn-Mttn
Because of space restrictions we provide the proof to Thm land all subsequent mathematical results within the supplementary section. This metric space describes the set of all SCM, which themselves are a set of functions in addition to product distribution, and they can all be compared against each other by the nature of their formulation. Under the CMMC hypothesis, it follows from Thm | that in fact parts of the original human mental models can be compared against each other thereby capable of dis-fagreeing. Interestingly, this comparability of SCM/human mental models allows for a trivial but crucial insight: if we additionally make the sensible assumption that there exists an 'objective' true SCM within the metric space, e" E C, then most hypothesizable SCM will in fact be wrong. This insight tells us that causality itself is not what will help in improving our models but being close to the true causality will. Note that while we considered an 'objective true causality", human mental models are of subjective nature. We make the argument that having access to many SCM-encodings of subjective human mental models can ultimately lead in their overlap- agreemeet to (parts of) the objective true causality i.e., a higher quantity equates to a higher likelihood. All the established key concepts thus far are being visually illustrated within Fig For machine learning research, observing that only a convergence towards true causality beneficial might not be surprising, but observing that set of 'subjective' SCM can provide information is indeed surprising if we consider interpretations. Interpretations are derivable from mental models and thus implicitly contain partial information on the latter's world dynamics representation (Chakraborti et al. 2017). Subsequently, we can query humans for their interpretations, then we can make use the acquired data, which by definition contain causal information, improve existing ML models by providing them with (parts of) the true causality. In the spirit of the motivation provided by Pearl that all MI. models should be aware of cause and effect. The benefits of an approach using interpretations derived from SCM two-fold (1) that by construction they are human understandable allowing for interpretable ML in which models can reason about the learnt and (2) that the models themselves become better which is beneficial to any downstream-tass
After establishing the connection between the formalism of causality and human mental models and using for arguing that derived interpretations can be used to improve ML models, we now move onto the question of how to derive such interpretations from the causal semantics in the first place while alongside providing intuitive examples.
3.2 ANSWERING HUMAN UNDERSTANDABLE QUESTIONS ABOUT DATA USING CAUSALITY
Upon establishing that interpretations are derivable from SCM and should be used for improving MI. models, this section provides intuitive understanding of how data, queries and causal seman-
4

---

Under review as a conference paper at ICLR 2022
SCM C
J(S(C), Q.D)
(unobserved Nature)
Graph Cursal Fffves
(Food Hoonss pooouity
Righ Age
padd FIL
felffal
"Mansl Hubiliyy (M) bad his bad Health which is mostly due his high Age (AT,
((V.0)
IHAA,F F.
Dota
Duery / Querter
othhoogn food Habbits (P) are goodt"
P(U)
Figure 2: Why Questions and Structural Causal Interpretations The unobserved nature depicted by SCM C on the left. The why-questton Q is individual-level query derived from some pop- ulation D originating in C. The graphical structure of C alongside its causal effect terms induces the sub-structure S(C). Following Thm 2 we arrive a human expressible answer: the Structural Causal Interpretation I(S(e) Q, D). Our paper's 's lead example concerning Hans's 's mobility is be- ing highlighted alongside the intuitive-level computation in the graph (right). (Best viewed in color.)
ties are being used in conjunction for generating interpretations both for explanatory and learning purposes. Figure is being used as lead illustration for the interpretability of causal models along- side different weal-world inspired examples. In the following, we consider (single) why-questions as goal to model inference they induce answers that reason c.g. ". ... because ... which are gener- ally accepted as inferences about causal quantities (Pearl 2009). To follow suit with the previously established examples around personal health, we could pose question like "Why is Hans's 's Mobility bad?" after observing the state of causal variables for the individual named Hans, with the causal attributes being age, nutrition/food habits, overall health and mobility respectively. Gíven a SCM that represents age and nutrition as being causal for the health and subsequently mobility, while nu- trition is generally also being affected by age. one can conclude the following human understandable interpretation as an answer to the initial question about Hans's 's rather immobile state:
Interpretation 1 (Hans's Questioned State of Mobility). "Hans's 's Mobility is bad because of his bad Health which is mostly due to his high Age although his Food Habits are good."
This causal interpretation of Hans's personal state captures both reasons and importances, a sequence of the CMMC hypothesis (see [ established previously. More importantly, Int has been constructed automatically from the available data and causal knowledge in response to the query. As a first observation towards the interpretation construction, note that the why-query contains a relative notion "why... ... bad?" that implicitly compares the individual Hans the ques- tionee's 's available data set i.e., remaining population. As a second key idea, note the trivial conclu- sion that by definition there can only exist a causal effect from some variable to another if and only if there is a causal directed path between the variables. As third and final concept, consider the difference causal effect i.e., that some variables exert more influence than others. Unifying these three building principles in formal manner, to be discussed in the subsequent section, allows for the generation of interpretations like Int [ That is, our SCM tells us that mobility of any individual will decrease with decreasing overall health and upon observing that Hans's 's health has been "bad" in the first place, we can conclude that the questioned state of mobility is due to Hans's 's health. We can now increase the granularity and reason recursively that, yet again, Hans's health is such due to his age and nutrition but this time we can introduce two more detailed distinctions. One, that Hans's nutrition is actually above average and therefore not the reason for his immobility. Second, that age's 's causal effect wages in more than that of nutrition, as a general statement of the given SCM, thus the asymmetry in attribution ("mostly due"). Conclusively, a Structural Causal Inter- pretation (SCI) depends on a measure of relativeness on a given data population, as implied by the given why-question, and on the structural equations provided by the assumed SCM It is worthwhile noting that the natural language choice of words to express the interpretation is not implied by the form of the SCI e.g., while Hans's mobility is said to be "bad", car's 'u's mileage is considered to be "low" rather than "bad". On another note, similar to how Pearl has argued that Bayesian Networks (BN) are a suitable representation scheme for causality (Pearl) 22099; 201T) because of BNs being direct representations of the world and not of reasoning processes, this work poses a parallel to the 'representation scheme' of interpretability using causal semantics. Interpretability becomes a part of the causal nature of the modelled. In the following, we mathematically formalize the established intuition on SCI and then proceed with the theoretical analysis of SCI-implications.
'The naming resembles that of SCM because of the importance of the structural causal information for generating any specific interpretation.
5

---

Under review as a conference paper at ICLR 2022
3.3 MATHEMATICAL FOUNDATIONS OF STRUCTURAL CAUSAL INTERPRETATIONS
Interpretations as established in the previous section (see Int act in response to a query. The queries or questions we pose are single-attribute assertions relative a population that correspond to the causal inquiry "why?", formally:
Definition (Single-hhy Question). Let xi E D(X) a scalar instance of X, let μ.x be some population score (e.g. the mean of a sample px := E I2) and let R be a binary ordering relation (e.g. RE E {{,>}) then a single-why question concerning is true assertion Qx R[ px).
Using De the query concerning our lead example from the previous section used for Int is expressed as Qaas := MaHn < μMM where PM is the average mobility an individual. The unification of the previously established three guiding ideas for generating interpretations (causal connection, strength and relativeness to population) can be formalized in set of logic rules:
Proposition (First-Order Logic Rules for Interpretations). Let g(X-+Y) E I be a causal effect estimator, let s(x) E {-1.1} be the sign of scalar, let Zx = -(lgZ=>X) ZEpay, be the set of absolute parental causal effects onto X. and pxx R as in Deffz] Furthermore, rule indicator is defined (x())=(-)1 signalling with which relation (if) rule j applies to input X and O otherwise. Then for any pair X E V, V. E pa(X) and some individual-sean sacaual---aarry y E D(Y).
(R1) Excitation:
R2 = (R()) A (Ra(y. NY) V RI (I,,xx)]] (((((((--))),))) (Y) Re(x,
(R2) Inhibition:
R2
(R3) Preference:
1xx >1= Y = arg maxzeez Z].
indicating for each rule if and how the causal relation Y - satisfies that rule (e.g. for RI either over- or inder-excitations For any causal sce- Mx) excitation (RI) and inhibition (R2) mutually exclusive.
define rule-set function (R(Y-X)E(-1,O,))
nario Cxy:=(g(Y->>))
The three rules in Prop I lare naturally derived from the principles established in the previous section. The rule-set function A forms the core of the interpretation process which we finally formalize after introducing one last mathematical concept to provide for the causal semantics of the interpretation:
Definition 2 (Causal Effect Matrix (CEM)). Let C be SCM and g IV x [VI - R the causal effect function, we refer to S())= == (g(i -> j))is E [VIV as the causal effect matrix of SCM C. As a special case, if (()) E [0, 1]|VXx|V| then we call the resulting matrix the sign of the SCM.
is important to note that the CEM in 2 is a sub-structure to the SCM i.e., it contains less information. The CEM eventually enables the unification of existing approaches from DAG-learning into a family of interpretable models. These very interpretations that pose one of our theoretical main results, Structural Causal Interpretations, we finally formalize and prove recursion:
Theorem 2 (Structural Causal Interpretations). Let Qx. S(C). and D E Raxd be single- why question on variable X, a CEM, and a data matrix respectively. Furthermore, let @h Vs = (v1.....te vn) denote concatenation and R be the rule-set function from Prop Then Structural Causal Interpretation is being defined recursively as
((Qx,(()) D) =
+
A(Z + X),
J(Qz, D)) ZEPA(X)
(3)
ZEpa(X)
with the base case being evaluated at the roots of the causal path to X, that is,
37 V (Z
X pa (2) + J(Qz, S(C), 0).
The recursive algorithm for computing S(C), in Eq terminates on any input.
Intuitively, the construction SCI in Thm considers maximum depth interpretations i.e., the re- cursion is being rolled out along the possible causal paths that converge into the query-variable until the exogenous root variables are being reached. Thereby, SCI implicitly define notion of detail to the interpretation based on the rellout-depth of the recursion. Considering again lead ex- ample in Int from the previous section, the maximum-depte interpretation would also contain an argument on Hans's Age and Food Habits relation e.g. '[...] good Food Habits because of his high

---

Under review as a conference paper at ICLR 2022
(Left)
(Right)
11
1: (A) Altstude, (1) Temperature: 2: (A) Age, (F) Nutrition. (H) Health, (M) Mobility
(C) Size, (S) Driving Style, (M) Fuel Savings; (T) Treatment, (7) Pre- Condition, (R) Recovery
"The temperature the Matterhom low because of the ligh altitud."" (Ground Truth)
"The temperatine of the Materhom is low because of the high altitude." (Learned)
"Hans's mobility
because
which mostly
high age, although nutrition good."
"Hons's 's mobilin, in spite his high is bad mostly because of his bad health which is bad mostly due to his good murrition.
3
"Your left milcage is low because of your small cur and your bad driving style."
"Your mileage
because of your small Kurtt did not recover because of his bad pre-conditions. although he treatment."
your bud driving style."
"Kurt did not recover becmuse of his bad pre-cond. which were had although he got treatment."
Table 1: Quality of Learned Interpretations. Ground Truth SCM (leftnormal) versus NOTEARS
from Zheng et al. (2018 8) (right;italics). The considered Why-questions can be found pronounced as well as formal within 2 respectively. Subtle differences between interpretations exist e.g., the last interpretation is right on the top-level but for the wrong reasons (T' - Z instead of T' - R).
Age (life-time experience)'. At each recursion step, the satisfaction of the rule-set function is being examined which ultimately dictates the reading of the interpretation. In a nutshell, the why-question determines the starting point for the SCI recursion while the given causal semantics and data provide for the interpretation specific to the model and the available information. In the following, we make the leap towards connecting interpretability with existing methodologies from the DAG learning literature by first introducing formally the family data-driven CEM estimators:
Definition 3 (Family of CEM Estimators). Let L D(C) x H + R be a scalar function on the data D E D(C) generated from some underlying SCM C (S, PN) and on some DAG-structured hypothesis space H. whose elements are defined as
X1
X f1
X
fij
X1
fin
X1
X,
X₂
(5)
fin
with n SS] and functions fúg describing -> J with scalar f(x)) E TR. The family of Causal Effect Matrix (CEM) estimators encompasses any learner that solves H" arg minнeи L(D, H).
Def provides a big picture view on methods that perform causal induction and more importantly it argues that any induction performed on the restricted hypothesis space of DAGs is fact causal induction because any resulting hypothesis respects the structure of SCM and will thus, by con- struction, resemble some SCM E E C even if this SCM does not resemble the sought SCM e which generated the available data D E D(C") i.e., e* + C. This big picture view is inline with (Pearl Mackenzie 2018) where a next generation learning systems capable of reasoning about cause- effect relations opposed to simple correlations is being endorsed. With Def we make two key observations, firstly, that methods that estimate an SCM are more powerful than CEM-cstimator:
Proposition 2 (SCM- and CEM-Estimators). Any SCM estimator is CEM estimator.
Thereby, additional structural/causal information will only benefit the modelling procedure. The second consequence of pese] our third and final main result in that a host of available (ncural) models from the DAG-learning literature are all interpretable:
Theorem 3 (Neural Induction Models are Interpretable). Algorithms for (causal) DAG-induction from data (e.g. NT (Zheng et al. [2088), CGNN (Goudet et al. [218)) DAG-GNN (Yu et al. 2019) NCM (e et al.) 2019)) can generate Structural Causal Interpretations.
Thm is important as it tells us that the causal reasoning, at least the level of CEM, performed by any model-type (includes neural network based models) can be queried for causal interpretations in the sense of SCI (Thm 2). While the ability to provide answers to questions in expressive manner which captures what has been learned by the model is high intrinsic value, in the following consider how existing interpretations be exploited for
means to improve model
7

---

Under review as a conference paper at ICLR 2022
Teepamay
Figure 3: Interpretations Improve Graph Induction. On the left a comparison of the error dis-
tributions when performing graph induction with and without existing interpretations. Alg with Ma NOTEARS is being applied. The number of falsely inferred graph links is being being duced significantly as predicted by Conj ! Technical details in Appendix. (Best viewed in color.)
learning in the first place. The following approach can be seen as an 'interpret to learn -approach as the interpretations provided by the model are being evaluated against existing interpretations (possibly provided by humans) for improving the overall estimate of the underlying causality. can also be seen as a regularizer on the model since it can extend on any loss formulation as long as the interpretation process I (usually I = J where refers to 3q can be incorporated into the corresponding optimization routine (i.e., differentiability in the case of neural approximations)
Conjecture (SCI Regularization). Let be defined as in Defl3 and let H be the true CEM. Regularizing L, := L + Σ, [I[QQx, D) 1.ll2 with existing and overlapping SCI I leads to more optimal solution arg minnew (D, H) H* <
arg minneh L(D, H) H*)
The interpretations contain informa- tion about their respective CEM. While CEM are hard to estimate and usually an integral part of the sought quantity of interest, corresponding (approximate) interpretations are of- ten times readily available and can thus be used. The regularization term introduced in Conj thereby penal- izes CEM that would not be able to account for certain explanations to the posed single-why questions. Fur- thermore, Conj poses an interesting direction of future research concern- ing more general classes of causal models that could enable a tighter in- tegration between current practices in deep learning and causality, as in (Xia et al. 2021 for general Neural Causal Models (NCM) and (Zečević et al.) 202Tb) for Graph Neural Networks based NCM.
Algorithm Dual Learning: Induct & Interpret
Input: Data D, Inductor Me. Interpret. I, Optimizer O Output: CEM H, Causal Graph G
I: Let H + 0
2: while
[I] do
3:
H,1 < 1MM (D) {induction from data]
4:
Qx, I"
5:
It I(Qx. H. D) festimate's 's interpretation)
6:
4 || r 1 {compare}
7:
0 0 {parameter update]
8: end while
poses
9: H + Ma (D). and G <- |tanh (H)
10: return H, G
EMPIRICAL ILLUSTRATION
To assist our theoretical results we provide an empirical illustration. Due space constraints, choose NOTEARS (NT, Zheng et (2018) as representative for NIM-based experiments and fur- ther only highlight relevant key results while pointing the extensive analysis in the Appendix.
Experiment 1: NIM-based SCI (Thm 3). Tab considers four data sets and single-why queries illustrated in Fig2 i.e. Deutscher Wetterdienst (DW, Mooij et al.) (2016). Causal Health (CH, Zečević et al.) aa)2 Mileage (M), and Recovery (R, [harig et al. (1986)). The algorithmic interpretations are sensible but differ in terms of quality. The SCT (Thm () generated using the leamned causal semanties match the given ground truth interpretations for the DW and M data sets, while differing only slightly for R and drastically for CH data sets. The violation of the linearity assumption might offer NT-specifii explanation. An extensive study is provided in the Appendix.
Experiment 2: SCI-based Learning (C.nj[I). We use Alg where we set I = Dg with latter being neural approximation to the SCI recursion from Fg to assure differentiability. The graph
8

---

Under review as a conference paper at ICLR 2022
Distribution
Human Data N 22
Unique Structure Distribution
Veriant 1
Verient
"Overlap"
"Mode"
Figure 4: Human CEM Agreement Variants. We consider two variants of measuring agreement within the human causal induction data: overlap (H1) and mode (H2). The H1-CEM structure is a greedy aggregation that chooses the most-voted edge type for each edge. The H2-CEM is simply the most firquently occuring CEM structure. (Best viewed in color.)
3
A
T-A
"Hans's mobility bad because his had health which is mostly due to his high age, although his nutrition is good."
H2
=H11
A
"Hans's 's mobility, spite his high age.
mostly becouse of his health which bad mostly due to his good nutrition."
"Kurt did recover because his had pre-conditions, although he got treatment."
112
"Kurt did not recover although his pre condition and fact got treatment." becouse of had pre-cond.. which bad although he got treatment."
"Kut did
Table 'Humans vs Algorithms'. Top: Edge plots per example where the bars denote the average value of given relation and the errors confidence intervals. Bottom: The SCI generated for the two human variants (HI, H2 from Fig 4 against an algorithm representative (NT, Zheng et al.) (2018). Human interpretations are (near- Jidentical to the ground truth from Tab (Best viewed in color.)
induction is being performed Erdos Rensi structures in data-scarce setting with only 10 data samples per graph induction. shows our empirical results on the error distributions for all the graphs with the best improving optimization being highlighted. We observe an increase in sample- efficiency. The interpretations contain valuable information about the underlying CEM that allow reducing the search space, thereby favoring Conj
[
An extensive study is provided in the Appendix.
Experiment 3: Algorithmic vs. Human Interpretations. We let 22 human subjects judge the qualitative causal structure for each of the four examples (questionnaire Appendix). In Fig we consider two variants of measuring the structure that the humans agreed upon. In Tab we show the quantitative edge plots (top) which expose common ground the human-based inferences and resulting SCI (bottom) which significantly outperform the algorithmic-based The humans results are support of the CMMC hypothesis [T. An extensive study is provided in the Appendix.
CONCLUSIONS AND FUTURE WORK
Starting from first principles. we discussed the connection between human mental/thinking modes and SCMs [Typ[T) which lead to the derivation of a metric space for SCMs (Thm We then derived three basic rules (Prop[]) that jointly with why-questions (DeffT) allow for establishing Structural Causal Interpretations (SCI, Thm We proved that any neural induction method (NIM) thus interpretable (Thm 3). Empirically, we showed that the NIM-SCI are sensible (E1), that SCIs help in improving learning (E2: Conj and finally human case study that allows for comparing NIM- and human- based SCI (E3). Following, an extension of the query type (Def [T and accordingly
SCI-formalism (Thm 2) might allow for more complex interpretations aligned in spirit with the intuition behind the PCH-levels L; (Pearl & Mackenzie 2018)) Baareiboim et al. 2020). Another natural followup perform large-scale user study for curating rich human-basee SCI data set.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
With our work, we have shown that we can seamlessly encapsulate several (deep) causal induction methods inside a single interpretation framework that is consistent with learned causal semantics. The major impact that work pursues is to provide human understandable answers to human understandable questions while respecting the learned (or assumed) causality. This endeavour can have several implications producing causally interpretable signatures, which could be predictive of the emergence and the success of new fields or discoveries.
Another concern can be the use of the human user study in our work. Although the questions asked are very general and do not pose any threat to the privacy of the participating users, we anonymize the responses before using them. To the best of our knowledge, privacy or conflict of interest concerns and obliges common practices and standards for experimental setups in behavioral and social sciences.
study does not raise any ethical,
REPRODUCIBILITY STATEMENT
We provide the code and data sets to reproduce the reported results through an anonymous online repository. Lastly, details on the conducted user study along with the received and anonymized responses are also provided in the appendix.
REFERENCES
Elias Bareinboim, Juan Correa, Duligur lbeling, and Thomas Icard. On pearl's hierarchy and the foundations of causal inference. Technical Report, 2020.
Dimitri Bourilkov. Machine and deep learning applications in particle physics. International Journal of Modern Physics A, 2019.
Philippe Brouillard, Sébastien Lachapelle, Alexandre Lacoste, Simon Lacoste-Julien, and Alexan- dre Drouin. Differentiable causal discovery from interventional data. arXiv preprint arXiv:2007.011 01754, 2020.
Daphna Buchsbaum, Sophie Bridgers, Deena Skolnick Weisberg, and Alison Gopnik. The power of possibility: Causal learning, counterfactual reasoning, and pretend play. Philosophical Transac- fions of the Royal Society B: Biological Sciences, 2012.
Ruth MJ Byrne Counterfactual thought. Annual review of psychology, 67:135-157, 2016.
Tathagata Chakraborti, Sarath Sreedharan, Yu Zhang, and Subbarao Kambhampati. Plan explana- tions as model reconciliation: Moving beyond explanation as soliloquy. IJCAI, 2017.
Clive Charig. David R Webb, Stephen Richard Payne, and John E Wickham. Comparison of treatment renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy. Br Med J (Clin Res Ed), 1986.
Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin. This looks like that: deep learning for interpretable image recognition. NeurlPS, 2019.
Hongming Chen, Ola Engkvist, Yinhai Wang, Marcus Olivecrona, and Thomas Blaschke. The rise of deep learning in drug discovery. Drug discovery today, 2018.
Max Chickering, David Heckerman, and Chris Meek. Large-sample learning of bayesian networks is np-hard. JMLR, 2004.
Kahneman Daniel. Thinking, fast and slow, 2017.
Alison Gopnik. Scientific thinking in young children: Theoretical advances, empirical research, and policy implications. Science, 2012.
10

---

Under review as a conference paper at ICLR 2022
Olivier Goudet, Diviyan Kalainathan, Philippe Caillou, Isabelle Guyon, David Lopez-Paz, and Michele Sebag. Learning functional causal models with generative neural networks. In Explain- able and interpretable models in computer vision and machine learning, pp. 39-80. Springer, 2018.
Thomas L Griffiths and Joshua B Tenenbaum. Optimal predictions in everyday cognition. Psycho- logical science, 17(9):677-773, 2006.
Masasi Hattori. Probabilistic representation in syllogistic reasoning: theory to integrate mental models and heuristics. Cognition, 157:296-322 2016.
Jake M Hofman, Duncan J Watts, Susan Athey, Filiz Garip, Thomas L Griffiths, Jon Kleinberg, He- len Margetts, Sendhil Mullainathan, Matthew J Salganik, Simine Vazire, Alessandro Vespignani, and Tal Yarkoni. Integrating explanation and prediction computational social science. Nature, 2021.
Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, Stefan Bauer, Hugo Larochelle, Bernhard Schölkopf, Michael C Mozer, Chris Pal, and Yoshua Bengio. Learning neural causal models from unknown interventions. arXiv preprint arXiv: 1910.0001 2019.
Justin Ker, Lipo Wang, Jai Rao, and Tchoyoson Lim. Deep learning applications in medical image analysis. IEEE Access, 2017.
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 2019.
Christoph Molnar. Interpretable machine learning. Lulu. com, 2020.
Joris M Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf. Dis- tinguishing cause from effect using observational data: methods and benchmarks. The Journal of Machine Learning Research, 17(1): 103-1204, 2016.
Nancy Nersessian. In the theoretician's 's laboratory: Thought experimenting as mental modeling. In PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association, 1992.
Judea Pearl. Causality. Cambridge university press, 2009.
Judea Pearl. Bayesian networks. 2011.
Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018
Derek C Penn and Daniel J Povinelli. Causal cognition human and nonhuman animals: A com- parative, critical review. Annu. Rev. Psychol., 58:97-118, 2007.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference. The MIT Press, 2017.
Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. Restricting the flow: Information bottlenecks for attribution. arXiv preprint arXiv:2001.00396 2020.
Patrick Schwab and Walter Karlen. Cxplain: Causal explanations for model interpretation under uncertainty. arXiv preprint arXiv:1910.123:4 2019.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In 2017.
Herbert A Simon. Modeling human mental processes. In Western joint IRE-AIEF-ACM computer conference, 1961.
Edward H Simpson. The interpretation of interaction contingency tables. Journal of the Royal Statistical Society: Series B (Methodological), 13(2):238-24 1951.

---

Under review as a conference paper at ICLR 2022
Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. Right for the right concept: Re- vising neuro- syuuolsymbblli concepts by interacting with their explanations. In CVPR, 2021.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML. PMLR.
Stefano Teso and Kristian Kersting. Explanatory interactive machine learning. In AIES, 2019.
Kevin Xia, Kai-Zhan Lee, Yoshua Bengio, and Elias Bareinboim. The causal-neural connection: Expressiveness, learnability, and inference. 2021.
Yang Xin, Lingshuang Kong, Zhi Liu, Yuling Chen, Yanmiao Li, Hongliang Zhu, Mingcheng Gao, Haixia Hou, and Chunhua Wang. Machine learning and deep learning methods for cybersecurity. IEEE access, 2018.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. Dag-gnn: Dag structure learning with graph neural networks. In International Conference on Machine Learning, pp. 7154-7163, PMLR, 2019.
Matej Zečević, Devendra Singh Dhami, Athresh Karanam, Sriraam Natarajan, and Kristian Kersting. Interventional sum-product networks: Causal inference with tractable probabilistic models. arXiv preprint 1r..v:2102. 2021a.
Matej Zečević, Devendra Singh Dhami, Petar Veličković, and Kristian Kersting. Relating graph neural networks to structural causal models. arXiv preprint arXiv:2109.04133 2021b.
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. In NeurlPS, 2018.
Xun Zheng, Chen Dan, Bryon Aragam, Pradeep Ravikumar, and Eric Xing. Learning sparse non- parametric dags. In International Conference Artificial Intelligence and Statistics, pp. 3414- 3425. PMLR, 2020.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX STRUCTURAL CAUSAL INTERPRETATION THEOREM
We make use of this appendix following the main paper to provide the proofs to the main theo- rems and propositions in addition to further assisting theoretical results, alongside details on the regularization experiment and details the execution of the human study.
A. PROOF FOR THEOREM
We argue strongly for the view that adequate or "truc" causality is what learning systems should seek to model opposed simply any causality derivable from the given data. In fact, we believe this misconception to be a strong oppressor of current research practices. An important consequence of the fact that different causal modellings exist, is that they become comparable in terms of their SCM specifications. The following theorem suggests that there exists always a well-defined metric space for SCMs of same size, allowing for a feasible distance computation between any two SCMs within.
Theorem 1 (n-SCM Metric Space). Let be a set of structural equations X, := fy N2) and PN product distribution, then Cn {(S, PN) =S] defines the set of all constructable
n-SCMs. Furthermore, define d (C), C2 + Eiz lo(s) s(S) + q q(PN N. PR) where
on the product distributions and g(j - i) E defines the (expected) causal effect of j on i where S,(j) refers the isolated dependency term within the
q = VJSD is the Pensen-Shannon-
struct.eq. i.e., f((pa,.N) = fN( + EjEp,, fta(i). Then (Cn, d) defines a metric space.
Proof. The absolute difference on the real numbers a metric (i.e., positive-deffnitensss,, symmetry, and triangle-inequality hold) and g(j + i) E IR, furthermore, g is chosen as the Jensen-Shannon Metric by construction. Finally, a sum of two metrics is itself a metric.
A.2 PROOF FOR PROPOSITION
The logic defined in Prop | posed the key to defining the actual interpretation scheme later on in Thm 2 is sensibly defined to adhere to the causal semantics of the system and in the following prove that the logic rules are also consistent w.r.t. to the causal semantics i.e., for any possible kind causal scenario, expressed through (a y, x, px px), we prove that either excitation (R1) inhibition (R2) occur but never simultancously, with preference (R3) optionally.
Proposition (First-Order Logic Rules for Interpretations). Lel g(X E be a causal ZEpa be the set of absolute parental causal effects onto X, and pxx as in DefL2 Furthermore, rule indicator is defined as 1x())((-)1 signalling with which relation (if) rule j applies to input X and 0 otherwise. Then for any pair X E V,Y E pa(X) and some aedividual-scalaen y E DYY).
effect estimator, let s(x) E (-1,1} be the sign of a scalar, let 2x = =11((
(R1) Excitation:
R1 = (s((((
(Rely. HY) (x,
(R2) Inhibition:
R1 R2 = RR (R(( -X))), 0) A (Y) A Re(2,
(R3) Preference:
1x
Y arg mrgzzxz2x 2/.
define rule-set function R(Y B(Y=+X)E(-1,001) indicating for each rule if and how the causal relation Y - satisfies that rule (e.g. for RI either over- or under-excitation For any causal sce- nario Cxy:=(g(Y -X), x. pxx px) excitation (RI) and inhibition (R2) mutually exclusive.
Proof. We have to prove that any Cxy:=(g(Y-+X)) y, uxx pxx) will trigger either RI or R2 in any case but never simultaneously. For this, consider the encoding {< 0, >-> 1} i.e., the relations are mapped to the binary numbers. Given some causal scenario Cxy, for both R1 and R2 we always have evaluation of the triplet T = (R(s(a(Y=)))) 0), R[,, JYY,, R(x, px)). It is is the total number of possible scenarios. where RI covers codewords (010, 011, 100, 101, 000, 111) and R2 covers the codewords (001, 110), and together they cover all codewords [R1| + R2 = [T]. Since any single scenario Cxy is uniquely mapped to a codeword through its triplet, will either trigger R1 R2 but never fall into both in parallel.
easy to see that [ = =23
13

---

Under review as a conference paper at ICLR 2022
A.3 PROOFS FOR THEOREM 2 AND PROPOSITION
2
Thm 2 being our second and main contribution, suggests a construction scheme based on the three base logic rules of arguably axiomatic nature (Prop I for generating human-understandaabe answers to single why-questions (Def[]) The rules are being recursively applied to the given causal effect matrix estimation (Def2), that is structure and parameterization, alongside data, The recursion for computing the SCI J in Eg 6 terminates for any input tuple (Qx,S()), D).
Theorem 2 (Structural Causal Interpretations). Let Qx S(C), and D E Ruxd be a single- why question on variable X, a CEM, and a data matrix respectively. Furthermore, let Di= v (U1...., vn) denote concatenation and R be the rule-set function from Prop Then Structural Causal Interpretation is being defined recursively as
3(0), =
-
A(Z + X),
3(Q,, D))
(6)
2xpa(X)
ZEpa(X)
with the base case being evaluated at the roots of the causal path to X, that is,
E : (2 + ... + X A pa( () = 00 b S(C), D) = 0).
(7)
The recursive algorithm for computing 3(xx, S(C), D) in Eq terminates on any input.
Proof. We have to show that the base case is being reached eventually for any input tuple
(Qx, S(C), D). The second tuple entry (right) in calls J with a query on the parents of a given
node X, that is, Qz, Z E pa(X) and otherwise same arguments. Since any SCM e = (S. P(U)) finite, i.e., the number of structural equations is an integer |S| = d E N, and C implies a DAG G. we are guaranteed to reach root node K for which pa( K) = 0 and since K is an ancestor to X, it X. This in fact is the requirement for reaching the base case.
also holds that K-
Since the CEM sub-structure of the SCM as "summarizes" the given structural equations to their causal effects for each of the edges within the SCM's 's induced graph, we can make the important observation that methods that in fact model the actual SCM (e.g. NCMs as in (Xia et al. [02T] or (Zečević et al. b0) are more powerful than methods that model the sub-structure (CEM).
Proposition 2 (SCM- and FEM-Estinators). Any SCM estimator is CEM estimator.
Proof. Let g: FJ H I represent the (expected) causal effect from j to i described by functional form f E F. A structural equation can be split into its dependency terms, fi (pa,, N,) = (N,) + E Ejepa, (i) such that the matrix M =
is CEM.
A.4 PROOF FOR THEOREM 3
Thm Bis key theoretical result that suggests that any of the existing neural causal induction meth- ods, independent of their assumptions" on the data- generating SCM, are in fact interpretable using the SCI from Thm 2 Thereby. Thm provides completely new perspective on both what (neural) 3
causal induction methods are capable of simply due the fact that they model causal relations and also more importantly how a human inspector can investigate the model's 's understanding of the given data in a humanly understandable manner, since SCI imply a direct translation to natural language (as is being detailled in below's 's section) they also imply transparent communication.
Theorem 3 (Neural Induction Models are Interpretable). Algorithms for (causal) DAG-induction from data (e.g. NT (Zheng el af.) [2088) CGNN (Goudet et al. (018)), DAG-GNN (Yu et al. 2019) NCM Ke et al. 2019)) can generate Structural Causal Interpretations.
Proof. Causal semantics at the level of CEM or above (c.g. SCM, Prop 2 allow by construction for SCI (Thm 2). what is left to show is that the mentioned neural induction methods at least estimate CEM. We start with our lead example method NT. We proceed with proving NT (Zheng et al. 2018) to be interpretable, the other proofs will be analogous. NT estimates a linear SCM whose structural
An example of such assumption might be that the underlying SCM that generates the data to be inducted from is linear (as in NOTEARS (Zheng et al. 2018
14

---

Under review as a conference paper at ICLR 2022
equations are of the form fpax. Nx) w pax +Nx. For CGNN (Goudet et al. 2018 the authors perform a score-based approach i.c., generative neural networks are being fit to the data for each of the pausible DtG-structures that the SCM could take on and the best-performing one is subsequently being selected. Again, since SCM can be converted into CEM (Prope CGNN interpretable via SCL. For DAG-GNN (Yu et al.] 2019), the authors make the same assumption of linear SCM as in (Zhegg et al. (0118) and adapt them to graph neural networks, X = f2(([ A fi (Z)) where X, A, Z are data, graph and normal noise respectively and fa the GNNs. For our final example, the NCM (Ke et al., 2019 Xia et al. (02T), the proof is trivial since the NCM model class a subset of the space of all SCMs and therefore also CEM-estimator (Prop
A.5 INTUITION ON CONJECTURE
While the establishment of SCI (Thm and the consequential insight that any neural induction method is thereby interpretable (Thm stand as separate results, we further considered the ques- tion of whether one could in fact make use of pre-existing SCI for learning. While we don't 't' provide proof, we generally observe empirically that, if a human modeller or any other type of a-priori inductor provides some SCIs, then one can use these SCIs to improve learning. An explanation for the observed improvement might lie in the required consistency of underlying CEM cuu-propertyy of the unobserved SCM). Le., the model needs to infer a structure that can provide for consistent interpretations to subsequent why-questions aligned with the answers provided by the existing in- terpretations (be human or other). Therefore, we conjecture that this is generally the case as our evidence supports belief that the statement is true, even if not proven. Formally, we state:
Conjecture (SCI Regularization). Let be defined as in Defß and let H be the true CEM. Regularizing Lr :- L A. [lZ(XxXxx H, D) 1:)12 with existing and overlapping SCI I leads to a more optimal solution arg minnen L, (D, H) - H* < arg minnen L(D, H) H*
A.6 ADDITIONAL THEORETICAL RESULTS
The following statements are an addition to the theoretical results established in the main paper. In Prop we state that any induction method is in fact a causal method, since there exist infinitely many possible SCMs that could act as the corresponding model. This view is inline with the SCM metric space theorem (Thm []. making an (approximately) right choice of SCM important.
method for mata- inference, M(D) = G. Let C denote the space of all possible SCM. Then it holds that 3 E C: = G.
Proposition 3 (Induction Methods). Let M be a
Proof. A proof by construction is sufficient. Let G (Va, Fa} be DAG (variables, edges) as described in the proposition (M(D)= = G). Then we can construct an SCM e where the structural equations of e use nodes VG to mimic Eg. Naturally, there will be infinitely many such C since parametric form and parameterization can be arbitrary. However, the proposition suggests for the existence of at least one such SCM for any graph, and we are done due to being arbitrary.
Another important observation in Prop 4 follows from the SCI's computation arguments (Thm 2 being the CEM and not SCM (where we have shown that SCM a more expressive model, Prop Le., the function which converts any SCM to their corresponding CEM as defined in Def will not allow for two SCMs coincide, that is, the function is injective (each mapping CEM unique).
Proposition 4 (Injective CEM Mapping). The function that maps SCM to CEM is injective.
Proof. We assume the general case where the fúj components of the CEM are a causal effect from i to j. Let F; : + M denote said function that maps from the set of all possible SCMs to the set of all CEMs. Since CEM is sub-structure of an SCM by definition (Def there is restriction that each edge i + j of a given SCM is being bijectively mapped to a causal effect fij for the CEM. Further, F is restricted to have model each of the structural equations. As Thm always find two SCMs of same size E, C2 E C, for which will hold d(C,, e2)> >00 where dj is the measure from Thm and thereby imply that (F((.) F(C2))>0 where da is matrix norm. n
F
suggests, one can
15

---

Under review as a conference paper at ICLR 2022
A.7 ELABORATION ON SCI PROPERTIES
The three basic logic rules (Prop I dictate how the SCI (2) will look like for some causal estimate of the system and any given query and data. In our implementation, we additionally provide the actual relation R as a return argument of each of the rules. This allows for a more fine-grained interpretation that gives insights on the specific relation to the current explaining reason. In a nutshell, it allows to extend a statement "Y because of X' to a more detailled one like 'Y because of X being low'. The general pronunciation scheme for the the three rules (excitation, inhibition, and preference), that allows for a human-understandabbl natural language version of the SCI, are as follows:
R1
Excitation
Y because of X [being low/high]'
R2
Inhibition
'Y although X [is low/high]" 'mostly" R1 or R2 pronunciation
R3
Preference
Table 3: Pronunciation Scheme. Right shows the natural language reading of a rule's activation.
The pronunciation of the details to the relation e.g. 'low'/'hihh" is context-dependent in that these words might need to replaced with adequatt//corresoodiig words suitable for the context i.e., 'the Matterhom is cold because of the high altitude' ('cold Temperature because of Altitude being high') is fine while 'the remaining car fuel is low because of the bad driving style' ('low Fuel because of Driving Style being bad') requires the context-adaptation ('low' - "had"). Another noteworthy detail to the SCI properties is the property of non-repeating causes within interpretations which reduces redundancy. Consider for instance our lead example on Hans's mobility [Int[]or or Fig 2) the CEM suggests that F can also be explained by A, since A + F. However, the corresponding SCI does not give this reason because of the aforementioned property which ensures that redundancy is being avoided. Lee, in the interpretation step before we actually explain H using both A and F, since F} - H, therefore, making it irrelevant to the query for explaining the relation between the parents (A, F). While we provided intuition on the derivation of the basic rules (Prop[] while alongside also providing lead example Fig we now additionally motivate the namings "excitation', 'inhibition' and 'preference' i.c., why we think they make sense. We took inspiration from neuroscience, where the former two terms relate to the way neurons interface with each other using their synaptic-dendric connections. The last term is term to propose relativeness and thus preference for one cause of over the other. All terms thereby adequately describe any causal path in qualitative terms while also providing an almost synonym-quality to the pronunciations (Tab 3).
A.8 EXP.1: ASKING NEURAL MODELS INTERPRET THE LEARNT (TIM) 3
We select NOTEARS Zheng et al. 2018) as representative data-driven induction method for the illustration in T which considers the data sets and single-why queries illustrated in Fig P i.e., Deutscher Wetterdienst (DW, Mooij et al. 2016). Causal Health (CH, Zežević et al. (2021a) Mileage (M), and Recovery (R,[Charig et al. ( (186)). Fig aadditiiiin visualizes the generated SCI that are in support of ThmBi.e. the interpretability of NIM. The SCI generated using the learned causal semantics are identical for the DW and M data sets, while differing only subtle for R and drastically for CH data sets. The former discrepancy occurs the second-level of reasoning i.e., the right top-level explaining answer is given to the question (i.e., "Kurt did not recover because of the problematic pre-conditions") but was contrasted wrongly (i.e., the treatment countering the state of condition and not affecting the condition). The latter discrepancy revolves around a totally different structure e.g. the learned model expects a direct cause-effect relation between age and mobility while also wrongly assuming that nutrition has detrimental effect on health. An explanation in the case of NOTEARS clearly the violation of the linearity assumption for the CH data set.
While in Thm 3 we prove that NIM are generally interpretable in the sense of SCI (Thm 2 for empirical illustration we also provide more examples of such NIM-based SCI, as we did with our lead examples for NT, in this case additionally for CGNN (Goudet et al.) 2018) and DAG-GNN YYu et al.] 2019). Tab shows an application of these methods to a superset of questions (that is, same and more) and same data we used for NT. It is crucial to note that the presented results have not been hyperparameter-optinne opttiiiiee (HO). Take for example CGNN, where candidate selection is exhaustive (brute force, and thus super-exponentis in the number of nodes) and the model selection heavily relies on the neural approximation. thereby, HO is likely to be important. In a nutshell, the motivation behind Tab is to present support for our theoretical proof on SCI-interpretabiliiyy of NIM
16

---

Under review as a conference paper at ICLR 2022
methods ie., also give empirical proof for several methods in practice (opposed to pure theory). To assess the quality of the SCI, it is important to note the assumptions made by the original method. E.g., NT and DAG-GNN assume linear SCM. Thereby, we have no guarantees for running such method in a non-linear data domain (which we do with the data sets DW and CHD). Interestingly, these assumptions can in fact be exposed by SCI. Consider the DW data set (Tabf first example), theory suggests that a linear model with Gaussian noise will exist in both directions X - Y and Y + X, thus being -on-identifiable (Peters et al. (017)) Methods like NT and DAG-GNN therefore pose the assumption that the given data comes, in this case, from a linear model with Gaussian noise ie.. the identifiability problem is being circumvented altogether. This is also the reason why different random seeds can lead to both modellings (A - T and T - A) for the DW data set (see in Tab how the SCI flips for M3=DAGGGNN for the two opposing DW queries). Another important note is that the uninformed SCIs "No causal interpretation ..." occur when the method's 's CEM estimate does not contain a causal path to the variable that is being queried i.c., the CEM will actually contain a non-trivial estimate of the underlying causal structure, even though the SCI returns trivial/empty interpretation since the variable of interest can not be reached within the estimate's ' structure with a directed path (i.e., the base case in Thm is trivially triggered). In fact, these negative 'no answer'- type of cases are important since the model need also be able to know when there is nothing to be known. For this case, also pose why-questions to which the ground truth is already a 'no answer' interpretation since there is no causal connection to the variable being queried by the why-question. The empiries Tab suggest, as theoretically proven (ThmB). that the NIM are interpretable and also that all 3 rules (excitation, inhibition and preference) are being used for the NIM-based SCI. As a positive example, consider example #3 for the CH data set where M1 captures the complex interpretation correctly up to preference and falsely assuming that nutrition (F) has negative causal effect on health (H)). A more interesting example (#8 for the R data set) shows that the main reason being bad pre-conditions (Z) is being captured but the model falsely assumes that those are because of the received treatment (T). To consider a negative example have a look at example #4 again for CH where the actual answer is a 'no causal interpretation' since age (A) is root node. However, M3 claims that the age is high because of the nutrition and mobility (M), then again because of health. While the statement wrong and also feels exaggerated, inspecting closely one detect the correct existence of the causal edge between mobility and health (H - M). Le., the model interprets wrongly, but its causal model is still partially valid.
A.9 ExP.2: USING INTERPRETATIONS TO IMPROVE LEARNING (COnJ I
We again make use of the NOTEARS Zhhng et al. 2018 method as representative of graph in- duction methods for the subsequent experiment in which investigate Conj[[i.e. whether existing interpretations can be used supervision signal to improve the quality of induction. To circum- vent the -ineeffeerentable nature of our recursive interpreter from Thm @we train neural network on a set of legal interpretations to mimic the interpreter while being fully differentiable. Mathemati- cally, we set I = Dw, in Alg I where Ju is said neural approximation to the SCI recursion from Eg 3 Following (Zheng al. 2018), we generate 70 random linear causal models following Erdos-Rent Renyi structures. We use graph induction to infer 70 more graphs, making 140 in total. For each graph we generate 50 random single-why questions to be answered, resulting in data set of 7000 interpre- tations. All the (discussed) details regarding the learning problem and the results are additionally being illustrated within Fig We extend the NOTEARS loss composition with this neural approxi- mation to the regularization we proposed in Conj [ and perform graph induction once with and once without the regularization (where between I and 50 interpretations are being observed). The graph induction is being performed in a data-scarce setting with only 10 data samples per graph induc- tion. Thus to infer the true causal structure the method ideally needs to perform sample-efficient 3 shows our empirical results on the error distributions for all the graphs while presenting the qualitative difference in the estimated graphs for the most significantly improved example. It can be observed that with the regularization the induction method can both identify more key struc- tures while significantly reducing the number of false links, thereby appearing to be overall more sample-efficiens An explanation would be that, as conjectured, the interpretations contain valuable information about the underlying CEM if the interpretations themselves were generated by a similar CEM, thereby striking structures that would lead contradicting interpretations.
17

---

Under review as a conference paper at ICLR 2022
A.10 ExP.3: HUMAN CASE STUDY, ALGORIHHMIC VERSUS HUMAN INTERPRETATIONS
We instructed N = 22 participants to answer our questionnaire (Fig 7 The questionnaire asked the following questions: Given pair of variables, does causal relationship exist (existence)? If yes, then which is the cause and which is the effect (direction)? If there are multiple causes for a single variable, then how impactful is each of the causes (preference)? All of these questions, alongside their responses, are of qualitative and subjective nature. It is important to note that the participants do not perform the actual induction from specific, provided data like the algorithms do i.e., the hu- man subjects are not given the variable names nor concrete data points that would allow them to find the rules for the specific data sets. Instead, they were only given the variable names/depictions, thereby having to induct from personal experience/lmdeesaaddigg essentially. This approach to hu- man induction is related to the experimental setups in (Griffiths & Tenenbaum 2006 Hattori 2016).
The motivating lead research questions we intended answer, and in fact do answer successfully with this experiment, are: What are CEM (Deff2 that (some) human could model? How does over- lap for human-based CEM occur? How do subsequent SCI (Thm 2 between humans and algorithms differ? In a nutshell, we wanted to investigate the similarity of CEMs between subjects in addition to the similarity between subjects- and algorithm- SCIs.
A caveat regarding the analysis and interpretation of human judgements is that sample bias may distort conclusions. Sample bias has long been identified within the behavioral and social sciences as limiting the generalization of results obtained in a specific sample to the population. A common methodological fix to counteract such biases to increase the sample size, see (Daniel, 2017 for a recent application and discussion. Certainly, the observed sample will affect the way the difference (to e.g. algorithm-based SCT) turns out to be, but then again our research question is not concerned with all possible human interpretations, but any. Furthermore, we chose data sets that model very general examples and thus offer accessibility to the general population since no single person might be an expert. Ultimately, this way of designing our experiment, while not removing sample bias of course, renders the bias's 's qualitative effect onto our subsequent investigation negligible.
In the following we provide discussion of several interesting and important insights discovered through the human user study. Nonetheless, it is important to note that our results like most mod- day interpretations of human behavior are of conjectural nature sensible, educated guesses essentially. During this discussion, we will point to specific aspects of the descriptive statisties displayed in Fig
8 The actual human data is also being appended for the sake of completion (Figs 10 12 13 14). The questionaire contains four examples with two, three, three, and four variables (or concepts) respectively that are being visually depicted in addition
concise textual description. We randomized the textual description of up to three variables across all examples for any randomly selected participant. Doing so, we allow for the randomized concept to reverse causal influence directions, thus, diminishing the bias of chance-selecting said causal direction - in nut- shell, this randomization scheme helps us in controlling for interpretation variance (or leeway) the subjects. Nonetheless, we still observed that for any variable pair (X, Y) the meanings of X and Y themselves could be interpreted differently, which ultimately resulted in False Negatives regarding agreement i.c., people will disagree technically although they actually agree. To give concrete example, consider the following: pre-condition in Example 2 can be interpreted as "the length of the medical history of a patient" (negative; increasing implies lower chance of recovery) opposed to "the state of well-being of a patient" (positive; increasing implies higher chance of recovery), thereby some subjects might choose Z1 + R while others will choose Zz +- R where Z, are the different interpretations of the "pre-condition" concept (and R denotes recovery), yet all subjects agree on an existing relation between the two variables: Z, ++ R. Also, some variables/concepts were more stable in their interpretation variance. To give yet another specific examplc, altitude and temperature in Example (Fig(7 are stable concepts while the aforementioned pre-condition in Example 2 is unstable (due to its interpretation variance/leeway). More importantly these differ- ent interpretations due to the ambiguity inherent in language become visible within the statistics. To stay inline with the previous example, consider the medical example (E2) within Fig (second row, middle) and specifically consider the edges T + R and Z + R. For the former relation the agreement between subjects is evident i..., the majority of human subjects will select this edge. For the latter relation, we clearly see the two previously discussed interpretations that subjects employ during edge decision. Le., for some subjects the edge between Z and R is positive and for some
18

---

Under review as a conference paper at ICLR 2022
others it is negative, while naturally all agree upon there being relation between the variable pair (Z ++ R) opposed to there being no relation (Z 4 R).
We observe a systematic approach and thereby non-random approach to cogee/sttructuressleecee by the human operators, see any of the subplots within Fig | Furthermore, there are only a few clusters even with increasing hypothesis space. Both the systematic manner and the tendency to common ground are evidence in support of the CMMC hypothesis [Hyp[T) and the argument on true causality to be more likely within the overlap of SCMs (Thm D.
Although we randomize the order of variables in addition to consistently presenting them in a simple line with the intention of not inducing any specific sorting/structure avoid bias, still observed apparent, unintended subject behavior. For instance, subject number 5 (Fig I0) only considered pairs presented next to each other being questioned although the other combinations are meant to be queried as well. While additional research needs to corroborate these observations, our data sug- gests that attention might have decreased over the course of the experiment for a subset of subjects as suggested by e.g. subject number 7 (O) where overall agreement with the subject major- ity is to be found but eventually at the very last example 'mistakes' occur (specifically, the subject highlighted that increasing age increases mobility', in stark disagreement with the majority of par- ticipants). We also observe that the increase in hypothesis/search space (i.e., more variables) comes with an increase in variance. TThis variance increase can be argued to be due to the progressive difficulty of inference problems as well as decreased levels of attention and potential fatigue across the duration of the experiment (e.g. consider the duplicate plots, third column, in Fig g where the number of unique structures that are being identified increases significantly) Yet another interesting observation concerns the aspect of time, consider subject number 17 (Fig l3) where there a cycle between treatment and recovery where the subject likely thought in terms of 'increasing treatment increases speed of recovery which subsequently feeds back into a decrease of treatment (since the individual is better off than before)' which seems like a valid inference but clearly considers the arrow of time. Yet another observation, some subjects faced questions of variable scope e.g. if there is a causal connection between nutrition and mobility, then some subjects considered energy as the mediator and since energy is not part of the variable scope, confusion might arise whether to place an edge between nutrition and mobility or not. In fact, for such a scenario the correct answer is to place an edge, since there exists a causal path from nutrition to mobility, via energy, even if energy is not displayed. Le., in causality, edge can/will talk implicitly about all the more fine-grained variables that are part of the causal edge/path.
The second data (E2) is an instance of the famous Kidney Stonde example (Peters al. 2017 where Z is a confounder that indicates the pre-conditions in terms of e.g. the size of the kidney stone, and it also illustrates the famous Simpson paradox (Simpson 1951 Pearl 2009 Peters al. 2017) where the recovery will favor one treatment in the overall statistics while being better for all of the non-consolidated views for the other treatment. We observe that not a single subject places the edge pre-condition to treatment (Z + T)) which is arguably at the core of Simpson's 's paradox. This observation gives an additional cue on why the phenomenon is called paradox because no human subject expects the existence of this connection and even actively neglect the existence.
Technical Details and Code. All experiments are being performed on a MacBook Pro (13- inch, 2020, Four Thunderbolt 3 ports) laptop running 2,3 GHz Quad-Core Intel Core i7 CPU with a 16 GB 3733 MHz LPDDR4X RAM on time scales ranging from a few sec- onds (e.g. evaluating SCI in Exp. up approximately an hour (e.g. SCI-based learn- ing in Exp.2). Our code is available at: httpp:: htttps://anonymouss 4open. science/r/ www.tttaaaa-aasaa-tneeeeeeee
-
Full-width Material. Following this page are the figures and tables (Fig were referenced in the corresponding sections of the appendix.
Tub 4 that
19

---

Under review as a conference paper at ICLR 2022
Structural Causal Model C
D ate
(i)) PN)
Structural Causal Interpretations
Causal Effect Matrix ((C)
(S((), Q, D)
M6 (nim Alb (wann))
"The Natterhorn cold because of the high aititude""
Query / Questies
Mobility (M) because
"Wily Harol' Mob bog?"
although Food Habits
Oei 5 Dee 1-17
"Your millage
the little romaining fucl.
(Erways
47 Wine,
"Kurt did not recover bocause although had surgory."
Figure 5: Extension to Zig In addition to the previous example, we add three more SCMs, both continuous and discrete alongside their why-questions Q and resulting Structural Causal Interpreta- tions ((S()) Q, D). Our lead example concerning Hans's 's mobility is being highlighted alongside the intuitive-level computation. (Best viewed in color.)
20

---

Under review as a conference paper at ICLR 2022
Structural Causal Interpretation (Thm
#1
Dataset: DW, Query: "Why is the temperature at the Matterhorn low?"
GT
"The temperature at the Matterhorn is low because of the high altitude."
M1
"The temperature at the Matterhorn is low because of the high altitude."
M2
"The temperature the Matterhomn low because of the high altitude."
Ma
"No causal interpretation for Matterhorn's is temperature."
#2
Dataset: DW, Query: "Why is the Matterhorn so high?"
GT
"No causal interpretation for Matterhom's is altitude."
M
"No causal interpretation for Matterhom's 's altitude."
M₂
"No causal interpretation for Matterhorn's altitude."
M3
"The altitude of the Matterhorn is high because of the low temperature.
#3
Dataset: CH, Query: "Why is Hans's 's mobility bad?"
GT
"Hans's ' mobility is bad because of his bad health which is mostly due to his high age, although his nutrition good."
M
"Hans's mobility is bad because of his bad health which bad because of high age and mostly due to his good nutrition."
M2
"Hans's mobility bad because of his good nutrition."
M3
"No causal interpretation for Hans's 's bad mobility."
#4
Dataset: CH, Query: "Why is Hans old?"
GT
"No causal interpretation for Hans being old."
M1
"No causal interpretation for Hans being old."
M2 2
"No causal interpretation for Hans being old."
Ma
"Hans is old because of his good nutrition and bad mobility, which is because of his bad health."
#5
Dataset: CH. Query: "Why is Hans's ' nutrition good?"
GT
"Hans's nutrition is good because of being older."
MI
"Hans's nutrition is good because of being older."
M2
"No causal interpretation for Hans's nutrition."
M3
"Hans's 's nutrition is good because of his bad health and mobility."
#6
Dataset: M. Query: "Why is your personal car's left mileage low?"
GT
"Your left mileage is low because of your small car and your bad driving style."
Mi
"Your left milcage is low mostly because of your small car and because of your bad driving style."
M2
"No causal interpretation for the left mileage."
M3
"Your left mileage is low because of your small car and your bad driving style."
#7
Dataset: M, Query: "Why is your personal car small?"
GT
"No causal interpretation for the car size."
M1
"No causal interpretation for the car size."
M2
"Your personal car's size is small because of your good driving style and fuel savings."
Ma
"No causal interpretation for the car size."
#8
Dataset: R, Query: "Why did Kurt not recover?"
GT
"Kurt did mostly not recover because of his had pre-conditions, although he got treatment."
M1
"Kurt did not recover because of his bad pre-conditions which because of the treatment he got."
M₂
"No causal interpretation for Kurt's 's recovery."
M
"No causal interpretation for Kurt's recovery."
#9
Dataset: R, Query: "Why did Kurt get treatment?"'
GT
"Kurt got treatment because of his bad pre-conditions."
M1
"No causal interpretation for Kurt's 's received treatment."
M2
"Kurt got treatment because of his bad pre-conditions."
Ma
"No causal interpretation for Kurt's 's received treatment."
Table 4: More NaM-based SCL. We prove Thm 3 for general NIM while pointing to some example methods from the existing literature on NIM. Here show the results of running the methods
M, 1:TT (Zhnng et al. 2018), 2:CGNN (Goudet et al. 2018), 3:DAG-GNN (Yu et al. 2019) on the four data sets Deutscher Wetterdienst (DW. eet al. (2021a)). Mileage (M), and Recovery (R, Charig et al. (9866) for the respective queries. As suggested, the methods are interpretable and reveal insights onto the learned causal semanties, while varying signficantly in quality in terms of accuracy relative to the ground truth (GT), Independent of accuracy, "No causal interpretation ... occur when the CEM estimate of M; contains no causal path to the queried variable X i.e., pax (1 (supported through GT sparsity). We also show GT interpretations that require a negative 'no answer' response by M,. The corresponding appendix sections cover a detailed elaboration for these NIM-based SCI (Thm
al. (016)), Causal Health (CH, Zečevid
21

---

Under review as a conference paper at ICLR 2022
Goal: the interpretation process (Eqs. 4,5) differentiable such that we can have regulation loss signal
Learning Problem
Target
Input Features
3(S(C)
MLP
Dxc
hi bod Health (1) which atthough bis Feod Habits are greet."
(7000, 1+1+10+10+10-10
(7000, 3*10"10)
Size: 10x10 graphs
Algrment: poss udges
70 Ground Truth Graphs
+70 NOTEARS notey graph estimetes
50 Queries per graph
Le., 777076+77--700
Results
BCE
(IoU)
common in Medical Imaging
MLP Architecture
RaLy
tanh
122
150
300
Figure 6: Differentiable Interpreter. To cope with the probelm of non-differentiaabiliy of the SCI Thm we propose a learning problem in which learn MLP mimic set of existing legal interpretations. The neural network thus approximates the construction of SCI while being fully differentiable. (Best viewed in color.)
22

---

Under review as a conference paper at ICLR 2022
thore cRUSS Instion
wou, then in which DeOLIMAT
relution betwean i and
pos tie rebotnn causes wowsingg Biveraaen
posttve rilatio, cauoss wcreinng Airrraee
there are mutiipl redooons, whot the onder strenghh
2nd noet importent cause
thorcty Use wsed "ronowary
and unaloguee Use requtive
bis drowing redationa hita
and
sorting
Left Manage
Shiving Style
Type
To proude more ciartyy depict the the ouccepts
persona depaeeperrence La, they only ore orange and buu boons.
TASTATE
Orcunal Trulh:
Nuttitoo
Mobility
HANTT
POP
Figure 7: Experiment Setup for the Human Case Study. The participants are being asked two questions: whether there is directed relation between some variable pair A and B, and when there are multiple causes how they behave relatively i.e., the order of strength in relations. We avoid bias drawing relations by randomizing the order and presenting the variables in a sequence, Induction being performed from personal 'data' /experience, rather than by looking matrix of data points. (Best viewed in color.)
23

---

Under review as a conference paper at ICLR 2022
45
-05
-05
Figure 8: Human Data Analysis: Qualitative, Quantitative, and Uniqueness. Statistics collected from the human data (N 22). The denote the four data sets: Deutscher Wetterdienst (DW, Mooij et al. (2016)), Causal Health (CH, ZZěević et al.] (2021a)), Mileage (M), and Recovery (R, Charig
1986)). The columns: qualitative edge distributions that show for each of the different edge type how often it was chosen respectively (left), quantitative edge distribution for each edge where the error bars denote confidence intervals (middle), and the unique structure counts where each bar depicts the frequency of qualitative structure discovered by the human subjects (right). Extensive elaboration on the setup, execution and results of this human study are to be found in the corresponding appendix section. (Best viewed in color.)
24

---

Under review as a conference paper at ICLR 2022
Data Human Subject #1
Data Human Subject #2
Erample
barpx
thads
Stampi
LiVe
Tresk
Recevery
Pro Conditoor
Trestment
Speed of Recovary
Pre Condttion
Fanmple
Dridng Style
Lef Hillager Fuel Remaayyn
Driving Sivle
Car Stoe
Valherddde Euroy Fant..
Eaample
Nutriton
Moailtyy
Heath
TAR-Bon
Mebilly
Heart L
Age
Data Human Subject #
Data Human Subject #4
Euampte
Terspeot
Treetment
Speed Reuwary
Pre-Canditon
Exmmple
Parpe
Left
Drtiing Style
Type
Esample
Dargx
2
3
Natriton
Figure 9: Actual Human Data Anonymized. Subjects 1-4, Answers followed the questionnaire in Fig 7 (Best viewed in color.)
25

---

Under review as a conference paper at ICLR 2022
Data Human Subject #5
Data Human Subject #
Beanle
Exampl
2
Treatmont
Recovery
Pre COndEEAn
Treatment
Speed of Secowery
Pre-Condition
Examdi
csample
Let: Milagg
Lef Managh? Fuel Bervaning
onieng Stye
lustition
MoDiiyy
Hoalh
Age
Nutritian
Moli
Hoath C
Age
Data Human Subject #7
Data Human Subject #8
Tamppenatur
Altiude
Tomperature
Trample
Crample
LMB
Trampl
looge
Driving styte
Left Sermaiiing
Driving Style
sample
Seample
f
Ago
age
Figure 10: Actual Human Data Anonymized. Subjects 5-8. Answers followed the questionnaire in Fig 7 (Best viewed in color.)
26

---

Under review as a conference paper at ICLR 2022
Data Human Subject #9
Data Human Subject # 10
Ceample
barple
1
Treatreank
Speet Raxovary
Phe CarcRiin
Trearment
Spood of Rosowery
Pre Cenditio
Taarple
Left Missge
Driving Style
Car Type
Drivng
1
burple
2
Mutttion
Mesellty
Hoalth
Age
NuoItOn
Haath
Data Human Subject #11
Data Human Subject # #22
Attlude
Temperstuee
Altrade
Temnzerture
Example
Dumpse
2
Faancle
kogg
style
Type
Driving
Puel Remaiiing
Exampple
Dampli
2
Figure 11: Actual Human Data Anonymized. Subjects 9-12. Answers followed the questionnaire in Fig 7 (Best viewed in color.)
27

---

Under review as a conference paper at ICLR 2022
Data Human Subject # 13
Data Human Subject #14
Esample
Attrade
Temperrature
Enarpple
3
Treakment
Speed Reaovrr
Fre Condition
Stample
Erample
Hual Kemoiinng
Left
Driving style
Cer Sizz
rituing
Type
Examplc
Esmmple
ALETTOOn
Mocertyy
Data Human Subject # #15
Data Human Subject # 16
bampt
Srample
Tempenatuure
Altbude
Tompersture
Scamcle
Trastmart
Spest of Recovery
Pho Pro-Znndoe Bp
Treetment
Recovery
Pre-Condiroon
Laft Hi Wilaggy Remaking
Deiving Style
Car SOA
Lefk Milcage
Driving Strk
Car Type
Nulriittn
Messiter
Reasth
NatrrSoo
Mitiliy
SCD
Figure 12: Actual Human Data Anonymized. Subjects 13-16. Answers followed the questionnaire in Fig 7 (Best viewed in color.)
28

---

Under review as a conference paper at ICLR 2022
Data Human Subject # 17
Data Human Subject #88
Esample
bampa
Almhade
Terppo
Erampe
Treetment
Speed Becovery
Fre-Condiiin
Treatment
Pre- Can-Candiion
Example
MOrQE
Let vaesge!
Drivig Style
car Son
Fuel Remeising
Fuel Romaiiina
Eostple
Nutrition
Moti
Health
AgT
Mobility
Health
Data Human Subject # #19
Data Human Subject # #20
Temparaturr
Alfrace
Eumperature
Damdt
tGt
Treatment
Speod of Recovery
Pre: Cardition
Damp
Minaged
Driving Style
Micage
Die Diamae
Murttoon
Hosilly
Featt
Age
Figure 13: Actual Human Data Anonymized. Subjects 17-20. Answers followed the questionnaire in Fig 7 (Best viewed in color.)
29

---

Under review as a conference paper at ICLR 2022
Data Human Subject # #21
Data Human Subject # #22
Tamperanus
Trastmert
Szeed Recevery
Pre-CordRion
Trestment
Speed Boowery
Pre-Cendton
Left Hiloapo/ Bensaiiin
Drivirg Style
Cer Size
Left Misapa! kng
Driing Stole
Car Size
MAF 1ion
Hoalth
Age
Nutriion
Mocelry
Heath
Ago
Figure 14: Actual Human Data Anonymized. Subjects 21 and 22. Answers followed the ques- tionnaire in Fig (Best viewed color.)
30


Reviews:
[{"review_id": "i0u2-qXeBJL", "reviewer": "Reviewer_2Fy5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper combines causality, interpretation, and machine learning in an intriguing way. \n\nLet me first briefly go over the paper in order. Section 3.1 introduces a hypothesis (CMMC) that mental models can be represented as an SCM and Theorem 1 talks about metric space.  Section 3.2 provides an example which overviews data-driven interpretation of why a variable attains a value. Then, Section 3.3 formalizes the notion of why question, first-order logic for interpretations, and causal effect matrix. These definitions lead to SCI theorem, and the authors further made a connection between Neural Induction Models and SCI-interpretability.\n\nTo begin with, the paper’s strong assumptions are not quite well articulated. \nWhile the causal model of reality can be very sophisticated, the assumed SCM is quite simple. In page 2, “product distribution over noise variables” and “independent, exogenous variables” imply that the SCM yields a Markovian model, meaning that we observe all the confounders among the measured variables. This is not typically assumed in causal inference literature where causal effect is trivially identified from observational data. Further, the Theorem 1’s definition of structural equation that f_i can be decomposed into an additive equation with no interaction among parents is very strong. This definition is lurking inside theorem as “i.e.” while this assumption is crucial in understanding the precise condition the later theoretical developments rely on and the limitations of the proposed method. Interpretation provided by edge-specific causal effects can be problematic in most situations (without the additivity assumption). If Y has two parents A and B, it may be possible to see that both A and B have positive effects on Y but they can have jointly a negative effect on Y. The author might want to consider such a combinatorial nature in SCI. \n\nNext, I find the wordings in the paper is a bit broad (“why question”, “interpretation”) which may confuse readers, and some of the definitions (what I perceived) are presented as proposition and theorem.\nThe definition of a single why question seems reasonable but the name doesn’t seem to be specific enough to properly represent what it really is about. Is every why question about the inequality based binary relationship? Similarly, the name for Proposition 1 \"First-Order Logic Rules for Interpretation“ seems a bit broad. Could you please provide any reason why it is a proposition not a definition. (what were claimed and proved?) The authors mentioned that “the three rules in Prop.1 are naturally derived from the principles established in the previous section.“ It would be very helpful if such principles were explicitly mentioned, and how the rules are naturally derived. SCI theorem is recursively applying the three rules in proposition 1 and concatenating the result. Similar to Prop 1, Is there any reason why it is a theorem rather than a definition for what the authors want to do with DAG in a specific way?\n\nI would like to point out a few more things with no particular order.\n\nWhere do you use the defined metric space?\n\nIn the middle of page 3. I couldn't fully follow the logic that “Interestingly, this comparability of SCM …” leads to “This insight tells us …”.\n\nGiven that the model is Markovian, P(Vi|Pa_i\\V_j, V_j)=P(Vi|Pa_i\\V_j, do(V_j)). Then the causal effect in the causal effect matrix is sum_{rest of parents } P(V_i | Pa_i\\V_j, V_j) P(Pa_i\\V_j). Hence, the causal effect can be driven from data (if you have a causal graph). Is it correct? Again, as mentioned earlier, it seems that just using this single-variable specific \"aggregated\" causal effect seems too restrictive. \n\nEq (3) is interesting since explanation (interpretation) often involves “back-tracking”! If we consider the interaction among parents, better back-tracking would be possible.\n\nDef 3 “Family of CEM Estimators“ seems to be one of the crucial definition but the explanation is lacking. You have a (typical) loss function (e.g., negative log likelihood) how good a “causal effect labeled DAG” is. What do you mean by “The family of CEM estimators encompasses any learner …”?\n\nWhy does Theorem 3 exclude any other conditional independence based or score-based causal discovery algorithm which would generate interpretable DAGs (up to its Markov equivalence in some cases) but only NIM-DAGs? Is this because this paper has very restricted assumptions on what an underlying SCM is (linear and Markovian)?\n\nThe algorithm 1 needs to be edited.\nInput: Can you share some concrete examples of bold I?\nLine 2, “while” is “for each I_i in I ? (since no i++)\nLine 4, how I_i itself returns two objects without calling any function on it?\nLine 5 returning I hat? \nLine 6 dangling closing parenthesis?\nLine 7. Is the updated parameter reused in Line 3 within the loop?\n\nTypos\n- In Figure 3. being being \n- age’s causal effect wages in more than that of nutrition → weighs in more?\n\nIt would be desired to have some discussion about the “probability of causation” in causal inference and its relevance to SCI / rule set. These are very related.\n\n===\nI updated my score to an accept. I find this paper contains several refreshing and interesting ideas, and the linear and Markovianity assumptions are reasonable for the sake of SCI. Given that the paper provides an interesting idea (which although I feel a little bit preliminary), I wish more researchers read, discuss the paper, and work on making the idea more concrete (e.g., connecting to path-specific effect and counterfactuals).\n\n\n\n", "summary_of_the_paper": "This paper investigates how interpretation arises within a DAG structure learned from data. The authors proposed Structural Causal Interpretation (SCI) which provides an explanation (interpretation) given data and a why question based on an assumed structural causal model. Under this definition of “interpretation” or “interpretability”, neural-network induced DAG models are considered interpretable. The authors demonstrated the usefulness of SCI.", "summary_of_the_review": "While I liked the overall idea of interpretation (which I still think about explanation) of a certain variable having a certain value through causal mechanisms, the paper seems over-promising what interpretation is about with a restricted, simplified causal model.\n===after discussion\nI liked the overall idea of interpretation (which I still think about explanation) of a certain variable having a certain value through causal mechanisms. Some of the assumptions in the paper seemed too restrict at first, but I later found reasonable for the sake of generating interpretation.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "cqf8azvh0NN", "reviewer": "Reviewer_bMJP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper generally has a very relevant topic of generating interpretations from DAG models learned from data. It makes interesting\nobservations, drawing connections between causal inference literature, more traditional DAG learning, and mental models. \n\nTechnically, I thought the paper was difficult to follow, in particular in the crucial parts. For example:\n- Proposition 1 is very confusing to me. The proposition introduces rules, but these actually seem three properties on a pair R1 and\nR2. Where do R1 and R2 come from? Similarly, a ruleset function has a range of {-1,0,1} but I do not see how it relates to the said \nproperties.\n- The main theorem (Theorem 2) I also found confusing. Maybe because of the same reason as in Proposition 1: it mainly seems to define a structural causal interpretation. The actual claim about this definition is quite obvious (i.e., that it 'terminates' if you define it as \na recursive algorithm) if you assume that the pa() function is derived from a DAG.\n- Theorem 3: again, I am not sure what is being proved here (and the proof itself does not help much). The only thing that seems to\nhappen is that it is observed that these algorithms learn SCMs, and therefore a CEM, and therefore can generate an interpretation.\nThe relevance that some of these models are 'neural' is unclear to me.\nI also strongly disagree with the \"big picture view\" that is given. It is claimed that \"any induction performed on the restricted hypothesis space of DAGs is in fact a causal induction\". It is well-known that Bayesian network learning algorithm do not necessarily learn causal models. If I understood the paper correctly, then the idea is that you simply learned a \"different\" causal model compared to the ground truth. In my opinion, this only obfuscates the matter at hand.\n\nIn my opinion, the main contribution of this paper is the approach to generate structural causal interpretations from CEMs. This is not completely new however, as there is quite some related work on generating explanations from Bayesian networks. In my opinion, it would\nhave been more suitable to compare to these type of methods. \n", "summary_of_the_paper": "This paper introduces so-called structural causal interpretations. Furthermore, using these interpretations, the authors aim to show that any neural induction method is interpretable. These ideas are illustrated in several experiments.\n", "summary_of_the_review": "In my opinion, the contribution is mainly in the definition of structural causal interpretations and how they relate to existing representations. For the moment, I think the contribution should be presented much more clearly in a major revision. I believe that only after this the significance of the results can be assessed much clearly. As a result, I do not recommend acceptance at this point.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U1XUIkZCiLY", "reviewer": "Reviewer_T5c7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The topic studied by the paper is important and novel. It sounds very smart to me to link the two problems (automatic interpretation generation from causal models and improving learning of the causal models using existing interpretations) and study them together.\n2. The complete set of theory developed.\n3. The findings, especially (1) any neural induction method (for causality learning) is interpretable; (2) SCIs (even if they only approximate interpretations) can improve the learning quality of the neural induction methods.\n3. Excellent use of examples for illustration.\n\nWeakness:\n1. A main concern I have on the paper is the clarity of its presentation. Although the logic is rigorous, the writing of the paper is not so easy to follow. In many places, the authors have assumed that readers know the meaning of a term used or the purpose of a statement, which leaves readers a lot of guesswork; and some statements are misleading. Some examples: \n--Introduction: the (important) term \"causal induction method\" is used without being introduced.\n--Introduction: It is claimed that this paper presents \"the first work on causal interpretation\" - this is misleading/confusing, since there has been work on providing causal interpretations, e.g. on a decision or prediction by a classifier.\n--Theorem 1: what does \"n-SCMs\" mean? why \"d\" is defined so?\n--Proposition 1: in R1\\noteq R2, what are R1 and R2 respectively?\n--Table 1: what does ' ' in the diagrams stand for?\n--Figure 3: What does each subfigure stand for exactly?\n--The 2nd paragraph of Section 4: What are the \"algorithmic\" interpretations?\nAdditionally, it may help if in the Introduction, authors could state explicitly the difference between this work and the current focus of the research in \"interpretable machine learning\", where the goal is to explain a prediction model or the prediction of an instance made by a model.\n\n2. Related work is not complete or lack necessary details. Specifically\n-- There has been quite a lot work on incorporating prior knowledge in Bayesian network learning. This should be mentioned as part of related work, since SCIs obtained from human knowledge (or mental process) can be considered as prior knowledge too.\n-- Is there any link between SCI and the work presented in the following paper?\nNielsen, U. H., Pellet, J. P., & Elisseeff, A. (2008, July). Explanation trees for causal Bayesian networks. In Proceedings of the Twenty-Fourth Conference on Uncertainty in Artificial Intelligence (pp. 427-434).\n-- At the end of Section 2 (page 3), the authors mentioned the work in (Stammer et., 2021). More details should be provided on (Stammer et., 2021) and how it is related to and different from the work in this current paper.\n\nExtra/minor comments:\n1. There are some English errors in the paper, e.g. \n-- missing articles\n-- Page 2, the sentence \"An SCM C induces a DAG G with edges ....\" is not grammatically correct.\n-- Page 4, caption of Figure 1, \"... than an average workout has positive)\" - please check this sentence. Some grammar problem too.\n2. Throughout the paper, the opening double quotation marks `` are not shown correctly. ", "summary_of_the_paper": "This paper proposes a new concept named structural causal interpretation (SCI) and the theory supporting SCI and its applications. Several contributions are made by the paper, including:\n(1) The theory and method for automatic generation of human understandable interpretations (i.e. SCIs) from causal models, including deep causal induction models.\n(2) The theory and algorithm for using existing SCIs (e.g. obtained from human) to improve the learning of causality by deep causal induction models.\n(3) Experiments and the small human study for illustrating the theory and methods.", "summary_of_the_review": "The research topic is important and new. The theory and method developed looks sound to me, and the findings are interesting and could have potential use.  However, as mentioned above, the readability of the paper should be improved, and more related work or details should be provided.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper contains *fresh* new ideas connecting mental models and SCMs and providing interpretations (explanations) from DAG models learned from data, including those learned by using deep learning. The usefulness of the theory is illustrated with experiments. The paper contributes some theoretical results, but the presentation has serious issues. In general, the reviewers found the paper hard to follow due to a lack of clarity in some notations, definitions, and assumptions. 

The paper was discussed in-depth and at length, including the reviewers, the AC, and the senior AC. After all, the gap between the current writing and what is expected from the camera-ready is a bit too large, and we feel it could be a disservice to the authors and community to have the paper accepted in its current form, without passing through another round of reviews. Unfortunately, we do not have any version of "conditional acceptance." 

Having said that, we feel the paper has the potential for having a significant impact, and we appreciate the novelty of the proposed approach and the connection among different fields. To avoid issues in the future, we would like to suggest the authors pay attention to the detailed feedback provided by the reviewers, including the discussion and the conversation with the AC, following the exchange on Nov/28. Some examples of points that could make the presentation clearer include 1) clarifying the contributions and providing more examples of the theoretical results, 2) making explicit that the results work for Markovian and additivity models, and 3) perhaps changing the title accordingly.
Decision: Reject