Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new concept named structural causal interpretation (SCI) and the theory supporting SCI and its applications. Several contributions are made by the paper, including:
(1) The theory and method for automatic generation of human understandable interpretations (i.e. SCIs) from causal models, including deep causal induction models.
(2) The theory and algorithm for using existing SCIs (e.g. obtained from human) to improve the learning of causality by deep causal induction models.
(3) Experiments and the small human study for illustrating the theory and methods.
main_review: Strengths:
1. The topic studied by the paper is important and novel. It sounds very smart to me to link the two problems (automatic interpretation generation from causal models and improving learning of the causal models using existing interpretations) and study them together.
2. The complete set of theory developed.
3. The findings, especially (1) any neural induction method (for causality learning) is interpretable; (2) SCIs (even if they only approximate interpretations) can improve the learning quality of the neural induction methods.
3. Excellent use of examples for illustration.

Weakness:
1. A main concern I have on the paper is the clarity of its presentation. Although the logic is rigorous, the writing of the paper is not so easy to follow. In many places, the authors have assumed that readers know the meaning of a term used or the purpose of a statement, which leaves readers a lot of guesswork; and some statements are misleading. Some examples: 
--Introduction: the (important) term "causal induction method" is used without being introduced.
--Introduction: It is claimed that this paper presents "the first work on causal interpretation" - this is misleading/confusing, since there has been work on providing causal interpretations, e.g. on a decision or prediction by a classifier.
--Theorem 1: what does "n-SCMs" mean? why "d" is defined so?
--Proposition 1: in R1\noteq R2, what are R1 and R2 respectively?
--Table 1: what does ' ' in the diagrams stand for?
--Figure 3: What does each subfigure stand for exactly?
--The 2nd paragraph of Section 4: What are the "algorithmic" interpretations?
Additionally, it may help if in the Introduction, authors could state explicitly the difference between this work and the current focus of the research in "interpretable machine learning", where the goal is to explain a prediction model or the prediction of an instance made by a model.

2. Related work is not complete or lack necessary details. Specifically
-- There has been quite a lot work on incorporating prior knowledge in Bayesian network learning. This should be mentioned as part of related work, since SCIs obtained from human knowledge (or mental process) can be considered as prior knowledge too.
-- Is there any link between SCI and the work presented in the following paper?
Nielsen, U. H., Pellet, J. P., & Elisseeff, A. (2008, July). Explanation trees for causal Bayesian networks. In Proceedings of the Twenty-Fourth Conference on Uncertainty in Artificial Intelligence (pp. 427-434).
-- At the end of Section 2 (page 3), the authors mentioned the work in (Stammer et., 2021). More details should be provided on (Stammer et., 2021) and how it is related to and different from the work in this current paper.

Extra/minor comments:
1. There are some English errors in the paper, e.g. 
-- missing articles
-- Page 2, the sentence "An SCM C induces a DAG G with edges ...." is not grammatically correct.
-- Page 4, caption of Figure 1, "... than an average workout has positive)" - please check this sentence. Some grammar problem too.
2. Throughout the paper, the opening double quotation marks `` are not shown correctly. 
summary_of_the_review: The research topic is important and new. The theory and method developed looks sound to me, and the findings are interesting and could have potential use.  However, as mentioned above, the readability of the paper should be improved, and more related work or details should be provided.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces so-called structural causal interpretations. Furthermore, using these interpretations, the authors aim to show that any neural induction method is interpretable. These ideas are illustrated in several experiments.

main_review: The paper generally has a very relevant topic of generating interpretations from DAG models learned from data. It makes interesting
observations, drawing connections between causal inference literature, more traditional DAG learning, and mental models. 

Technically, I thought the paper was difficult to follow, in particular in the crucial parts. For example:
- Proposition 1 is very confusing to me. The proposition introduces rules, but these actually seem three properties on a pair R1 and
R2. Where do R1 and R2 come from? Similarly, a ruleset function has a range of {-1,0,1} but I do not see how it relates to the said 
properties.
- The main theorem (Theorem 2) I also found confusing. Maybe because of the same reason as in Proposition 1: it mainly seems to define a structural causal interpretation. The actual claim about this definition is quite obvious (i.e., that it 'terminates' if you define it as 
a recursive algorithm) if you assume that the pa() function is derived from a DAG.
- Theorem 3: again, I am not sure what is being proved here (and the proof itself does not help much). The only thing that seems to
happen is that it is observed that these algorithms learn SCMs, and therefore a CEM, and therefore can generate an interpretation.
The relevance that some of these models are 'neural' is unclear to me.
I also strongly disagree with the "big picture view" that is given. It is claimed that "any induction performed on the restricted hypothesis space of DAGs is in fact a causal induction". It is well-known that Bayesian network learning algorithm do not necessarily learn causal models. If I understood the paper correctly, then the idea is that you simply learned a "different" causal model compared to the ground truth. In my opinion, this only obfuscates the matter at hand.

In my opinion, the main contribution of this paper is the approach to generate structural causal interpretations from CEMs. This is not completely new however, as there is quite some related work on generating explanations from Bayesian networks. In my opinion, it would
have been more suitable to compare to these type of methods. 

summary_of_the_review: In my opinion, the contribution is mainly in the definition of structural causal interpretations and how they relate to existing representations. For the moment, I think the contribution should be presented much more clearly in a major revision. I believe that only after this the significance of the results can be assessed much clearly. As a result, I do not recommend acceptance at this point.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates how interpretation arises within a DAG structure learned from data. The authors proposed Structural Causal Interpretation (SCI) which provides an explanation (interpretation) given data and a why question based on an assumed structural causal model. Under this definition of “interpretation” or “interpretability”, neural-network induced DAG models are considered interpretable. The authors demonstrated the usefulness of SCI.
main_review: This paper combines causality, interpretation, and machine learning in an intriguing way. 

Let me first briefly go over the paper in order. Section 3.1 introduces a hypothesis (CMMC) that mental models can be represented as an SCM and Theorem 1 talks about metric space.  Section 3.2 provides an example which overviews data-driven interpretation of why a variable attains a value. Then, Section 3.3 formalizes the notion of why question, first-order logic for interpretations, and causal effect matrix. These definitions lead to SCI theorem, and the authors further made a connection between Neural Induction Models and SCI-interpretability.

To begin with, the paper’s strong assumptions are not quite well articulated. 
While the causal model of reality can be very sophisticated, the assumed SCM is quite simple. In page 2, “product distribution over noise variables” and “independent, exogenous variables” imply that the SCM yields a Markovian model, meaning that we observe all the confounders among the measured variables. This is not typically assumed in causal inference literature where causal effect is trivially identified from observational data. Further, the Theorem 1’s definition of structural equation that f_i can be decomposed into an additive equation with no interaction among parents is very strong. This definition is lurking inside theorem as “i.e.” while this assumption is crucial in understanding the precise condition the later theoretical developments rely on and the limitations of the proposed method. Interpretation provided by edge-specific causal effects can be problematic in most situations (without the additivity assumption). If Y has two parents A and B, it may be possible to see that both A and B have positive effects on Y but they can have jointly a negative effect on Y. The author might want to consider such a combinatorial nature in SCI. 

Next, I find the wordings in the paper is a bit broad (“why question”, “interpretation”) which may confuse readers, and some of the definitions (what I perceived) are presented as proposition and theorem.
The definition of a single why question seems reasonable but the name doesn’t seem to be specific enough to properly represent what it really is about. Is every why question about the inequality based binary relationship? Similarly, the name for Proposition 1 "First-Order Logic Rules for Interpretation“ seems a bit broad. Could you please provide any reason why it is a proposition not a definition. (what were claimed and proved?) The authors mentioned that “the three rules in Prop.1 are naturally derived from the principles established in the previous section.“ It would be very helpful if such principles were explicitly mentioned, and how the rules are naturally derived. SCI theorem is recursively applying the three rules in proposition 1 and concatenating the result. Similar to Prop 1, Is there any reason why it is a theorem rather than a definition for what the authors want to do with DAG in a specific way?

I would like to point out a few more things with no particular order.

Where do you use the defined metric space?

In the middle of page 3. I couldn't fully follow the logic that “Interestingly, this comparability of SCM …” leads to “This insight tells us …”.

Given that the model is Markovian, P(Vi|Pa_i\V_j, V_j)=P(Vi|Pa_i\V_j, do(V_j)). Then the causal effect in the causal effect matrix is sum_{rest of parents } P(V_i | Pa_i\V_j, V_j) P(Pa_i\V_j). Hence, the causal effect can be driven from data (if you have a causal graph). Is it correct? Again, as mentioned earlier, it seems that just using this single-variable specific "aggregated" causal effect seems too restrictive. 

Eq (3) is interesting since explanation (interpretation) often involves “back-tracking”! If we consider the interaction among parents, better back-tracking would be possible.

Def 3 “Family of CEM Estimators“ seems to be one of the crucial definition but the explanation is lacking. You have a (typical) loss function (e.g., negative log likelihood) how good a “causal effect labeled DAG” is. What do you mean by “The family of CEM estimators encompasses any learner …”?

Why does Theorem 3 exclude any other conditional independence based or score-based causal discovery algorithm which would generate interpretable DAGs (up to its Markov equivalence in some cases) but only NIM-DAGs? Is this because this paper has very restricted assumptions on what an underlying SCM is (linear and Markovian)?

The algorithm 1 needs to be edited.
Input: Can you share some concrete examples of bold I?
Line 2, “while” is “for each I_i in I ? (since no i++)
Line 4, how I_i itself returns two objects without calling any function on it?
Line 5 returning I hat? 
Line 6 dangling closing parenthesis?
Line 7. Is the updated parameter reused in Line 3 within the loop?

Typos
- In Figure 3. being being 
- age’s causal effect wages in more than that of nutrition → weighs in more?

It would be desired to have some discussion about the “probability of causation” in causal inference and its relevance to SCI / rule set. These are very related.

===
I updated my score to an accept. I find this paper contains several refreshing and interesting ideas, and the linear and Markovianity assumptions are reasonable for the sake of SCI. Given that the paper provides an interesting idea (which although I feel a little bit preliminary), I wish more researchers read, discuss the paper, and work on making the idea more concrete (e.g., connecting to path-specific effect and counterfactuals).




summary_of_the_review: While I liked the overall idea of interpretation (which I still think about explanation) of a certain variable having a certain value through causal mechanisms, the paper seems over-promising what interpretation is about with a restricted, simplified causal model.
===after discussion
I liked the overall idea of interpretation (which I still think about explanation) of a certain variable having a certain value through causal mechanisms. Some of the assumptions in the paper seemed too restrict at first, but I later found reasonable for the sake of generating interpretation.
review_title: null

Meta-review: The paper contains *fresh* new ideas connecting mental models and SCMs and providing interpretations (explanations) from DAG models learned from data, including those learned by using deep learning. The usefulness of the theory is illustrated with experiments. The paper contributes some theoretical results, but the presentation has serious issues. In general, the reviewers found the paper hard to follow due to a lack of clarity in some notations, definitions, and assumptions. 

The paper was discussed in-depth and at length, including the reviewers, the AC, and the senior AC. After all, the gap between the current writing and what is expected from the camera-ready is a bit too large, and we feel it could be a disservice to the authors and community to have the paper accepted in its current form, without passing through another round of reviews. Unfortunately, we do not have any version of "conditional acceptance." 

Having said that, we feel the paper has the potential for having a significant impact, and we appreciate the novelty of the proposed approach and the connection among different fields. To avoid issues in the future, we would like to suggest the authors pay attention to the detailed feedback provided by the reviewers, including the discussion and the conversation with the AC, following the exchange on Nov/28. Some examples of points that could make the presentation clearer include 1) clarifying the contributions and providing more examples of the theoretical results, 2) making explicit that the results work for Markovian and additivity models, and 3) perhaps changing the title accordingly.
Decision: Reject