Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors list 5 technical contributions:
1. the structural causal explanations algorithm - "a new algorithm (SCE) for computing explanations from SCM making them truly causal explanations by construction"
2. an illustration of "how SCE fixes several of the shortcomings of previous explainers"
3. an application of "SCE algorithm to several popular causal inference methods" (I assume this is referring to the statement in the abstract, "Since SCEs make use of structural information, any causal graph learner can now provide human-readable explanations.")
4. a discussion "using a synthetic toy data set [of] how one could use SCE for improving model learning"
5. "a survey with 22 participants to investigate the difference between user and algorithmic SCE" (I assume this is referring to the statement in the abstract, "We conduct several experiments including a user study with 22 participants to investigate the virtue of SCE as causal explanations of SCMs.")

The authors also make the following claims in their abstract and introduction (and in the main body of the paper), which appear to be distinct from the claims listed in their technical contributions section:

6. SCE is a step forwards in terms of (causal) explanatory interactive learning (XIL). "In explanatory interactive learning (XIL) the user queries the learner, then the learner explains its answer to the user and finally the loop repeats. [...] Thus as a step towards causal XIL, we propose a solution to the lack of causal explanations."
7. CXPlain does not output "truly causal" explanations. "[...] we propose a solution to the lack of causal explanations. Specifically, we use the popular, proclaimed causal explanation method CXPlain to illustrate how the generated explanations leave open the question of truly causal explanations. [...] as we show in this work [CXPlain] still leave[s] open the question of truly causal explanations" 
8. The explanations generated by SCE are human-understandable because they can be expressed using natural language. "We provide a new, natural language expressible (thus, human understandable) explanation algorithm with SCE."
9. "Throughout this paper we provided several arguments in advocacy of Causal XIL as the key paradigm of interest for future research and application."
10. By using SCE, any causal graph learner can provide human-readable explanations on any query of interest, and this was not possible before. "Since SCEs make use of structural information, any causal graph learner can now provide human-readable explanations. [...] [Theorem 1] tells us that any causal graph learner ever invented and that will ever be invented can provide causal explanations on any query of interest consistent with the learned model thus reflecting the learnt. [sic]"
strength_and_weaknesses: # Strengths

The paper's topic is very relevant to the field of explainable AI and the particular subfield of explanatory interactive learning (XIL).

The concept of an algorithm which recursively traces the ancestors of a variable which the algorithm is explaining to construct an explanation, as in Definition 4, is interesting and novel to my knowledge. The concept exists in Hall's notion of production but it hasn't been used to algorithmically construct explanations to my knowledge.

# Weaknesses

Overall, it seems that very strong claims are made in the introduction and abstract, and the claims are significantly weakened when the technical contributions are listed. Because these strong claims are not supported by the paper, they should not be present. I address these claims first, as they seem to be the strongest.

6. **SCE leads to improvements in (causal) XIL.** XIL is mentioned heavily in the abstract/introduction and in the discussion of Figure 1, which illustrates SCE being used in an XIL setup (a setup where users repeatedly query a system for explanations to better understand it). However, it is not shown in the paper that SCE actually leads to a better user experience in XIL. XIL and the setup in Figure 1 is not mentioned again in the body of the paper until section 4.1, where the authors argue CXPLain (the prior method) has shortcomings with respect to SCE, and in section 4.4, where the authors attempt to answer the question "What does SCE explain about the causal intuition that humans have that could provide for Causal XIL?". I do not see any mention of XIL in the arguments for SCE over CXPlain in section 4.1 or the analysis of the study in section 4.4. A human study would be useful to demonstrate this is the case.
7. **CXPlain does not output "truly causal" explanations.** Already, "truly causal" is a vague term which is not defined in the paper. Section 4.1 argues that SCE is superior to CXPlain because it provides additional information, but it is not shown that CXPlain's output is *wrong* or *not* "truly causal". In the example given (Figure 2), it seems reasonable to me to state that Hans' age is the main contributing factor to his (lack of) mobility, the explanation CXPlain seems to give, and the paper doesn't explain why this might be wrong. Thus, the claim that CXPlain does not output "truly causal" explanations doesn't seem to be supported in the paper.
8. **SCE explanations are human-understandable because they can be expressed using natural language.** First, I examine the claim that SCE explanations can be expressed using natural language. It seems that this is true for rules 1 and 2 (excitation and inhibition). However, rule 3 doesn't actually seem to correspond to the phrase "Y is *mostly* because of [...]"; it seems that it would be more accurate to say "[...] contributes the most to Y", and this may not constitute an explanation. For instance, it seems wrong to say "The amount of money a charity received was high mostly because of Alfred, who donated 2 dollars" while 500 other people each donated 1 dollar. So I conclude that SCE explanations can indeed be expressed using natural language, but these expressions may be misleading and may not align with what we intuitively consider an explanation. Further discussion of this point in the paper is warranted. Next, I examine the claim that SCE explanations are human-understandable due to their ability to be expressed using natural language. I could not find any content in the paper supporting this claim (including the human study in section 4.4, which did not show SCE explanations to humans); a human study should be used to support this claim.
9. **The paper argues in advocacy of Causal XIL.** I am unable to find any such arguments after reading the paper and searching for the term "XIL" in the paper in or before section 4.4, where the claim is made.
10. **Any causal graph learner can now provide human-readable explanations on any query of interest.** First, I examine the claim that this was not possible before (implied by the usage of the word "now"). I agree that causal graph learners' purpose is not to provide explanations. Next, I examine the claim that this is possible now. First, as discussed in point (8), there are issues with claiming that the generated explanations are now human-readable due to misalignment with human intuition. Second, it is not shown that explanations on larger graphs (for instance, 50-100 nodes) are human-readable. Third, causal graphs learners generally do not output a single graph but an equivalence class of graphs, and SCE doesn't operate on equivalence classes of graphs. Fourth, SCE is limited to queries comparing variables' values to their means (Def 1); this seems far from "any query of interest". Thus, there is still a large gap between causal discovery algorithms and human-readable explanations on any query of interest, which SCE does not bridge.

Next, I address the listed technical contributions. Overall, contributions 3-5 are not clearly stated, and their significance is unclear even with the context of the introduction and abstract. Contribution 1 requires more engagement with the prior literature on explanations and a clear statement + discussion of its limitations and assumptions. Contribution 2 relies on two qualitative arguments in section 4.1 which I do not think are sound.

1. **The structural causal explanations (SCE) algorithm + the claim that SCE explanations are "truly causal"** Regarding the construction of the SCE framework, there seems to be very little justification for Definitions 1-3 when taking into account prior literature.

    1.1. Regarding Def 1, why are "Why" questions constrained to variables' relationships with their means (Def 1)? Why aren't all kinds of why questions allowed, as in (Halpern, 2016)? It is stated after Explanation 1 that our "Why" questions are asked about *individuals*; if this is the case, are the unobserved variables in the SCM fixed?

    1.2. Regarding Def 2, how is each causal effect $\alpha_{X\to Y}$ specifically defined? Is this a unit-level causal effect specific to a setting of unobserved variables $\mathbf U$, or an average causal effect (a.k.a. average treatment effect)? If it is a unit-level causal effect, how is this supposed to be obtained, given that it is not possible to observe each variable's counterfactuals without the SCM? Is the assumption that we have the full SCM and the specific unobserved variable setting? If so, this should be stated explicitly in the contributions and the abstract, as it is even stronger than the assumption of just knowing the full SCM. And is the full SCM required? Or just the causal graph? If so, a causal graph is not sufficient to deduce all interventional effects from observational data, unless the graph is Markovian; if Markovianity is assumed, it should be explicitly stated as an assumption in the abstract/contribution section.

    1.3. Regarding Def 3., what is the intuitive justification for each rule? Examples in cases of linear SCMs are given. As mentioned in my comments on claim (8), we can obtain unintuitive behavior with rule 3 even in linear cases. I think we can also obtain unintuitive behavior with rules 1 and 2 if the causal effect of increasing & decreasing is on average 0, but there are specific regions of increase/decrease where there is an effect of increasing/decreasing the variable (e.g., a sine curve). How does this change the justification for each rule? If the assumption is that each function is linear and/or monotonic, this should be stated explicitly in the contributions and the abstract; in addition, justification for each rule should be given, as well as discussion of the rules' limitations.

    1.4. Regarding Def 4., it seems like a causal scenario $(\alpha_{X' \to Y'}, x', y', \mu_{x'}, \mu_{y'})$ is required for *every* pair of variables among the ancestors of the explained variable $X$. Is this correct? This isn't made clear in Def 2, and the causal scenario is referred to as a "single tuple", which makes it seem that there is only one causal scenario inputted to the SCE algorithm. In addition, why is the dataset $\mathbf D$ required for the SCE algorithm if the causal effects $\alpha_{Z \to X}$ are already known from the causal scenario? Is there some additional step of transforming a dataset to a causal effect which is not included among the definitions? And from what distribution (observational, interventional, etc.) is the data in the dataset drawn?

    1.5. Finally, it's not clear what "truly causal" means (as opposed to "causal" or "purportedly causal"), as this is not defined in the paper. The paper argues that SCE's outputs are "truly causal" by construction; it is not clear to me why this is the case.

2. **SCE fixes several of the shortcomings of previous explainers.** Section 4.1 argues that SCE is superior to CXPlain because 1) some aspects of the causal graphical structure (variables with direct vs indirect effects) cannot be distinguished using CXPlain's outputs and 2) CXPlain's explanation does not give information on how changing each input variable will change the outcome.

    2.1. The observation seems to be true, as CXPlain does not output any kind of recursive or graphical structure. But why is this a shortcoming? Is the explanation given by CXPlain wrong? Can't users look at the causal graph (which is required as an input to SCE) to obtain this information?

    2.2. This seems to be a valid point about CXPlain. But is it true that SCE does give this information in general? It is certainly true in linear cases, where the causal effect can be modelled with a single value. What about in non-linear cases, such as the sine wave case mentioned in my comments in (1.3)?

3. This seems to be an action the authors took rather than a result. Why are the applications mentioned in (3) novel/significant? 

4. **Improving model learning using SCE.** It's not quite clear what "model learning" means here; this point should be clarified. If this refers to section 4.3, then there are some points to clarify. What is the exact definition of SCE regularization? How is error computed? Is the difference in Fig 3 statistically significant? Doesn't SCE already take in the true causal graph, so isn't including an SCE-based regularization equivalent to passing the graph learning method additional information on the graph (this of course depends on what the definition of SCE regularization is)? All of these question would need to be answered to assess the benefit of SCE regularization to causal discovery.

5. This seems to be an action the authors took rather than a result. Do the results of the study mentioned in (5) study support any novel/significant claims about SCE?

Actionable feedback:

1. Either remove claims 6-10 from the paper or include them in the listed contributions section and include supporting evidence for them in the body of the paper as suggested above. For instance, XIL seems interesting, but the connection to SCE doesn't seem to be made; it would strengthen the paper to make this connection. In general, please do not include strong claims in the introduction/abstract which are then weakened in the listed contributions section.
2. Remove claims 3-5 from the paper, as they do not seem to be contributions in themselves, and optionally replace them with novel/noteworthy conclusions from the listed application, discussion, and survey. Claim 5 could possibly be replaced with "analysis of a user survey of 22 participants on 4 explanation settings that explores challenges for explanation methods to tackle in the future".
3. Regarding claim 2, highlight specific cases where CXPlain fails at its stated goal. For instance, construct two examples where CXPlain outputs the same set of scores but SCE outputs different explanations. Conduct a human study to show that humans find explanations generated by SCE more useful for performing a task (or higher-rated according to some other criteria) than explanations generated by CXPlain.
4. Regarding claim 1, connect with the literature in greater depth to clearly state the assumptions/limitations of Definitions 1-3 in the abstract and contributions sections (or to change Def. 1-3 to encompass explanations as explored in prior work); some pointers include Halpern's approach to defining explanations and causation, which is cited in the paper (Halpern, 2016), and (MIller, 2017 - https://arxiv.org/abs/1706.07269). Clarify answers to the questions above in 1.1-1.4. Remove usage of the phrase "truly causal" or define it specifically.
clarity,_quality,_novelty_and_reproducibility: Clarity: As mentioned above, stronger contributions are listed in the introduction and abstract than in the technical contributions section after "we make several contributions:", so it is hard to determine exactly what the claimed contributions are. Furthermore, the listed contributions are vague; for instance, "(III) we apply the SCE algorithm [...]", "(IV) we discuss [...] how [...] [to] use SCE for improving model learning", and "(V) we perform a survey [...] to investigate the difference between user and algorithmic SCE" do not seem to be results in themselves and require the reader to guess what the authors' actions are demonstrating based on what has been stated in the abstract/introduction. The notation in Def 3 is unclear and hard to read. What are $R_1$ and $R_2$? It seems there should be an exists statement in ER1 and ER2 on $R_1$ and $R_2$. Why is $s$ necessary? Why is the output of $ERi(\cdot) \in \{-1, 0, 1\}$ rather than a binary output? Several other important technical points are unclear because it is not clear how the specifications in text translate to the notation (among others mentioned in the weaknesses section: is the paper examining individual explanations or global/SCM-level explanations, or something in between with population-level explanations?).

Quality: The content of the paper does not support its claimed contributions. See the weaknesses section.

Originality: The concept of an algorithm which recursively traces the ancestors of a variable which the algorithm is explaining to construct an explanation, as in Definition 4, is interesting and novel to my knowledge. The concept exists in Hall's notion of production but it hasn't been used to algorithmically construct explanations to my knowledge.

Reproducibility: The authors released their paper's code.
summary_of_the_review: While the paper is targeted towards an important field and introduces an interesting general concept of how to construct explanations, the content of the paper does not support its claimed contributions. The paper also needs reorganization to make its contributions clear, as it states stronger and distinct contributions in its introduction/abstract when compared to those that are listed in its contributions section. The paper also needs to clearly state the assumptions and limitations underlying its method, as described in the weaknesses section (my comments from 1.1.-1.4). Therefore, I recommend that the paper be rejected.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this submission the authors present an end-to-end procedure for explanatory interactive learning. Most prominently, this submission focusses on a logical approach to constructing causal explanations of binary ordering relations, i.e. answers to "why" questions. They go on to test their approach in a survey of 22 participants.
strength_and_weaknesses: Strengths:

This paper addresses an important problem, and is clearly written. I found the Causal Hans example to be particularly helpful in understanding the paper's motivations, the problem scope, and the approach taken. 

Opportunties for Improvement:

Despite the fact the paper addresses an important problem and communicates the approach clearly, unfortunately I have many signficant concerns about this work.

(1) Theorem 1 is vague to the point of being unfalsifiable. To see this, consider an absurd structure learning algorithm that simply selects a random graph without looking at the data. While the statement is literally true that we could compute SCE, it says nothing about the quality of the derived explanation. Should a reader interpret the Theorem literally, in which case it is uninformative about explanation quality?

(2) If my interpretation of Theorem 1 is correct, and it is claiming that the output of a sufficiently well behaved structure learning algorithm (e.g. a sound and complete algorithm that returns the entire markov equivalence class of graphs), then the Theorem is incorrect. Graph structure learning algorithms can not return unique structural causal models, as many structural causal models map to the same causal graph. In other words, there is a type mismatch between the output of a structure learning algorithm (equivalence class of causal graphs) and the inputs of the SCE procedure (a unique SCM). See Bareinboim et al for discussion of the relationship between SCMs and graphs.

(3) The paper describes "why" queries in english language that map to formally to search procedures over counterfactuals. However, the authors do not take into account that counterfactuals involve holding exogenous noise fixed between factual and counterfactual worlds, nor do they consider the large literature on causal explanations with SCM. This is a bit perplexing, given that the authors cite Halpern 2016 as an analogous set of definitions.

(4) The empirical results are presented in a way that is extremely uninterpretable. For example, the rightmost column of Figure 8 has no labels or numeric values at all. It is very important to have (at least) a summary of the empirical findings in the main body. In fact, I do not see anywhere in the appendix a table or graph comparing the proposed approach with CXPlain on a well defined metric.

Minor concerns

Page 3 - "A great deal of research in causality (especially for ML) is concerned with leveraging observational data to reason about causal relatinoships (also known as identification), ..." This is not exactly right. The task would be better described as "effect estimation". Identification, or identifiability, is when one wants to assess whether their causal assumptions are necessary to yield unique causal conclusions.

Page 4 - "We call Q_X := R(x, \mu^X) a single why question if Q_X is true". This seems like a type mismatch. If Q_X is the relation, then it is always a question. Is the intention here to say that the why question Q_X is a function that maps from R to explanations or answers?

Bareinboim, Elias, et al. "On pearl’s hierarchy and the foundations of causal inference." Probabilistic and Causal Inference: The Works of Judea Pearl. 2022. 507-556.
clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear, except for the presentation of the empirical results. The paper appears to be novel.
summary_of_the_review: The paper presents an interesting and important idea in end-to-end interactive causal explanations. However, I have significant concerns about the theoretical and empirical claims.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The focus of this paper is on Explanatory Interactive Learning (XIL), which consists of a loop in which the user queries the system, the system explains its predictions and the loop restarts. 

An assumption here is that this interactive loop is interesting to both the system (improving its prediction accuracy) and the user (increasing their trust in the system).

The authors seek to causal XIL. Shortly the research question is how the ML system would provide causal explanations.

Additionally, they argue that previous explanation methods are not causal, even when they assume a Structural Causal Model (SCM). Their strong baseline is the CXPlain method, and they show that CXPlain doesn't provide causal explanations.

Hence, the authors propose an explanation method founded in a given SCM. They refer to their method as Structural Causal Explanation (SCE). Additionally, they conducted a user study to investigate the causal explanations based on SCMs.

Aiming to be self-contained, the paper should add some explanations on its relatedness to Actual Causality (mentioned at the end of Sec. 2). 

It is a bit confusing when the authors mention M to be simultaneously (i) a valid why question and (ii) some proxy SCM (assuming that the hidden SCM is M^*).
strength_and_weaknesses: Strengths:

- This paper presents a new causal-based explanatory method. Since many learning algorithms produce black-box models (e.g., Deep Neural Networks), explanatory methods are needed, and the causality aspect of the proposed method turns its application more viable in real-world settings (e.g., decision-making in healthcare).
- When guiding the readers through a toy example, the authors provide the necessary intuition about their contribution.
- The tentative validation  (see weaknesses below) of the failure of previous methods to explain ML predictions is relevant to highlight the contribution of this paper.

Weaknesses:

- As far as I understood, the proposed method assumes the learned (from data) causal graph, not the SCM. For instance, in Sec. 4.2, the authors investigate the possibility of generating causal explanations from the learned causal graph.
	- Or is it the case that after learning the causal graph, their method learns the coefficients of a linear model to infer the equations of the SCM? (assuming linear structural causal equations)
In any of these possibilities (or other understanding), the authors should clarify this issue in the text.
- There is a lot of additional information in the Appendix, including important discussions, such as:
	- The relationship between the proposed method and Actual Causality.
	- Possible shortcuts to overcome the hidden confounder limitation of the proposed method.
There is no doubt that the authors provided a detailed Appendix for key points, and there is no sufficient space to include this information in the main text. My concern is that, at least, the above bullets should be in the main text.
- As discussed in Sec. 3.2, a limitation of the proposed method is that it doesn't consider hidden confounders. This assumption is unrealistic in several real-world scenarios.
- In Sec. 4.1., I did not understand the two shortcomings of CXPain. Since the SCM is given to both (i.e., SCE and CXPlain), it contains the information to derive the causal relations among the variables. Isn't it possible to use the input SCM to specify (i) direct/indirect causes among CXPlain's output? and  (ii) infer the causal effect among the output of the same explanation?
- In terms of limitations, I believe that would not be easy to learn the causal graph for high-dimension data  (e.g., images/videos) that maintain semantic information in the graph. Hence, if that is the case, the authors should highlight the discreteness requirement of datasets for which their method is suitable.
- Besides the cases of queries and explanations, I would like to see some general methodology to evaluate the quality of several queries. If that is not possible yet (i.e., it is an open research problem), the paper deserves some discussion on this.
- Given definitions and the analysis of theoretical parts of ER and SCE, it is still unclear how there are a loop works and how the input from decision-makers, for instance, may alter the final explanations.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow. Aside from already mentioned parts that are exclusively in the Appendix. In particular, the toy example of Hans problem enables a clear instantiation of the proposed method and clarifies some difficulties in understanding.

There is no doubt that this article advances the understanding of explanations of machine learning methods by making proper assumptions (i.e., more artifacts than data alone) and by providing causal answers.

Reproducibility criteria are a strong point in this paper. The authors provided both the code and data. I did not have the opportunity to reproduce the results, but everything seemed fine.

Minor suggestions:

- Include (FOL) in the Parag. after Def. 2, since it is the first appearance of the First-Order Logic term.
- Fix sentence in Parag. 1/Sec.3.2.
- Fix "algoirthm" to "algorithm" in Proof of Theorem 1.
- Fix "the that the..." to "that the..." at the end of Sec. 4.1.
- Define NT in Sec. 4.2.
summary_of_the_review: This paper address a relevant topic, i.e. explanations of learned models, and do this by exploiting concepts and techniques from the Causal Inference literature. The authors provide a toy example that guides the readers throughout the paper. Additionally, they validate several hypothesis and provide detailed information in Appendix.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of deriving causal explanations for the decisions of opaque Machine Learning models. 
Unlike current state-of-the-art methods, this work proposes an explanation method that aspires to consider causal interactions by exploiting the knowledge of a given SCM. In particular, the novelty of the approach relies on the capacity to distinguish between direct and indirect effects and the possibility of identifying information on the causal effect with positive and negative cases.
strength_and_weaknesses: Strengths: 
+ The paper is self-contained, and the contribution is well structured at the conceptual level. Thanks to straightforward examples and infographics, it is also clear for the reader to understand.
+ Related work is condensed but rather complete, and background discussion places the current work in a relevant and impactful study area. 
+ Due to the approach's user-oriented nature, using a user study to evaluate the understanding of explanations is an engaging strategy. 

Weaknesses: 
- The availability of the full SCM is a strong assumption. In general, it is not testable and may thus not hold in practice. 
- Authors should introduce a discussion on possible limitations of the algorithm
- All datasets used for the experimental evaluation contain a maximum of 4 variables. I would suggest to test the proposed approach w.r.t. dataset with higher dimensionality.


clarity,_quality,_novelty_and_reproducibility: - The paper is easily readable.
- Aspects of the contribution exist in prior work. The contributions are only marginally significant.
- A GitHub repository is provided by the authors. Experiments in the paper are fully reproducible.
summary_of_the_review: A well motivated paper with extensive and comprehensive experimentation. The evaluation of the paper would improve if the authors could be able to address the weak points highlighted above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper develops a new end-to-end procedure for explanatory interactive learning based on causal reasoning. The paper’s proposal was received in mixed ways by the reviewers. On the one hand, reviewers Upiw and j3te considered the method conceptually exciting and the idea of pursuing human validation compelling. On the other hand, reviewers 9oUq and vYZo raised some issues related to the technical and conceptual claims made by the paper. The technical quality and precision of the last two reviewers were high, and I appreciated reading their reviews and engagement. 

After all, both sets of reviewers raised valid points — the paper has novel and compelling ideas but, at the same time, has serious technical concerns that need to be addressed before publication. Given the interesting ideas behind the paper, I would like to encourage the authors to take the detailed feedback provided into account and improve the presentation and clarity in its next iteration.
justification_for_why_not_higher_score: The paper contains intriguing ideas but is not ready yet for publication due to some technical issues. For more details, please refer to the reviews and the subsequent discussions.
Decision: Reject