Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper develops a method for identifying causal effects on a response variable from observational and interventional data, with the restriction that under interventions only the response variable can be measured. The method does not assume away latent confounders. Theoretical justifications and some empirical evaluations of the method are presented .  
main_review: This paper tackles an interesting question and the proposed method has novel elements. However, details of the paper are not easy to follow. At the moment I am puzzled by the following aspects of the work:

1. The paper proposes that if the effect of a variable X on the response variable cannot be identified by the generalized backdoor criterion (GBC) even given the true MAG with all the purity information, "Fail" will be returned for this effect. But why not intervene on X in such a situation to directly estimate the effect? In fact, in Section 3.4, the "Fail" situation is supposed to be "detected" by intervening on X. Then why not use the interventional data there to estimate the effect in question instead of returning "Fail"?  

2. It is also unclear to me how those MAGs that do not license identification by GBC can ever be ruled out in the proposed method. For such a MAG, any effect estimated from the interventional data seems to be consistent with it, as the MAG does not entail what the effect is by GBC. 

3. The definition of local MAG is given on p. 5 relative to one variable, but Algorithm 1 seems to use local MAGs in a more general sense, which is not clearly defined (as far as I can see).

4. The statement of Theorem 9 is a little curious on two counts. First, as I indicated in point 1 above, the locution "the causal effect of X on Y is identifiable by GBC with the interventional data of Y under intervention on X" sounds odd because the causal effect of X on Y is identifiable with the interventional data of Y under intervention on X, and it is unclear why "identifiable by GBC" is so important. Second, the theorem sounds like a simple tautology: the causal effect is identifiable by GBC if GBC does not fail to identify it.   

Minor comments:

i. In Proposition 1, since G is supposed to be an ancestral graph (as opposed to a partial ancestral graph), should "PossDe" be simply "De"?

ii. In Definition 3, the terminology "deterministic marks" is misleading. "Definite marks" or  "determinate marks" sounds better.

iii. Also in Definition 3, a local MAG is typically still a *partial graph*, so not an ancestral graph, right?

iv. In Definition 4, "descendant" or "possible descendant"?
limitations_and_societal_impact: Adequately addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a method to identify the causal effect of each treatment variable on a response variable under latent confounding. The method starts from a PAG and further learning its structure by using the Generalized Backdoor Criterion (GBC) or resorting to interventional data where only the response variable is observed. The process finishes when all effects have been identified. For this, the authors introduce a graphical condition based on the notions of "local MAG" and "critical variables" that allow them to determine the existence of a minimal conditional set (MCS) that is valid for adjustment for some effect P(Y | do(X)).
main_review: The setting considered in the paper combines elements from structural learning and data-driven causal inference. As far as I know, the task of identifying all effects while learning the structure posed in the paper is novel. In particular, in the presence of latent confounding and having observational data plus interventional data where only the outcome is observed as input. Relevant related work seems to be properly cited.

The claims made in the paper seem well supported both by theoretical analysis and experimentation. While the method relies on the GBC, I wonder why the Generalized Adjustment Criterion (Perković et al. 2015) is not a better choice since it is known to be sufficient and complete for adjustment in the context of DAGs, MAGs, CPDAGs, and PAGs. The algorithms are described with enough detail and their complexity has been analyzed.

I find the paper well-written and clear, with a few exceptions. In particular, I find the wording and subsequent explanation of Theorem 9 confusing. What does it mean for a causal effect to be "identifiable by interventions"? 
The experimental section is clear and provides a compelling comparison with other methods.

The results presented in the paper address a causal inference setting that could arise in many practical scenarios. Allowing for only the response variable to be observed under intervention provides flexibility compared to other methods that require observation of all non-intervened variables. The method also seems to improve, compared to previous work, in terms of computation complexity and the number of required interventions.

Other comments:
- Consider adding a title to Definition 3.
- Line 180: "GBC dose not fail"
- Line 347: "of first part" (missing article)
limitations_and_societal_impact: I am satisfied with the discussion of the limitations provided in the paper.
I can't foresee any societal impact of the work that needs to be addressed at the moment.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Based on the ancestral graph framework, the authors tackle the problem of identifying an interventional distribution $f (Y | do (X))$ by only observing the outcome $Y$ after each possible intervention. This work extended previous work by Wang et al. assuming no latent variables on the graph. 
main_review: As most of us know, the graphical model that best approximates a DAG with latent factors is an Acyclic Mixed Directed Graphs (ADMGs), which preserves not only conditional independence constraints but also dormant conditional independence constraints after one performs fixing operations on the original ADMG. But because the characterization of MEC of an ADMG is still an open problem, most of the current works consider a relaxation of ADMG -- MAG, as in the current submission. Unlike ADMGs, MAGs do not preserve dormant conditional independence constraints; also, the characterization of MEC of MAG has been solved by Richardson and Spirtes (2002).

This paper seems to be an extension of an earlier work by Wang et al. 2020 ICML, in which Wang et al. also consider only observing the outcome after active interventions, but under causal sufficiency assumptions. This paper, however, considered a more difficult problem, for which unmeasured factors/confounders are allowed. 

The theoretical development is new but it is not easy to evaluate if the intervention scheme considered in this paper is very practically relevant. To me, the most (or in my opinion, the only) meaningful causal discovery setting is in genomics where large-scale intervention data is available to actually test the results, but in genomics, it is rarely the case that we only get to observe $Y$ after the intervention. As a result, I give a score of 7 to this paper.

Besides the general comment, I have more specific comments or questions regarding the submission:

(1) More recently, the research group of Elias Barenboim has also solved the problem of characterizing the so-called interventional MEC of MAGs when one has mixed observational and interventional data. Though the authors decide to take a different route (i.e. local search), it seems to me that it is entirely possible to characterize the MEC of MAGs when interventional data is available but we only get to observe $Y$ in each intervention. Have the authors considered such a problem? If not, is it by any means helpful for settings similar to what the authors are trying to solve?

(2) In practice, decision variables might not be all observables except $Y$. In this setting, if I am only interested in $Y | do (X_{i})$ for $i$ from a subset of $[p]$, would there be a more efficient algorithm than simply applying ACIC but just focusing on this subset?

(3) Intervening decisions variables one by one also seems restrictive. It is quite reasonable to consider cases where multiple variables are intervened on. If one can intervene in more than one variable, what are the technical challenges of developing theory?

(4) A discussion on the potential generalization of this work to ADMG will be helpful to reach a more general audience of causal inference.

Minor points:
(1) Line 143: I would prefer to add its complete name when the authors first mention MCS.

(2) Line 115: "(identifying failing of GBC is counted)." Does it mean that "failing to identify $Y | do (X)$ by GBC"?

(3) After reading the paper, my understanding is that ACIC is sound but not necessarily complete. I would suggest the authors discuss this issue at the end of the paper and mention it in the introduction.

(4) The paragraph below Prop 1 seems hard to understand. In the second unidentifiable case, do you mean that a generalized backdoor set may not exist?
limitations_and_societal_impact: This work is purely theoretical so no immediate potential negative societal impact.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper extends a previous work where a method was presented for learning causal effects when some interverventions can be conducted, but such that only the distribution of the response variable is observable after the intervention. This paper extends the mentioned previous work by allowing for possible latent confounders, which requires listing possible alternative structures in an active manner.
main_review: PROS: 
- Extending a previously published idea to a situation where there are latent confounders. 
- The derivations seem correct.

CONS:
- The motivation for the paper is not very well elaborated. The paper mentions an application in healthcare, where patients are not fully checked after treatment and only their illness is observable by doctors, but this seems overly simplistic, especially because there is no corresponding real data experiment.
- The presentation is a bit hasty. First of all, the main text is not self-standing, but it's necessary to read at least ref [8] and the supplementary to grasp the main idea. Also, some abbreviations are used in the text without defining them first (at least ACI, MCS). Obviously the page limitation restricts what can be put to the main text, but I think it would improve the paper if the main text focused more on motivation, key points, and intuition.
- Novelty compared to ref [8] seems a bit limited.
- Based on experiments the difference of the presented active learning approach to a "random" approach seems small (Fig. 3). Also, the experiments are somewhat limited in general.
limitations_and_societal_impact: There is little on limitations and no discussion on negative impacts. I don't think there are serious ethical impacts, but more discussion could be provided on the limitations.
review_title: null

Meta-review: The paper explores a causal setting where we are interested in understanding the impact of possible interventions on a single response variable, using observational data (which might suffer from latent confounding) and limited interventional data. The limit on the interventional data is that we only observe the response variable, and nothing else. This work extends several other papers that have dealt with similar setting, with the important distinction of allowing latent (i.e. hidden) confounding in the observational data. 
The paper is a theory paper, and the reviewers agreed that it contains a novel contribution and that it is technically correct. 

The main concern of the reviewers was about the applicability of the specific scenario the authors propose; however, the overall decision is that the real theoretical contribution is important enough. Furthermore, I believe it is often difficult to know in advance which learning scenarios or which techniques will end up useful. Since the work is both rigorous and novel, the recommendation is to accept the paper. 

The reviewers made some thoughtful suggestions in their reviews and the discussion period, and I encourage the authors to incorporate some of the conclusions in their revised version. 
Decision: Accept (Poster)