Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper provides necessary and sufficient graphical conditions for ‘optimal’ adjustment sets in causal graphical models with hidden variables, where the optimality is defined by a newly proposed metric, instead of the commonly used smallest asymptotic estimation variance.
main_review: I think this paper needs to justify and further explain the proposed adjustment information (Definitions 1) and information-theoretical graphical optimality (Definition 2). Intuitively, an estimator is statistically preferred if it is less biased and less variable than others. In Section 1, the authors mentioned that maximizing the adjustment information formalizes the common intuition to choose adjustment sets that maximally constrain the effect variable and minimally constrain the cause variable. I am wondering whether and how this “common intuition” is related to statistically preferred estimators. (Note that, the variance is  a measure of variability, but not the only one.) Besides, please provide a more detailed explanation of the linkage between this “common intuition” and the definition of adjustment information.
limitations_and_societal_impact: The authors have described the limitations in Section 4.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper defines the optimality of a (valid) adjustment set in terms of a notion called 'adjustment information.'  Then, it focuses on estimators for which maximizing adjustment information equates to minimizing asymptotic estimation variance. The authors provide a sufficient and necessary graphical criterion for the existence of an optimal adjustment set under this notion. They also give an explicit and efficient construction of an adjustment set called the O set, which is proven to be optimal whenever graph optimality holds (the constraints in the graph are sufficient to determine an optimal adjustment set).
main_review: The results presented by the authors are novel and advance previous work by considering the hidden variable setting and characterizing the existence of optimal adjustment sets for estimators satisfying certain information-theoretic properties. It appears relevant work has been appropriately cited and discussed. 

As far as I can tell, the results and claims presented are supported both theoretically and experimentally. The authors are very clear about the assumptions needed for their results to hold and also consider its limitations. Experiments consider both scenarios where the assumptions hold and where they don't; making sense of observed results.

I find the paper well written. Relevant previous work, definitions, and results are introduced clearly and making good use of several examples. The experimental section and further discussion help understand the benefits of using the proposed result and its limitations. I was wondering why the experimental section only considers the case where $\mathbf{S}=\emptyset$.

Overall, the paper makes important theoretical and practical contributions to the problem of identifying causal effects, using covariate adjustment, in settings where the graphical structure is not fully known. In particular, the results advance the state of the art for choosing and constructing optimal adjustment sets for certain estimators.
limitations_and_societal_impact: The limitations of the proposed criteria and algorithm have been well explained, discussed and empirically verified.
I can foresee any societal impact that needs to be addressed at this time.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The problem of selecting optimal backdoor adjustment sets to estimate causal effects in graphical models with hidden and conditioned variables is addressed.
main_review: This paper solves the problem of selecting the optimal backdoor adjustment sets by leveraging the theory of information theories. This paper is motivated by the interesting observation that the optimal admissible set is the set providing the most information (adjustment information, Def. 1). 

However, I have to confess that I couldn't capture the main claim of the paper. Does this paper answer that O-set in Def. 4 is the optimal backdoor admissible set whenever there exists a valid admissible set? It seems that the main claim in Thm.3 only guarantees the optimality of O-set for some special cases. 

Despite its interesting and novel approaches, I also had to agree that it is hard to understand this paper. I will discuss those in detail in **Readability** paragraphs. Here are some main questions regarding the contribution/novelty of this work:  

- I am not sure about the distinction between SSR20 and the result of this paper. Up to my understanding, Thm 2 of SSR20 provides an (statistically-) optimal adjustment set for the nonparametric setting, and the only condition is the existence of the back-door admissible set. That is, I don't understand which particular points have never been addressed by SSR20.
- In Eq. (3), why do you consider the conditioned variables S? Why did you prevent the possibility of S being arbitrary? (what's the role of the assumption S \cap des(X) = \emptyset in Line 87?)
- Line 109-110, why is this a valid intuition? That is, what would be the specific benefits of invoking the information theory?


## Readability 
Here are comments discussing which particular parts I felt difficult to understand. I also added my comments and recommendations to help to improve comprehensibility of the paper. 

- In general, it is not desirable to put the essential contents in the Appendix, because readers must go back and forth to parse the main and appendix at the same time. For example, it's hard for me to understand Thm. 3 and Line 78.
- Please write the assumption more explicitly. I think one of the crucial assumption is the linear SCM setting (where all the variables are in the linear relation), as you wrote in line 125-126.
- I'd like to add some overview or guideline for every starts of the section, Definitions, Theorems, and Lemma. For example, I couldn't guess what authors will discuss in Section 2.2 even after reading the first paragraph.
- In Line 62-63, I'd recommend to give a detailed definition of ADMG, because readers might not want to look for the definition of ADMG from other sources. Without the definition, it's hard to parse the graph in Fig. 1; For example, what does the bidirected edge mean?
- In Line 62-70, what is the definition of An(X), De(X)? Do they include the variable X? Please write it more explicitly.
- In Line 62-70, I am not sure why you wanted to introduce the notion of MAG, given that this notion has never been used. Since MAG, PAG, Markov equivalence, "visible edge" are difficult concepts, I recommend to drop those in your work unless it's essential.
- In Section 1.1, {X,Y} are singleton? Otherwise, I recommend to use the bold letters.
- In Figure 1, what is the meaning of "-" and "+"?
- I think the question in Line 106 is misleading and unclear what the research question is. Why did you give vancs (in line 105) before giving the research question? It distracts readers who want to focus on the research question. Also, the goal is supposed to minimize **asymptotic** estimation variance, but the word "asymptotic" is absent. Finally, since the line 106 is the main statement giving the research question, I think this deserves more than a line. Please highlight it more explicitly, because, as a reader, this is the very sentence that I want to know from Section 1.
- In Line 122 and Eq. (7), I think this term is a mean-square error, not asymptotic variance.
- I'd recommend to move line 125-126 after Assumption 2, given that this gives a concrete picture what you want to study within the assumption 2. Without concrete examples for Assumption 2, it's hard to parse this paper.
- I don't parse Def. 2 (what is the meaning of "no other Z' or for all other Z'"?). Does it mean that Z is optimal if Jz is maximized among all Z' that is a valid adjustment set?
- It's hard to parse Def. 4, because there is no definition for the notation that you used for defining C.
- It seems that Thm 3 provides special cases when O-set is the optimal. It seems that there must be the optimal set even when the conditions in Thm. 3 fails (for sure, O-set is not the optimal one in this case). Can we have a result that we always can have a O-set whenever there are valid back-door admissible set?
limitations_and_societal_impact: * As written above, I'd like to recommend to write the linear SCM assumption (line 125-126) more explicitly. 
* Does this result hold for the singleton X,Y?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors propose a criterion to select the optimal adjustment set. The authors propose a criterion (Eq. 4 and Def. 2) based on information theory to select which adjustment set is best. And they build the connections between this criterion to the common used minimized asymptotic variance (Eq. 7) for linear Gaussian causal model. Based on these, they propose a necessary and sufficient criterion for the existence of an optimal set. However, it is inefficient due to the large number of comparisons. Hence, the authors present an algorithm to find the o-set, which could be as the optimal set under some conditions as Thm. 3.
main_review: After rebuttal: The authors have addressed my questions. I think this paper tackles an interesting problem. I am happy to see it accepted.

-------------------------
Although I try my best to read this paper, it is hard for me to understand the details. Hence if some parts I summarized in the ``Summary'' part are inconsistent to the paper, I will be grateful if the authors could point them out.

The proposed method is technically solid. And the experimental results are convincing. My main concern is about the motivation to provide the criterion that does not depend on the distribution. As shown by ``Our goal is to provide graphical criteria for optimal adjustment sets, i.e., criteria that depend only on the structure of the graph G and not on the distribution'' on Line 143. I do not think this target is well motivated. In reality, we often have observational data. In that case, the distribution is known (although it is estimated) and which set is optimal for example in Example F seems not to be agnostic. Hence I look forward to an illustration for the motivation to provide the criterion that does not depend on the distribution. 

I am curious about the computational complexity of Lemma 2. It seems that this step will not cost so much because there are usually not so many valid adjustment sets. If it does not cost so much in reality, it is feasible to directly find the set by Lemma 2. What's more, if we know the (estimated) distribution of all the observed variables, we could just find the optimal st by Lemma. It seems to be more sensible than the o-set.

It is a bit puzzling by the sentence ``Z that maximally constrains the effect variable Y and minimally constrains the cause variable X'' on Line 110 implies. Could the authors please provide a detailed illustration? What does the constrain here mean?


A suggestion:
I understand the equation on line 124 as f(H_{Y|X,Z,S}-H_{X|Z,S})=\frac{1}{\sqrt{n}}e^{H_{Y|X,Z,S}-H_{X|Z,S}}, is it right? The format right now is not quite strict. 
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The paper aims to get around a known negative result in the theory of covariate adjustment due to Rotnitzky and Smucler: in hidden variable causal models, an adjustment set that minimizes the variance of the target parameter does not depend only on the model, but also on the element within the model (in other words, not only on the graph, but also on particular parameter settings).

To address this, the authors propose a criterion based on information theory that aims to maximally constrain the outcome, and minimally constrain the treatment. They then investigate cases when this criterion yields a variance-minimizing adjustment set.

While initially reviewer opinion on this paper was split, discussions with the authors led to a consensus that the paper is an interesting, novel, and worthwhile addition to NeurIPS proceedings.
Decision: Accept (Spotlight)