Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper shows the more detailed steps and pseudo-codes for finding frond-door adjustment sets in causal graphical models. Starting from the definition of front door criteria, this paper goes through the implementation of each criterion and offers a complexity analysis.
strengths_and_weaknesses: The strength of this paper is showing the detailed steps for implementing frond door criteria in causal graphical models, and it finally extends by listing all frond door sets satisfying Pearl's front door criteria. 
The weakness is that no demonstration or evaluation is available for given algorithms. 

questions: I think the complex statements are better to be shown in the main text.


This algorithm works by removing nodes from the full set of candidates, which could be all nodes except for the X and Y.
Does this algorithm take into account "evidence variables"?
If we find a full list of FD sets from a causal graphical model, and if we introduce evidence variables, should we restart from the beginning?


Have you implemented these pseudo-codes? 
I think it's better to double-check it by implementation on top of proofs and hand-written examples.
It is in a polynomial-time algorithm in terms of big-O analysis, but the overall running time may be longer than one might imagine when there are many nodes and edges in the graph. This algorithm can be served as a baseline for possible future works that improve running times and may be useful for implementing do-calculus. They are very close to script languages such as python already, and if possible, showing the results through running the program or providing it as open-source could be even more helpful to scientists who may use this FD strategy for experiment design. 


limitations: I think this work is not relevant to this section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a feasible algorithm for listing adjustment sets (one at a time in reasonable time) satisfying the front-door criterion.
strengths_and_weaknesses: This is very strong paper, very clearly written and easy to follow. The theory and algorithms are clear and make sense. It should be easy to implement the algorithm described. The idea is novel, since no such algorithm currently exists in the literature, and it is significant, since it could easily be incorporated into existing software and used. The only criticism is that it does not include an experimental section in which the algorithm is test for accuracy and speed on particular models.

Minor point.

Line 252, "no more node" should be "no more nodes".
questions: I did have some questions after having read the paper, which I didn't know how to answer.

* For medical or ethical reasons, it may be necessary to avoid including certain variables in an adjustment set. Is this straightforward to do with this algorithm (i.e., simply not including the prohibited variables in any set in the course of the algorithm), or could it possibly be the case the simply excluding certain variable will prevent allowable front-door adjustment sets from being discovered?

* For ethical reasons (fairness, for example), it may be necessary to disallow certain paths of influence in a model (e.g., race affecting the outcome variable). Can this be done inside the algorithms, or some one have to simply list example front-door adjustment sets and simply ignore the offending examples?
limitations: The last sentence addresses societal impacts: "We hope that the methods and algorithms proposed in this work will help scientists to use the FD strategy for causal effects estimation in the practical applications and are useful for scientists in study design to select covariates based on desired properties, including cost, feasibility, and statistical power." This can of course be made more than a hope; perhaps answering questions such as the above (or others) might help make that a reality.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Given the underlying causal diagram, this paper provides two algorithms:
1. To find an admissible front-door adjustment set in polynomial time (if one exists).
2. To enumerate all admissible front-door adjustment sets with polynomial delay.

These algorithms allow constraining the search to include and/or exclude specific subsets of variables which could help selecting estimands with desired properties.
strengths_and_weaknesses: Strength:
1. In my limited knowledge, this paper is one of the first papers to provide algorithms to find/list all admissible front-door adjustment sets (given the underlying causal diagram).
2. Finding valid front-door sets is an important topic and of relevance to the NeurIPS community. Through this work, the authors take a first step to close the gap between what is known for back-door adjustment sets and what is known for front-door adjustment sets.
3. By constraining the search space to include/exclude specific subsets of variables, the authors provide flexibility in selecting estimands with different cost, availability, privacy, and statistical power.

Originality:
1. The algorithm/approach proposed in the paper relies on a mix of novel concepts (e.g., GETDEP, GETNEIGHBORS) as well as ideas known in the literature (e.g., TESTSEP, LISTSEP). However, the key novelty lies in unifying these concepts to propose FINDFDSET and LISTFDSETS.

Weakness:
1. This paper assumes the knowledge of the underlying causal diagram in order to find admissible front-door adjustment sets. In most real-world applications, the causal diagrams are unknown and this requirement is quite restrictive. 

Clarity:
1. In general, the paper is quite well written. Further, the algorithms and the functions are generally well explained in the text. While the authors do a decent job, the function GETDEP is a bit complicated to understand. I strongly recommend the authors to add running examples covering various scenarios that may arise in this function to make it easier for the readers.
questions: Suggestions: 
1. As mentioned by authors in section 1 (line 77) and in section 5 (lines 393-395), there has been a lot of work on finding/listing back-door admissible sets when the underlying DAG/MAG/PAG/CPDAG is known. More importantly, there has also been some work on data-driven approaches for finding/listing back-door admissible sets using only an anchor variable when the underlying causal diagram is not known. For example: (1) Entner et al 2013 -- Data-driven covariate selection for nonparametric estimation of causal effects [https://proceedings.mlr.press/v31/entner13a.html], (2) Cheng et al 2020 -- Towards unique and unbiased causal effect estimation from data with hidden variable [https://arxiv.org/pdf/2002.10091.pdf], (3) Shah et al 2021 -- Finding Valid Adjustments under Non-ignorability with Minimal DAG Knowledge [https://arxiv.org/pdf/2106.11560.pdf]. In order to provide a comprehensive background regarding what is know for finding/listing back-door admissible sets, the authors should also talk about these works.
2. As mentioned in the weaknesses section above, this paper assumes the knowledge of the underlying causal diagram in order to find admissible front-door adjustment sets. In contrast, the recent work of Bhattacharya and Nabi-- On Testability of the Front-Door Model via Verma Constraints [https://arxiv.org/pdf/2203.00161v1.pdf] -- explores finding admissible front-door adjustment sets in a data-driven manner using only an anchor variable when the underlying causal structure is not known similar to the works mentioned above. The authors should differentiate their work from this work.

Typo:
1. Did the authors intend to say $3^n$ instead of $2^n$ on line 339?

limitations: In my opinion, the most important limitation of this work is the requirement of the underlying causal diagram which is acknowledged by the authors in Section 5. This limits the application of this work in most real-world scenarios where the causal diagram is not known. 

Therefore, in addition to finding/listing front-door admissible sets from incomplete or partially specified causal diagram as mentioned by the authors (in lines 395-396), it would be useful (for future research) to develop data-driven methods when the underlying causal structure is not known (except an anchor variable) by taking inspiration from the works of Entner et al 2013, Shah et al 2022, Cheng et al 2022, and Bhattacharya and Nabi 2022.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper develops an algorithm, i.e. FINDFDSE() to find an admissible adjustment set that satisfies the front-door criterion in a given causal DAG. The solution is to solve a variant of the problem that imposes constraints I for given sets I and R.  Based on the algorithm FINDFDSE(), a  sound and complete algorithm is developed to enumerate all front-door adjustment sets with polynomial time.  Moreover, the detailed theoretical analysis supports the soundness of the developed LISTFDSETS() algorithm.

strengths_and_weaknesses: *Strengths:
1. This work is well-motivated and organised. A large number of examples make the paper easy to follow and understand. 
2. The technical quality of the work is high. The steps of both algorithms are also very clear.

*Weaknesses:
1. Assumptions are not clear. To my understanding, the given causal DAG is a very specific class of DAGs. For example, $\mathbf{X}=\\{X_1, X_2\\}$, $\mathbf{Y}=\\{Y\\}$, $\mathbf{R}=\\{A,B,C\\}$; the graph is formed: $Y\leftrightarrow X_1 \rightarrow A\rightarrow B\rightarrow Y$; $Y\leftrightarrow X_2 \leftarrow A$; $B\rightarrow C \leftarrow X_2$; $C\rightarrow Y$. $\forall v\in \mathbf{R}$, we have TESTSEP($\mathcal{G}_{\underline{\mathbf{X}}}$, $\mathbf{X}$, $v$, $\emptyset$) = FALSE. Thus, FINDFDSET() returns $\perp$. However, for a single $X\in \mathbf{X}$,   FINDFDSET() returns $A$ for $(X_1, Y)$ and $C$ for $(X_2, Y)$, respectively. Therefore, the generalization on $\mathbf{X}$ may not be perfect.

2. The ``completeness'' is not specified. Furthermore, there is no evidence for completeness. 

Some minors:
1. Definition 1, "There is no back-door path from $\mathbf{X}$ to $\mathbf{Z}$"  should be "There is no unblocked back-door path from $\mathbf{X}$ to $\mathbf{Z}$". 

2. $\mathbf{PA}\_{V}$ and $\mathbf{pa}\_{V}$ are not defined.  



questions: Q1. is the first weakness.

Q2. In Fig. 1b, If the edge $A\rightarrow D$ is replaced by $A\leftarrow D$, is there still an admission front-door adjustment set $\mathbf{Z]$? If the answer is no, are there some graphical criteria to handle such a case?
limitations: 1. The causal DAG and the sets $\mathbf{I}$ and $\mathbf{R}$  need to be provided. In many real-world applications, this information is not available. Hence, the application of the developed algorithms in this work has limited application at present. 

2. For multiple treatments and outcomes $(\mathbf{X, Y})$, the developed algorithms may only be able to solve very few cases, and most of them return $\perp$.
review_title: null

Meta-review: metareview: The paper presents new algorithms for finding and enumerating sets satisfying Pearl's front-door (FD) criterion given a causal diagram. While the reviews have made some helpful suggestions which the authors should implement, they all agree that the paper makes a solid contribution to our understanding of this topic.

confidence: Less certain
Decision: Accept