Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors take a further step by introducing active interventions in differentiable neural network-based methods. The intervention target is selected based on maximizing the disagreement between the post-interventional sample distributions under the hypothesis graphs. The experimental results verify the effectiveness of the active strategy compared to random selection.
main_review: Strengths: 
1. The paper is written clearly;
2. the tackled problem is well-motivated;
3. I think the idea that select the interventional variable by maximizing the disagreement between the post-interventional sample distributions under the hypothesis graphs is novel.

Weaknesses:
One main aspect where some improvements could make the paper better is the experiment part. The comparison results between the proposed method and DCDI are supportive in only verifying that the active strategy is better than random selection. However, according to my understanding, although the novel point of this paper is to present an active causal discovery method based on continuous optimization framework, the tackled problem is still actively causal discovery. In the literature, there have been many classical methods regarding actively causal discovery (e.g. Active learning of causal networks with intervention experiments and optimal designs, Two optimal strategies for active learning of causal models from interventional data). Hence, in addition to illustrating that under continuous optimization framework the active strategy works, it is necessary to illustrate that the proposed method could preform better than the previous active causal discovery methods, which convince the readers that the proposed method is the best in this kind of tasks. In addition, could the authors give more details about the baselines? The baselines do not quite match the tasks in this paper, hence I want to know some execution details. For example, GIES is for obser. + passive inter., the active strategy is not involved. How do the authors select the intervention variable? ICP does not exploit the information about which variables are softly intervened (taken by dynamic environment). The comparison between the proposed method and ICP seems not quite fair. Do authors make some modifications? Towards active causal discovery, I think the two methods I mentioned above seem to be more suitable as baseline methods.

I have one another question. The authors claim that the proposed method is applicable for multi-node intervention. I agree with that. However, in this case the complexity of Line 2 of Algorithm 1 is quite large. How do the authors address this problem? 

Other suggestions or typos: 
1. I think it will be better if the authors give more details about training functional parameters in the paper or appendix. The current version made me have to read paper DSDI.

2. A missing full point in the paragraph of Assumptions (Page 3).
summary_of_the_review: The main reason that I give a slightly negative score in the current is that I think the experiments are not supportive enough. Different from some theoretical causality paper, this paper tackles a practical and widely researched problem. Hence I think the experiments need to be designed carefully. I look forward to more clues from the authors. If they could address my concerns and questions, I am very happy to increase my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method to select interventions that enable efficient identification of the underlying causal structure. In particular, the method picks the intervention that exhibits the highest discrepancies between post-interventional sample distributions generated. The authors provide experiment results to demonstrate that the proposed method has a better performance over the other baselines, and also improves the sample efficiency.
main_review: Overall, the proposed method is interesting and novel. However, since the proposed method is based on a heuristic without theoretical analysis, as also acknowledged by the authors, I think it is important to demonstrate empirically with a fair experiment setup that the proposed method works well compared to the existing baselines. I am willing to increase my score if the authors are able to address my comments below.

Strengths:
- The paper is well written.
- The proposed method is interesting and novel.

Weaknesses:
- The proposed method lacks theoretical analysis.
- It is not surprising that GES, NOTEARS, and DAG-GNN do not perform well, as they are based on observational data. I would encourage the authors to include more baselines that handle interventional data, such as IGSP [1] or UT-IGSP [2], together with nonparametric test.
- It would be better to provide what score is used for GIES and GES. Have the authors considered the generalized score [3] that better handles nonparametric relationship for a fairer comparison?
- Based on my understanding, the authors used the original ICP [4] that is based on linear model. It may be better to also include its nonlinear variant [5].
- Figure 23 seems to show that DCDI performs better than DCDI+AIT. It would be better to include similar experiments in Tables 1 and 2 for DCDI.
- How did the authors ensure a fair comparison between the proposed method based on active intervention, and the baselines based on observational data and/or random interventions? E.g., what are the number of samples/interventions used for both cases?
- The paper would be stronger if it includes the structural intervention distance (SID) [6] that may also be informative.

Minor comments:
- Based on my understanding, the experiments in Tables 1 and 2 are based on discrete data. It would be better to explain how NOTEARS handles discrete data.
- For the real datasets considered in Section 4, did the authors experiment with the original datasets, or only adopt their ground truth causal graphs and generate synthetic data based on these graphs? It would be better to make this clear in the main paper. The current description sounds like it is the former (e.g., the last two sentences in the subsection "Structure discovery: flow cytometry and asia dataset"), although it seems unclear how active interventions could be applied there.

References:
1.  Permutation-based Causal Inference Algorithms with Interventions, 2017.
2.  Permutation-Based Causal Structure Learning with Unknown Intervention Targets, 2020.
3. Generalized Score Functions for Causal Discovery, 2018.
4.  Causal inference using invariant prediction: identification and confidence intervals, 2015.
5.  Invariant Causal Prediction for Nonlinear Models, 2018.
6.  Structural Intervention Distance (SID) for Evaluating Causal Graphs, 2014.

-----------
After reading the authors' response and the other reviews, my concerns about the experiment setup persist. I agree with Reviewer ma3K that some part of the experiments could be further improved to ensure a fair experiment setup.

summary_of_the_review: Overall, the proposed method is interesting and novel. However, since the proposed method is based on a heuristic without theoretical analysis, as also acknowledged by the authors, I think it is important to demonstrate empirically with a fair experiment setup that the proposed method works well compared to the existing baselines.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Structure learning from observational data is a long-standing challenge that could benefit from creative uses of interventional data. The authors explore the use of active learning in a continuous optimization framework to better traverse and narrow down the space of potential graphs that describe the data. The contribution is a heuristic to choose the "best" intervention target based on a comparison between samples from an intervened SCM with the current best guess of an underlying causal discovery algorithm (in which the proposal is embedded).
main_review: Sampling from interventional distributions and quantifying its difference with the available data is creative as an approach to guide the selection of interventional targets. Unfortunately, the paper is very hard to follow and seems to rely excessively on DSDI, both to understand the proposed approach and in applications despite the authors claiming that the proposal can be used with any differentiable causal discovery algorithm (contributions). The exact procedure is complex, it involves sampling DAGs and parameters, and should be clarified. 

Am I correct in assuming that in every training iteration of the causal discovery algorithm, the proposed approach samples DAGs, interventional distributions, and samples from each interventional distribution to be compared using the proposed score? What is the computational complexity?

Along these lines, the exact procedure is unclear. Algorithm 1 vaguely mentions "perform an intervention" on line 3 without details of what kind of interventions are considered (e.g. do, soft, what exact values are set, etc.). Next, it is meant to draw samples from $\mathcal P$ which is defined as a set of functional parameters which I understand are conditional distributions. How are the conditional distributions chosen? What is the functional relationship between parents and children?

My main concern is about the claim that one can meaningfully sample from constructed interventional distributions. For this to be valid one would need access to the underlying structural causal model in general, which is far beyond the DAG we seek to estimate in the first place - I don't believe that the constructed interventional distributions give any useful guidance to select interventions as interventional distributions are not identified in general. The use of the term "interventional data" I find misleading as nowhere in the algorithm description of experiments (unless I missed it) do the authors make use of a data source in which an experiment has been performed. 

One argument repeatedly made in the paper is that existing structure learning algorithms are computationally expensive. Yet, experimental evaluations are made on graphs of less than 20 nodes, on relatively simple graphs, and without run time comparisons. This claim and others are far from justified.

If the main contribution is the score, this is not sufficiently discussed or justified. What makes this a good score? Do we have any guarantees that it favors good interventions?

summary_of_the_review: Ultimately my opinion is that without additional theoretical results or better justification and exposition of the proposed approach to advance the state of the art, the contribution is limited.
review_title: null

Meta-review: This paper proposes an active intervention-targeting mechanism for causal structure discovery. After the discussion, there was a consensus among the reviewers that this paper needs another round of revision to address the lingering concerns. These concerns include providing a more fair experimental setup (e.g. by properly distinguishing and designing proper experiments for the observational, random intervention, and targeted intervention settings). Since the paper lacks theoretical guarantees (which is OK and not a requirement for acceptance), the merits rest on providing a thorough and fair experimental evaluation.
Decision: Reject