Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors considered the problem of experiment design in an active setting. They proposed a gradient-based method called GIT, which considers the norm of structural gradient as the acquisition function for selecting a node for an intervention. The experimental results showed that GIT outperforms previous work in low-data regimes.


strength_and_weaknesses: Strengths:
- GIT can work with gradient-based causal discovery methods.
- Experiments showed that GIT provides better estimates in the low-data regimes.

Weaknesses:
- The proposed method is limited to categorical variables and it works under the causal sufficiency assumption which may not hold in practice.
- There is no theoretical guarantee for the output of the proposed method.
- Some parts of the paper are vague and it is required to be more precise about the claims.

My specific comments regarding the submitted paper are given in the following:

- Is there any theoretical guarantee on the performance of GIT in terms of the number of interventions? Can we show that it is an approximation algorithm? 
- Is GIT a consistent estimator in the sense that we can find the true causal structure as the number of interventions goes to infinity?
- Why is the norm of the structural gradient, a good choice for acquisition function? It would be good to provide some justifications for it.
- In Algorithm 2, the authors considered a Monte Carlo scheme which might be time-consuming. It is good to mention how many samples should be generated and what is the computational complexity of the proposed method.
- On page 4, it is mentioned that "Virtually any existing gradient-based causal discovery method fulfills these requirements." This sentence is so vague. What are the exact requirements? Why do all the gradient-based methods satisfy these requirements? What does it mean "Virtually"?
- On page 9, the tile "Soundness of GIT" is misleading. In some toy examples, it was shown that GIT picks reasonable choices but it does not mean that GIT is sound. Moreover, it is not clear what the authors mean by "soundness" here. It might be the case that they want to show the optimality of the proposed method.
clarity,_quality,_novelty_and_reproducibility: Some parts of the paper are not generally well-written and some sentences are vague. Regarding the novelty, in my opinion, the idea of using the norm of the structural gradient is somehow novel but there is no justification in the paper about it. It seems that the results are reproducible based on the explanation in the appendix.
summary_of_the_review: The proposed solution is limited to categorical variables and it is assumed that causal sufficiency is satisfied which may be not held in practice. Moreover, there is no theoretical guarantee on the performance of the proposed solution. In addition, the idea of using the norm of the structural gradient is not justified in the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a gradient based rule for selecting intervention targets that is used with the ENCO experimental causal discovery algorithm. Empirical results are favorable for the proposed method compared to other procedures for ranking intervention targets.
strength_and_weaknesses: The paper is generally well written and the empirical results appear to argue favorably for the approach.

The biggest weaknesses are the paper is not very self contained and the novel methodological content is very brief without correctness proof or other theoretical results.  Consequently, it’s difficult to both gauge significance and fully understand the motivation. Aside from the limited methodological content on GIT, ENCO (the algorithm which takes the GIT rule as input) is also not described in the paper.

While the empirical results are favorable, it’s not clear how realistic performing many interventions is.

clarity,_quality,_novelty_and_reproducibility: The paper is clear, but not self contained. The novel content is a rule that is used with an existing algorithm ENCO so novelty is somewhat limited.
summary_of_the_review: The paper is clear and the empirical results are favorable, but there is limited novel methodological content which builds on prior work which is not described in detail.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies the problem of learning a causal DAG from observational and interventional data. The method follows an active learning approach where the goal is to reduce the number of interventions to learn the underlying DAG. The core contribution of this work is on using the gradient of a given score function with respect to the structural parameters to guide which node(s) to intervene next, that is, the method falls under the umbrella of score-based approaches. Also, as interventions might be too costly or impossible to perform, the method is capable of using simulated interventions from the running DAG. Several experiments are provided in synthetic and real-world datasets.
strength_and_weaknesses: **Strengths:** 
The paper is mostly easy to digest. The introduction clearly presents the problem at hand and the main contributions of the paper. The rest of the sections are mostly well-written but I believe some points deserve more details/discussions and perhaps some re-organization of the paper, see below for more details. Extensive experiments are provided and code for reproducibility is included.


**Weaknesses:** 
The main weakness is that it is not clear why following the gradient information is a sound approach for choosing the nodes to intervene. There is some attempt to do so in Appendix A but the writing is very unclear there, the authors use notation not previously introduced in the paper, and provide little to no detail of the meaning of these new variables. The statements in Remark 4 are also somewhat confusing, I cannot strictly see how that justifies using the gradient for the node selection. 

clarity,_quality,_novelty_and_reproducibility: ### Clarity

1. It is not clear if the suggested approach truly attempts to minimize the *number* of interventions. In particular, the experiments have been run with a fixed $T=100$ for all methods, for example. How can one then conclude if the proposed approach is better or worse w.r.t. such metric? Note that this is different from the total number of interventional samples, for which I believe Figures 7 and 8 correspond to. It would be enlightening to see how many iterations (i.e., interventions) each of the methods take until an SHD of zero is reached.

1. It should be precise if the method performs single- or multi- node interventions or both. In the experiments, I am under the impression that the method performs single-node interventions, but when reading Algorithm 2, its output states "select **batch** of interventions". What does that mean, a multi-node intervention? If so, how many nodes does one choose? I could not see anything on this in the paper.

1. There also should be more details on how the interventions are performed, specifically, what values/distributions are used to perform the interventions. Algorithm 2 only mentions "intervention target $i$" without giving an idea of the intervention itself.

1. I might be misunderstanding something here but, in Figures 2 and 3, the violin plots for a higher amount of data ($N=3200$) seem to perform worse/comparable to the low data regime ($N=1056$) for each of the methods. Given that a higher EAUSHD is better, I am puzzled about this behavior and would appreciate it if the authors could clarify.

### Novelty

In my perspective, the novelty of the method relies heavily on whether using the gradient is truly a sound approach for selecting nodes to intervene. This is because Algorithm 1 is not novel as it is a typical active-learning procedure, and then the sole technical contribution is Algorithm 2, for which there is no justification at all about its soundness in the main text. For instance, the title of the paper reads "trust your \nabla", and I was hoping to see a formal discussion about why one can truly trust the gradient.
summary_of_the_review: The main concerns I have about this work can be found above. For such reasons, I am inclined to propose a revision of this work.

Minor things:
* How many graphs were sampled to estimate the gradients? Have you observed the behavior of the algorithm w.r.t. the size of sampled graphs?
* Is there any particular challenge to testing continuous instead of categorical distributions?
* The references Eberhardt (2012a) and Eberhardt (2012b) are the same paper.
* Third line of second paragraph in Page 2, ,,imaginary'' -> ``imaginary''
* In some parts of the paper I was under the impression that any gradient-based approach could be used (e.g., NOTEARS), however, if I am correct, the method requires a Bayesian structure learning approach to be able to sample graphs.
* In eq.(4) $\tilde{P}$ still uses $PA_{(i,G)}$ even though it was stated that hard interventions are assumed in the paper. I got confused if that was actually the case or not.
* In Section 5, the **datasets** paragraph only mentions 3 real-world datasets and later in the same section other 3 appear (child, asia, alarm).
* In Appendix A page 14, $\Lambda_{c,\iota} \to \Lambda_{\iota,c}$.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose an approach for active intervention targeting that can be applied to existing gradient-based causal discovery methods by defining a gradient-based score to guide intervention design. The authors have conducted extensive experiments and detailed analysis, but the method overall lacks sufficient theoretical support.
strength_and_weaknesses: Strength: Good writing and extensive experiments. This paper systematically and clearly organizes the related work on gradient-based causal discovery using both intervention and observation data, and points out exactly one of the main challenges - intervention design, because interventions are very costly. The article puts forward a practical method for this problem, and conducts extensive experimental verification, supplemented by detailed analysis, which are convincing to me.

Weaknesses: The article spends a lot of time telling us the story about gradient-based causal discovery using both intervention and observation data, and also uses a lot of space to explain the experimental design and results. However, the proposed method, which should have been the core content of the article, was only described in a section less than one page, which is very difficult to understand. I don't know whether it is the reason for the author's writing or the proposed method itself, in my personal opinion, there are two theoretical defects: First, through experiments in Section 5, I agree that the score function that satisfies “GIT aims to choose the intervention target which induces the largest update of the parameters modeling the causal structure.” may be correct, but can you please give a theoretical proof? Otherwise, I cannot judge the scalability of this method; Second, why can “imaginary interventions” replace the real intervention data? Besides experimental results, can you please give more convincing theoretical proof? And I am a little confusing about how to obtain the distribution of the interventional distribution in Equation 4. You only show it is “generated assuming graph G and intervention”, but I can't find the relevant content about how to obtain it from the main text or the Appendix. Can you please explain it to me?

clarity,_quality,_novelty_and_reproducibility: Clarity: Fair. The article is overall clear, except for the content in Section 4, which is confusing.
Quality: Fair. The problem to be solved is clearly stated and the experiments are extensive, but the method itself has some theoretical defects (or perhaps the author may not explain it clearly).
Novelty: Fair. The proposed method mostly is based on existing work. The relatively novel point is the proposed score function to guide intervention design (if theoretical proof can be given).
Reproducibility: Poor. Although the experiments design is very detailed, as mentioned above, the proposed method itself is quite confusing.

summary_of_the_review: The problem to be solved in this article is relatively important, with clear overall writing and detailed experiments, but the method itself is confusing and lacks theoretical proof.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors study intervention targeting for learning causal DAGs using gradient-based methods, and consider how to exploit gradient information in experimental design as well as score-based learning. Given the amount of prior work on gradient-based methods, concerns about novelty and the lack of theoretical guarantees led to lukewarm reviews. After discussion, these concerns were not fully addressed, and so I cannot recommend acceptance at this time.
justification_for_why_not_higher_score: Concerns about novelty and the lack of theoretical guarantees that were not fully addressed during discussion
Decision: Reject