Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors proposed to efficiently learn a causal structure through interventions under a causal sufficiency assumption. The key idea is how to design experiments (intervention) in a way to figure out the causal structure where they adopted Bayesian in that (i) use of an (approximate) posterior distribution over candidate graph structures and parameters and (ii) (approximate) Bayesian Optimal Experimental Design in performing experiments. More specifically, what variable and which value to intervene to get a sample is determined with respect to maximizing mutual information between the parameters (structural and functional) and observation, where they adopted Bayesian Optimization. Single and Batch designs are proposed with two options for Batch, Greedy and Softmax-based.

strengths_and_weaknesses: Strengths
- The paper focuses on scaling up Bayesian causal discovery, and there are several places where the authors tried to incorporate recent works to approximate intractable quantities.
- The efficiency of the algorithm is demonstrated empirically.

Weaknesses
- The main idea of the paper looks like a collection of existing methods (BALD, DiBS, BO, Soft Top-K). Surely, putting them into a piece neatly to solve a problem would require expert knowledge of the ingredients. 
- Tests on a synthetic graph, where the expected number of edges per vertex is one, will likely result in multiple independent components. Experiments on small/medium size graphs with a bit more dense edges are desired. Furthermore, tests under unobserved confounders (i.e., violation of assumptions) would be a nice addition.
- Noise variance seems small (but is an understandable choice given that it would not likely affect comparisons among methods).
- The authors misused the term SCM in many places. SCM consists of functions, and it induces a causal graph (not part of it). The graph can be identified, and parameters can be learned, but SCM won’t be typically identified. (Line 66, recovering the original SCM, Line 44 consists of a DAG structure … )
questions: - Please define beta in Line 193-4 in the main text.
- Gaussian noise is assumed, but it is unclear where the assumption is explicitly used. It doesn’t seem to be related to Line 188-189
- What is the main novelty of the *-CBED? Also, it would be nice if you could emphasize the contributions of the paper?

limitations: - Causal sufficiency assumption and that all variables are intervenable. These two guarantees identifying a causal graph.
- Only one variable is intervenable at a time.
- Assumption 4 prohibits the use of discrete variables.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a bayesian intervention design method for single-node interventions, including the target values, in causal models with additive noise.
strengths_and_weaknesses: *** EDIT after rebuttal *** I think the authors did a great job in the rebuttal and answered most of my (and other reviewers') concerns (including the similarity with previous work on GPs for BOED [49]), so I changed my score to weak accept.

Strengths:
- Intervention design is an important, and in my opinion, understudied problem
- The paper presents an interesting and to the best of my knowledge theoretically sound method

Weaknesses:
- The paper seems to ignore a lot of related work on intervention design, mostly focusing on differential causal discovery. This is a bit weird since several of these these papers are cited in [1] and [38] and the whole terminology of this paper is based on related work that isn't cited, e.g. intervention design with a variety of settings (active/passive, multiple targets, budgeted, latent variables and also Bayesian): https://proceedings.neurips.cc/paper/2015/hash/b865367fc4c0845c0682bd466e6ebf4c-Abstract.html, https://proceedings.neurips.cc/paper/2020/hash/f57bd0a58e953e5c43cd4a4e5af46138-Abstract.html, https://proceedings.mlr.press/v70/kocaoglu17a.html, https://proceedings.neurips.cc/paper/2017/hash/291d43c696d8c3704cdbe0a72ade5f6c-Abstract.html, https://proceedings.neurips.cc/paper/2021/hash/0b94ce08688c6389ce7b68c52ce3f8c7-Abstract.html, https://papers.nips.cc/paper/2019/hash/5ee5605917626676f6a285fa4c10f7b0-Abstract.html  and many more

- It seems to be also missing another even more related work on optimal experiment design with GPs: https://openreview.net/forum?id=Ih_ogoAw5G


questions: Some of the related work https://papers.nips.cc/paper/2019/hash/5ee5605917626676f6a285fa4c10f7b0-Abstract.html shows that maximising the expected information gain might not be the optimal strategy in their setting, so I was curious about how this would apply in the setting of this paper

In general, I would be curious about how this work compares to all of the missing related work.
limitations: The paper proposes an interesting approach, although limited to single-node interventions and ANMs (which seems a strong assumptions). One big limitation in my opinion is that it fails to cite most of the related work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a Bayesian approach to learning causal relationships from a mix of observational and interventional data. Compared to existing approaches, the proposed method can cope with the nonlinearity of the causal relationships and can select both target and value to intervene with to facilitate more efficient learning.
strengths_and_weaknesses: Strengths:
* The paper tackles a long-standing problem in causal discovery. Learning from a mix of observational and interventional data is an interesting and relevant setting in causal inference and discovery.
* The proposed approach is reasonable and is capable of handling a few challenges as mentioned in the paper including nonlinearity and choosing both the target and value to intervene.
* The paper is well written and easy to follow.

Weakness:
* The proposed method is somewhat non-surprising and there is a lack of formal (theoretical) justification of the performance of the proposed method.
* While experiments on both synthetic and real-world data demonstrate the performance of the proposed method compared to alternatives. It is not clear to me whether the proposed method has been compared to sufficient and competitive enough alternatives. The use of only one real-world data, while understandable, is also somewhat restrictive. The paper can also benefit from a robustness study evaluating when the key assumptions in the paper are violated.
questions: I don't have additional questions other than my concerns expressed in the weakness section.
limitations: It is not clear whether the authors have sufficiently discussed the limitations of the paper. Assumptions such as causal sufficiency may be considered as limitations.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper considers Bayesian optimal intervention selection for active causal discovery. The problem setting is causally sufficient causal graphs under additive noise models and atomic interventions. Importantly, both the target nodes to intervene and intervention values that will be assigned to the target nodes are optimized in a batched setting. Mutual information between the observations and the parameters is optimized in Bayesian Optimal Experimental Design context. Variational inference is used to approximate the posterior over SCMs. After thorough derivations, Monte Carlo approximations are used to estimate MI for single intervention design. For selecting the target value over a continuous domain, a Gaussian Process is used along with Upper Confidence Bound mechanism. Furthermore, two approaches (greedy and top-k selection) are proposed for batch design. Experiments on synthetic datasets demonstrate the effectiveness of each piece of the proposed method. Experiments on semi-real datasets also show promising results.
strengths_and_weaknesses: The paper is well-written, with a sufficient amount of details in the main body and a lot of secondary information on the appendices. The proposed method consists of many components, some borrowed from the existing literature such as GP-UCB and DiBS posterior model. The paper does a nice job putting them all together coherently and explaining the steps. 

The considered problem is general, and I believe the main contribution is this generality: allowing non-linear systems, scaling up to 50 nodes (though in quite sparse graphs), and the ability to set the intervention values rather than just finding targets. 

Especially, the last point seems important. Although there are other information-theoretic approaches to causal discovery, to my knowledge, the present paper is the first one that studies the effect of the target value and supports its claims through extensive experiments.

The only major weakness is the assumption of atomic interventions. For instance, ABCD (Agrawal et al., 2019) allows multiple nodes to be intervened simultaneously. Although the proposed batch solutions are helpful for faster causal discovery, the assumption of atomic interventions can be a limiting factor.
questions: Regarding the weakness above, do you see an extension to non-atomic interventions possible within the proposed framework?

On a smaller note, L53-54 states: “existing work on active causal discovery focuses exclusively on selecting the intervention target”. Although it solves a different problem, "Matching a Desired Causal State via Shift Interventions" (Zhang et al. 2021 NeurIPS) can be a relevant work that also aims to find the intervention values.
limitations: Assumptions of the paper, e.g. Gaussian noise and atomic interventions, are clearly stated in the paper. Perhaps the second point can be addressed in a bit more detail.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes to solve the problem of causal discovery through an experimental design method which selects interventions based on the mutual information between an intervention and the SCM. As evaluating the mutual information in closed form is not possible, the authors propose to use a BO algorithm to identify its maximum which corresponds to the experiment that should be evaluated next. The authors additionally extend the method to cases in which the experimenter wishes to select a batch of data points.
strengths_and_weaknesses: Overall I think the problem the authors are addressing is important. However, in terms of novelty, this paper seems to be an extended and detailed version of [42] where all the main ideas were already presented. One important difference is in the selection of the intervention value together with the intervention set. It would be great if the authors could provide a list of contributions to highlight the novelty of the paper. A list of detailed comments and questions is given below.

**Background section.** Even though I am very familiar with the topic discussed and the notation generally used in causality, I found the background section difficult. I believe this is due to the missing definitions and typos. 
- Line 106-108 is confusing because of the typos. Do you need a $i$ for $f$, $\gamma$ and $\sigma$? Is $\theta$ a collection of these parameters for all $i$’s?
- In general, causal discovery refers to the problem of learning the DAG and not the functional relationships. While both are connected in this paper you seem to be addressing these two problems jointly. Is my understanding correct? Can you clarify this point?
- The notation is a bit imprecise and a few things are not defined in section 2. For instance E at line 85. In addition, you need to define what you mean with the do operator and what $x_w$ is. What are $\phi$ and $\Phi$ at line 127? The parameters of the posterior distribution on the SCM (which was also not introduced)? 
- What is VCN?
- First formula after line 89, typos in the parenthesis I guess it should be $do(X_w = x_w)$. Same for parenthesis in Eq 1.
- Add at least one reference for the background section, for instance when you define SCM. SCM generally refers to $(V, U, F, P(U))$ (U are the exogenous variables). The SCM is associated with a DAG. 
- Line 106 -> $f_i$
- Provide at least one reference for the definition of mutual information. It seems to me that the first terms should only be conditioned on D.

**Method section.** 
- Clarify if it is possible to observe all variables after implementing an intervention. Is $Y = V\backslash X_j$?
- How is the prior on $\Phi$ defined? This is the joint prior distribution on the graph and the parameters of the functional relationships and the noise variances? This is never explicitly stated. 
- Even though there exists a discrete set of intervention variables that we can manipulate (especially given Assumption 4) the number could be very large depending on the size of the graph. I don’t see how this makes the method scalable.
- I think you should compare finding the optimum of the MI with BO with a grid approach that finds the optimum by numerically evaluating the function MI for every j on a set of points. I would be interested in seeing what is the reduction in terms of function evaluations that you get with BO.
 

**General comments.** 
- Assumption 4. With target to you refer to the intervened variable? Where is this assumption used? What is preventing you from doing interventions on more than one variable? Is it cause exploring interventions on all subset of variables would make the approach less scalable?
- Assumption 3. Where is this assumption used?  Is this assumed to be able to write the likelihood?
In general it is not clear why and where these assumptions are needed.
- The authors speak about causal discovery at scale. However, the approach presented requires enumerating all possible DAGs on which the prior needs to be defined. How is this problem addressed?
- I guess selecting the intervention level is relevant when the intervened variables are continuous. However this is not stated anywhere in the paper. 
- I am confused about the use of observational and interventional data. While the observational data are used to compute (4) (via DiBS?) how are the interventional data collected sequentially used? Are these just included in $\mathcal{D}$? The authors should clarify how exactly $\mathcal{D}$ is used in the computation of (4) and how the interventional data are treated.
- Are Alg 1 and 2 just giving one step of the overall algorithm? For how many steps is this repeated? I could not find this mentioned for the experiments presented.
- Regarding scalability, as mentioned above, it would be great if the authors could clarify what makes this method scalable with respect to the existing ones. The fact that the experiments are on graphs with a large number of nodes is great but is the capability of dealing with a large number of nodes a feature of the method or of the implementation?

questions: See section above.
limitations: Yes, this is discussed in the supplement. I cannot identify additional potential negative societal impact for this work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Causal discovery (CD) is an important field of research that can help find the causal relationship of the variables in a system. Only using observational data for finding causal structure, in the form of a Directed Acyclic Graph (DAG), is insufficient as there could be multiple DAGs that are Markov Equivalent. To resolve the uncertainties, interventional data can be incorporated to disambiguate between hypotheses. This work proposes a novel method based on Bayesian Optimal Experimental Design (BOED) that efficiently selects both intervention targets and values via the Bayesian optimization approach to find the true nonlinear structural causal model. Batch strategies, i.e., Greedy-CBED and Soft-CBED, are developed to select the most informative subset of interventions and values, as opposed to a single intervention at a time, to find the SCM.

Throughout the experiments, the authors demonstrate that their work can achieve better performance than existing methods on both synthetic (Erdos-Renyi, Scale Free) and real-world datasets (single-cell regulatory networks).

strengths_and_weaknesses: 
Strengths:
i) This paper is written exceptionally well; details of notations, definitions, equations, and derivation of theoretical results are provided.
ii) The proposed method is novel, as well as the proposed work, can find nonlinear SCMs and can select both intervention and values, in prior works are limited to linear models and can only select the intervention target.
iii) This work provides a comprehensive ablation study (Figure 3) so that we clearly better understand the effect of each adding component (such as the effect of selecting the intervention values, batch v.s. non-batch, and soft-CBED v.s Greedy CBED, on the E-SID performance. )

Weakness:

I do not particularly find a significant weakness in this work.
questions: 
i) Could the authors elaborate what is the newly introduced random variables $$ \Phi $$ in equation (3) when measuring the mutual information?

ii) What is the initialization strategy for the Gaussian Process model?

iii) What would happen when the number of variables in the SCMs is higher than 50?


limitations: Possibly the authors could provide some examples (such as the single-cell gene regulatory network) when the method fails to identify the true SCM.
review_title: null

Meta-review: metareview: The paper proposes a Bayesian optimization strategy for causal discovery under causal sufficiency and additive noise. The main point is to choose interventions that maximize mutual information between parameters and observations. The procedure combines techniques from several existing methods. The authors have successfully addressed questions raised by reviewer PxXf about related work. Several reviewers praise the writing (SUP9,3PzT,25Ya). Overall, this is a strong paper, with atomic interventions as the main limitation (see reviewers 25Ya,h91K)
confidence: Certain
Decision: Accept