Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a framework to simultaneously infer a causal graph as well as the specific causal model, including specific relationships between the variables from observational data. 
To do so, a bayesian approach for joint inference is outlined in the contribution.
The approach also enables learning from additional, new experimental data - in sense of a standard active learning methodology.
A experimental setup justifies the framework as it can show the success of the algorithm.
strengths_and_weaknesses: 

## Strengths

* The problem is well structured, with precise mathematical notation to distinguish certain problem classes

* The choice of the problem - i.e. to extend inferring from only observational to observational and experimental data - is somewhat novel in a sense, that most research in the field is focussed on inferring from observational data alone

* Recent advances, such as the DiBS approach, are considered and brought to use in context of the framework.

* An accompanying codebase provides a computational implementation of the approach, including jupyter notebooks which facilitate understanding the "how-to" of the codebase.
    the problem is well formulated and the manuscript is clear and well written

    the formal style with highlighted Theorems as well as the choice of notation is consistent with the relevant literature of the field

    The real world datasets are well chosen and fit the problem, for the experiments several methods for comparison are included

## Weaknesses

* As the approach outlined is somewhat experimental - in that a "real" active learning approach would necissate manipulation / intervention into the setting - I would have expected a stronger / more elaborate discussion setting, 


*   I would have expected to have more discussion of state-of-the-art / comparable methods. The methods for comparison introduced only later could be already discussed theoeretically in an earlier section

* The idea of causal reasoning as formulated in the paper (i.e. via do-formalism interventions) would benefit from some more explanation, ideally using examples.
questions: * Personally, I consider the terminology using exogenous and endogenous variables for latent and observed variables (L. rather confusing (Def. 1)  somewhat confusing as my understanding is more aligned with variables determined by the model and variables not determined by the model (see https://en.wikipedia.org/wiki/Exogenous_and_endogenous_variables )

 
limitations: Limations, especially regarding feasability and tractabilty, are well discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces a new framework, ABCI, for end-to-end Bayesian causal inference. Starting from no data, ABCI iteratively designs experiments (interventions) to maximize the information gain for the target causal query. ABCI maintains uncertainty over the causal model over time by maintaining a set of latent particles, corresponding to causal graphs, which are used to compute the information gain (through a Gaussian-process based marginal likelihood). ABCI is distinguished from previous works by 1) targeting (and optimizing interventions) for a specific causal query $q(\mathcal{M})$, which can be any function of the SCM; and 2) modeling linear non-Gaussian SCMs. Empirically, ABCI methods are shown to provide improved performance compared to random-intervention baselines. 

strengths_and_weaknesses: This is a solid work which tackles the important problem of designing optimal experiments for answering causal queries, which are not identifiable from observational data. Bayesian inference is a principled approach through which to maintain uncertainty, and the ABCI framework is a sensible and well-engineered end-to-end solution for inferring a target causal query. In that sense, ABCI fills an practical gap that previous works did not fully address.

On the positive side, ABCI brings together a number of previous ideas, such as maintaining uncertainty through graph samples, maximizing information gain, and using Gaussian processes to model non-linear relationships with tractable marginal likelihoods, to form a useful end-to-end framework. The paper investigates targeting specific queries by developing utility functions (information gain) for arbitrary functions of the SCM. The benefits are demonstrated experimentally, where $U_{CR}$ is shown to converge faster than $U_{CML}$ on the specific causal query.

My key concern is that there does not appear to be significant novel components introduced in this work; nor does there appear to be significant novelty in the approach to integrating the components: for instance, computation appears to be the key limiting factor, and there does not appear to be theoretical advances in this regard. Alternatively, comprehensive empirical analysis could provide justification and insight regarding the efficiencies and bottlenecks in the ABCI pipeline, e.g. ablation studies considering different approaches to each step. 

More technically, I have some concern over the nature of the uncertainty representation maintained by ABCI. In particular, ABCI maintains a set of latent particles $\textbf{Z}$, corresponding to graphs, and it appears that the GP marginal-likelihood and posterior is computed fresh in each timestep $t$. This is concerning for two reasons. Firstly, the number of samples maintained by DiBS (Appx D.7) is chosen to be very small (5), which seems unlikely to provide a sufficiently diverse uncertainty. Secondly, the idea of maintaining a set of graphs for evaluating information gain already appeared in Agrawal et al. [1], and the method proposed appears to just use this without taking advantage of modeling the full SCM and targeting a specific query (e.g. maintaining GP posterior also), which is the key novel component of this work.  

From a presentation standpoint, the problem is well motivated and the prose is generally of very high-quality; I found the paper to be very pleasant to read. The description of Bayesian causal inference is thorough, perhaps the clearest that I have seen in the literature. However, in my view the paper goes too far in this direction; too much of the main paper is describing (the use of) existing work, without spending enough time highlighting the novel contributions, and perhaps even more importantly, justifying the choices being made (see Questions below). As a result, just reading the main paper, it’s difficult to tell what are the key novel insights.  

Overall, I think the work is solid and a good practical contribution, but I am not entirely convinced that the novelty of the framework itself, or the analysis and insights thereof, is sufficiently strong.

questions: Key questions:
- What are the essential novel insights/results that we should take away from this paper, that were not previously known? 
- What is the justification (e.g. computational) for the choices made? For instance, can other models be used in place of Gaussian processes? Why was Bayesian optimization chosen? Can alternative Bayesian structure learning methods be used in place of DiBS?
- In Appx D.7, what likelihood $p(x|z)$ is used for SVGD, and why? Is it the GP marginal-likelihood?
- In Appx D.7, it is mentioned that only 5 latent particles are maintained while using DiBS; intuitively, this seems insufficient to maintain enough graphical uncertainty. Have the authors tested the effect of changing this hyperparameter?
    
Other:
- Is it possible to use gradient-based optimization for the utility (Eqn 4.11)? E.g. score function estimator?
- Have the authors considered using a Laplace approximation for the GP hyperparameter posterior? 
- In line 268, it is shown that simplified utilities are available for cases where the query is either the full graph, or full SCM. Are there any practical benefits (e.g. in terms of estimation) in these cases?

limitations: The authors have appropriately addressed the limitations through discussion.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a Bayesian approach to causality that combines causal discovery (of the causal graph and mechanisms) and inference (about a question of interest). They use GPs with additive noise for the mechanisms to be able to have analytics posteriors for them. They use the previously introduced DiBS framework to optimize over a soft version of the graph (using a continuous latent variable Z that conditions independent edge choices) using a variational method. They propose a generic method for handling different kinds of causal inference queries (e.g. to discover the graph vs to discover a particular edge or estimate a causal effect on some variable given some hypothetical interventino). An active learning Bayesian experimental design algorithm is proposed which exploits the estimated posteriors to estimate and optimize information gains wrt possible interventions.
strengths_and_weaknesses: Pros:
- The approach is very appealing because it is highly principled, based on a Bayesian framework that is important to 
- It is original in addressing together the causal discovery and the causal inference questions, something very important in because in general (especially for finite data and the generally unidentifable settings) there will be many more than one graph G that is consistent with the given data.

Cons:
- the additive noise and GP assumptions are convenient but it would be good to think about getting rid of them (although this is clearly for future work, this paper is already quite dense)
- the computational complexity of the proposed approach should be clarified (see below): the fact that experiments are done with d=8 variables and starting with n=5 examples and minibatches of size 3 suggests that computational scaling may be an issue that should be better covered in the limitations section.


questions: 1. first sentence after eq 3.3 may be confusing for some readers, since Y may be a function of some condition or intervention; what the authors probably mean is that q(M) is a posterior that concentrates mass on a single distribution (whereas 3.3 is a mixture and P(Y|M) is not a Dirac).
2. line 203: why do we need to distinguish between root nodes and non-root nodes? (see also next 2 questions)
3. Eq 4.4: why can we factor p(f,sigma|G) for NR but not for R?
4. line 212: why is this true for NR but not for R?
5. sec 4.1: do we need to solve (i.e. O(n^2) time) for the GPs of all the nodes after each sample of graph G? is that the computational bottleneck?
6. how does the overall algorithm complexity scale with the size of the graph and the number of examples? what are the computational bottlenecks in the overall procedure?

limitations: - see the above questions about computational complexity, which should be discussed in the limitations section too
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes Active Bayesian Causal Inference (ABCI) that is a fully-Bayesian active learning framework for integrated causal discovery and reasoning, which jointly infers a posterior over causal models and queries of interest. They implemented ABCI for the class of nonlinear additive Gaussian noise models using Gaussian processes. To parametrize the combinatorial space of causal graphs and having tractable posterior, they applied a continuous latent probabilistic graph representation. 
strengths_and_weaknesses: They studied a practical and important question that is answering to a causal query without knowing the causal graph. The paper is well-written. 

The proposed approach, unlike most exiting methods, does not learn the causal structure and then answer to the causal query of interest. This makes it both novel and more data efficient. 

Although the proposed ABCI is a general framework but as it is mentioned it is not a trackable algorithm for general causal models and hence it is implemented for nonlinear Gaussian models. How can this algorithm be used for model classes that contain discrete variables?

Applying information gain framework for selecting the next best intervention is standard in Bayesian approaches but it is suboptimal as it is greedy. Is there any guarantee of how far the solution is from the optimal solution (e.g., if the objective is sub-modular then it is a constant approximation)? Moreover, as it is suggested by the experiments, there is not so much gain compared to the random selection method considering the additional computational cost of design experiment line in ABCI algorithm.

The overall method is computationally intense which makes its scalability questionable. A discussion on the computational complexity of ABCI would be interesting. The experiments are also for small graphs. 

As it is mentioned, this work is built on causal sufficiency which is a limiting assumption and in most real-world applications it does not hold. Having said this fact, suppose that the target causal query is structure learning. In this case, having observational data, we can learn the essential graph of the corresponding MEC. On the other hand, the authors of “Learning Causal Graphs with Small Interventions” show that designing minimum number of interventions with limited size to learn the causal structure is NP-hard. When the actable set of variables are limited in size, then it is interesting to see how ABCI algorithm performs compared to the aforementioned work.

Under the causal sufficiency and using the rules of do-calculus, all interventional queries become identifiable from observational data given the causal graph. It is important to see whether ABCI algorithm in the learning interventional distribution task learns the whole graph or only a part of it before finding the interventional distribution of interest. 

Minor:
Introduction: "Additional structural assumptions (e.g., linearity)". Linearity is not a structural assumption. It is an assumption on the underlying generative model. 

questions: Please see the above comments.
limitations: Please see the above comments.
review_title: null

Meta-review: metareview: This paper proposes a Bayesian active learning framework for integrated causal discovery and reasoning. In the framework, one sequentially designs experiments that are maximally informative about a target causal query, collect the corresponding interventional data, update the  beliefs, and repeat. Through simulations, the authors have demonstrated that the approach is more data-efficient than existing methods that only focus on learning the full causal graph. This allows one to accurately learn downstream causal queries from fewer samples.  All the reviews are positive. 
confidence: Certain
Decision: Accept