A Meta-Learning Approach to Bayesian Causal Discovery

Authors: Anish Dhir, Matthew Ashman, James Requeima, Mark van der Wilk
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: neural processes, bayesian causal discovery, transformers

Abstract:
Discovering a unique causal structure is difficult due to both inherent identifiability issues, and the consequences of finite data.
As such, uncertainty over causal structures, such as those obtained from a Bayesian posterior, are often necessary for downstream tasks.
Finding an accurate approximation to this posterior is challenging, due to the large number of possible causal graphs, as well as the difficulty in the subproblem of finding posteriors over the functional relationships of the causal edges.
Recent works have used Bayesian meta learning to view the problem of posterior estimation as a supervised learning task.
Yet, these methods are limited as they cannot reliably sample from the posterior over causal structures and fail to encode key properties of the posterior, such as correlation between edges and permutation equivariance with respect to nodes.
To address these limitations, we propose a Bayesian meta learning model that allows for sampling causal structures from the posterior and encodes these key properties.
We compare our meta-Bayesian causal discovery against existing Bayesian causal discovery methods, demonstrating the advantages of directly learning a posterior over causal structure.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730452368818

thread_type:
official_review

<thread_object>
creator: reviewer_diij

<content>
summary:
The paper introduces the Bayesian Causal Neural Process (BCNP), a novel approach for Bayesian causal discovery that leverages meta-learning to address challenges in traditional methods. BCNP learns a mapping from datasets to posterior distributions over causal graphs, thereby bypassing the need to explicitly infer the posterior over causal mechanisms and allowing for efficient sampling from high-dimensional spaces. By encoding key properties like permutation equivariance and edge dependencies, BCNP accurately approximates the true posterior and generates acyclic graph samples. The paper demonstrates BCNP's effectiveness through experiments on synthetic and semi-synthetic datasets, showcasing its superior performance compared to existing Bayesian and meta-learning models for causal discovery.

strengths:
* The paper is well-written and clear and includes a range of well-designed experiments.

weaknesses:
For me, there are two major concerns regarding the novelty and contributions of the proposed method to the community:

1. The paper presents the direct mapping from data to a posterior over graphs ($P(G|D)$) as a main advantage, bypassing the need to model causal mechanisms. However, learning a distribution over causal mechanisms is crucial for many applications, such as causal inference and treatment effect estimation [1, 2]. Could the authors elaborate on potential applications of the proposed approach beyond the discovery of the causal graph?
2. The proposed method builds on several modules from existing work. For instance, the representation of the posterior distribution as permutation and upper triangular matrices is also similar to [3]. Although the method outperforms baselines in graph-based metrics in causal discovery, the specific novelty remains unclear, particularly given the first concern.

[1] Emezue, C. C., et al. (2023). Benchmarking Bayesian Causal Discovery Methods for Downstream Treatment Effect Estimation. arXiv, 2307.04988. https://arxiv.org/abs/2307.04988v3

[2] Schölkopf, B., et al. (2021). Towards Causal Representation Learning. arXiv, 2102.11107. https://arxiv.org/abs/2102.11107v1

[3] Cundy, C., et al. (2021). BCD Nets: Scalable Variational Approaches for Bayesian Causal Discovery. arXiv, 2112.02761. https://arxiv.org/abs/2112.02761v1

questions:
1. As my main concerns, I would appreciate the authors to elaborate on the abovementioned weaknesses.
2. Given that the proposed method does not utilize interventional data, using CPDAG E-SHD (as used in previous papers such as in BayesDAG) might be more reflective of the performance of the models compared with E-SHD.
3. Have the authors tried any experiments on low data regime where the number of samples is close to the number of nodes? This is particularly important since Bayesian Causal Discovery is most desirable in low data regime where $n \approx d$. Do you have an intuition about how the proposed model performs in these particular settings?
4. In several experiments, other baselines such as BayesDAG and DiBS achieve a better E-SHD compared to the proposed method. Could you elaborate on why this happens?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730728800175

thread_type:
official_review

<thread_object>
creator: reviewer_fwq7

<content>
summary:
In Bayesian causal discovery, the goal is to approximate the posterior over graphs. Since computing the posterior over graphs requires computing posterior over functional parameters, previous approaches have tried to approximate the posterior over function parameters.  This paper uses neural processes to approximate the posterior over graphs directly by learning a map from the dataset to the set of distributions. The neural process is an encoder-decoder network that claims to sample, permutation-invariant, acyclic graphs that capture edge dependencies.

strengths:
The paper proposes a technique that can directly sample from the posterior over graphs. The main contribution of the paper is in the architecture of the encoder-decoder network that captures necessary properties like the permutation invariance by cross-attention, and ayclic DAGs and edge dependencies by sampling the decomposition of a DAG into permutation and lower-triangular matrices. The comparison of the proposed model is done with existing meta-learning approaches and Bayesian approaches using multiple metrics. The proposed model is shown to be competitive on synthetic and simulated datasets. The paper is written clearly and flows well.

weaknesses:
My main concern is that of this being an incremental contribution. The framework is not new and the only contributions I see are a change in how permutation invariance and acyclicity is incorporated. Please let me know if I am missing something else. I have a detailed list of questions below whose answers might help strengthen the paper.

questions:
1) The ideas that the paper uses to impose acyclicity and permutation invariance seem applicable in general. Can existing meta-learning approaches be modified to get the same? While this is not the focus of the paper. It would make it stronger to highlight it if that's the case.

2) The experimental validation for the case when the true posterior is known, is done only for the two-variable case. Is it possible to scale this up? 

3) How can the posterior over the graphs be used for inference on that of the functional parameters? 

4) How is the performance on Gene Regulatory Network simulated data like SERGIO? I believe it would be useful to add that in given its importance. 

5) The paper says that the metrics average over the posterior. I believe this is a problem with any metric. Are there alternatives that the authors propose?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730877919301

thread_type:
official_review

<thread_object>
creator: reviewer_cqb9

<content>
summary:
This paper introduces a Bayesian meta-learning model for causal discovery, named the Bayesian Causal Neural Process. This "meta-learning" algorithm learns a model from pairs of datasets and their associated causal graphs. Specifically, the model incorporates a neural process to address model uncertainty and employs an encoder-decoder architecture carefully designed to achieve desirable properties for causal discovery in a meta-learning context. These properties include permutation invariance to sample order, permutation equivariance to nodes, and acyclicity, achieved by decomposing the adjacency matrix into a lower triangular matrix and a permutation matrix—some of which build on existing approaches. The authors demonstrate the empirical superiority of this model over other explicit Bayesian and Bayesian meta-learning models

strengths:
- As summarized in Table 1, this approach implements several key desiderata for a Bayesian meta-learning model more effectively than existing alternatives.
- The paper is well-written, allowing readers to easily follow the motivation and desiderata of Bayesian meta-learning models.

weaknesses:
- While I appreciate the overall architecture of the model, it is challenging to identify which components provide truly novel technical contributions. For instance, the results on pages 4 and 5 appear to largely rely on existing findings, and, if I'm not mistaken, page 6 includes results from Annadani et al. (2024). The contributions of this paper are not clearly or explicitly articulated.
- The Bayesian prior is learned through pairs of datasets and directed acyclic graphs (DAGs). From my perspective, learning P(X,G) is likely very challenging. Indeed, Appendix B.1 notes that, for a case involving two variables, the model was trained on 200,000 datasets. I understand that Section 4.1 is intended to evaluate the posterior, but in Appendix B.2, 500,000 datasets are used for training. This information would be better suited for the main text, and details on computational resources should be transparently reported there as well (they are not provided even in the appendix).
- I find it difficult to be fully convinced by the meta-learning approach to causal discovery. While I understand the importance of establishing a strong Bayesian prior, can we realistically expect access to hundreds of thousands of dataset-graph pairs?

questions:
- For DiBS and BayesDAG, the authors mention that they “keep all other hyperparameters to their preset values.” What if these hyperparameters were tuned? Would the results differ significantly? For instance, DiBS and BayesDAG perform well on the Syntren dataset for certain metrics, so tuning may further widen this performance gap.
- In the caption, it says, “Each dataset contains D nodes and N samples.” Does this imply that all datasets must be of the same size? What if the sample sizes vary, such as 100, 200, or 500? Should N be set to 100, or can larger datasets (e.g., 200 and 500 samples) be split into multiple parts (e.g., 2 or 5 parts) to train the model? In that case, should there be weighting adjustments (1/2, 1/5), also training perhaps only for data-related parameters and not for those related to the graph prior?
- If there is no training involved, does your model function like a standard Bayesian causal discovery method with a uniform prior?
- Each variable might have distinct functional priors, such as some being linear, others non-linear, discrete, or having different means. Does the encoder “implicitly” reorder these variables (perhaps using attention mechanisms) to align them by their functional type? I’m trying to conceptually grasp how the encoder handles such diverse functional characteristics under the hood.
- What if permutation equivariance among nodes was not enforced? Would this potentially improve performance, especially if nodes are ordered consistently between training and test data?
- In Line 382, could you clarify what is meant by “correlations between edges” when dealing with just two variables, X and Y, with at most a single edge?

Minor comments
- It would be better to have some annotations in Figure 2 (given that the figure takes lots of space)
- Not quite sure whether it is sufficient to say the dependence in permutation and lower triangular binary matrices is modeled because they share the same representation R0.
- 047, 137 citep for Wenzel et al. 
- 115 citing PC, GES might be good (given you have some space for 119)
- 141 Bayesian metal → meta
- 305 Is the grammar correct? (The matrix … and thus … )
- 340 space before Mena
- 389 In section section 4.2
- 389 its 
- 414 space after comma
- 480 enable
- 485 period
- BayesDAG is not highlighted in Table 4 for Linear data/SHD
- Figure 2 shouldn’t o be the input to Theta? or onto the edge between R^L1 to Theta?
- Sorry for suggesting my preference here but can we align Theta and Phi (parameters, vertically) and align Qs and As with the same horizontal line as Theta and Phi, respectively, and put Gs at far right.

(I can change to weak accept (or accept) depending on the response and others' reviews.)
</content>
</thread_object>

</thread>
<thread>
creation date:
1731044263201

thread_type:
official_review

<thread_object>
creator: reviewer_xbxp

<content>
summary:
This paper proposes the Bayesian Causal Neural Process (BCNP), a Neural Process-based framework for learning a posterior distribution over causal graphs given a dataset $P(G \mid X)$ . The proposed method is scalable and captures distinctions between graphs within the same Markov equivalence class. Since BCNP learns causal structures across multiple (synthetic) datasets, it is considered a meta-learning approach.

strengths:
1. This paper provides a well-structured and self-contained summary of prior works in Bayesian causal discovery, allowing readers to clearly follow the evolution of Bayesian approaches for learning causal graphs. As a reviewer, this summary enables me to track the advancements in the field and understand the recent progress in addressing challenges like scalability and uncertainty in causal inference.

2. The paper clearly outlines its unique contributions in Table 1, highlighting significant advancements over existing methods. The proposed model addresses critical aspects such as acyclicity, permutation invariance, and edge dependencies—all of which are critical for accurate causal graph learning. These improvements over prior approaches directly enhance the reliability and precision of inferred causal structures, making the contributions both clear and impactful.

3. The proposed BCNP is particularly attractive due to its innovative use of Neural Processes to directly sample the posterior distribution $ P(G | X) $. This approach enables BCNP to achieve scalability and uncertainty-aware inference by learning across tasks (= datasets), allowing the model to efficiently adapt to new datasets while maintaining a robust representation of causal uncertainty. This combination of scalability, direct posterior sampling, and adaptability positions BCNP as a strong advancement over existing Bayesian causal discovery methods.

weaknesses:
1. One of the main limitations of this paper is the assumption that there are no latent confounders between variables. This assumption may limit the applicability of the method to real-world datasets where unobserved confounders are common. 

2. I think this method is heavily relying on the assumptions on the choice of modeling class of functions F, the types of graphs and the noise model. Unlike fully nonparametric approaches like the PC algorithm, which do not impose strict functional or noise assumptions, BCNP’s effectiveness depends on these choices being well-suited to the data.

3. The proposed framework may be sensitive to hyperparameter settings and architecture choices, such as the hyperparameters for the encoder-decoder network, or the types of prior distributions. Fine-tuning these parameters is often data-dependent, which could reduce the model's robustness and generalization capabiltiy.

questions:
1. Is it nontrivial to consider the setting where latent confounders between variables exist? 

2. Is the proposed method computationally efficient when a function class is chosen as neural networks? 

3. How sensitive the proposed framework is regarding to the sparcity of the graph? 

4. Are there any interesting example where the Bayesian approach beats the Markov-equivalence-based method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732190445573

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response

comment:
We thank the reviewers for taking the time to review our paper.
We appreciate that they consider the contributions "both clear and impactful" `#xBXp`, that the paper is well-written `#DiiJ,#fWq7,#CqB9`, and that our proposed model is competitive `#fWq7`.
We also appreciate comments that the experiments were well-designed `#DiiJ`, that our approach "implements several key desiderata" `#CqB9`, and that our work is a "strong advancement over existing Bayesian causal discovery methods" `#xBXp`.


### Contribution 

We propose the first model that combines meta-learning and a fully Bayesian approach to causal discovery to parametrise the full Bayesian posterior over causal structures. 
Our method is the **only one that provides not only accurate samples, but also samples from the correctly parametrised space** (acyclic graphs with depdendent edges) with permutation equivariance over nodes baked-in (Table 1 and Section 4.1).
We thus show that following the consequences of a fully Bayesian view of meta-learning prescribes changes that actually leads to better performance (results in Section 4). We believe this is an important novelty.

The _promise_ of meta-learning is that it can remove the accuracy bottleneck of approximating an explicit posterior. So if meta-learning is working correctly, it should perform better than existing explicit Bayesian methods.
The existing methods that have attempted meta-learning approaches before (AVICI, CSIvA) were only concerned with parametrizing a single graph rather than the full distribution.
This lead to methodological choices (see L155, Table 1) that mean that **these methods do not parametrise a distribution over the correct object (acyclic graphs with dependent edges) and result in innacurate approximations of the Bayesian posterior**.
This is shown in Section 4.1, and is evident by the fact that these methods (AVICI, CSIvA) **do not outperform explicit Bayesian models** (e.g. Table 2).

In contrast to the above, our method targets an approximation of the full Bayesian posterior by **proposing a decoder (L264)** that captures important properties of the posterior (L155, Table 1) and we provide a **novel loss** to train the network using meta-learning (section 3.3). We then show that this provides samples from the correctly parametrised space (acyclic graphs with dependent edges) and **outperforms explicit Bayesian models** (Table 2). 
Our changes, prescribed by the fully Bayesian view, are thus important to **deliver on the promises of meta-learning**.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732889169477

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised Manuscript Uploaded

comment:
We thank all the reviewers for their constructive feedback that have helped improve this paper.
 
We have uploaded a revised manuscript, with changes marked in red, that hopefully address all the reviewers concerns.

Should any further questions or concerns arise, we are happy to provide clarification during the remaining discussion time.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors propose a heuristic algorithm to approximately learn the posterior distribution over causal graphs given observational data, by posing the problem as a meta-learning objective. Compared to learning the ML or MAP estimate DAG, this objective is much harder but I agree can carry value for certain downstream tasks. So a useful heuristic is valuable, which seems to have been demonstrated via experiments and convinced the reviewers at least to some degree.

Based on my reading, I would say please populate Bayesian causal discovery literature. There is a very large body of work for Bayesian causal discovery but the current citations are very, very limited. Citing any related work is just as important as any other part of paper-writing since not citing properly breaks the connection the next generation of researchers we hope will have to the past.