A Meta-Learning Approach to Bayesian Causal Discovery
Authors: Anish Dhir
Matthew Ashman
James Requeima
Mark van der Wilk
Venue: ICLR 2025 Poster (2025)
Keywords: neural processes
bayesian causal discovery
transformers
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We propose a method that allows for sampling from an approximate posterior over causal structures using neural processes.

Abstract: Discovering a unique causal structure is difficult due to both inherent identifiability issues, and the consequences of finite data.
As such, uncertainty over causal structures, such as those obtained from a Bayesian posterior, are often necessary for downstream tasks.
Finding an accurate approximation to this posterior is challenging, due to the large number of possible causal graphs, as well as the difficulty in the subproblem of finding posteriors over the functional relationships of the causal edges.
Recent works have used Bayesian meta learning to view the problem of posterior estimation as a supervised learning task.
Yet, these methods are limited as they cannot reliably sample from the posterior over causal structures and fail to encode key properties of the posterior, such as correlation between edges and permutation equivariance with respect to nodes.
To address these limitations, we propose a Bayesian meta learning model that allows for sampling causal structures from the posterior and encodes these key properties.
We compare our meta-Bayesian causal discovery against existing Bayesian causal discovery methods, demonstrating the advantages of directly learning a posterior over causal structure.

Published as a conference paper at ICLR 2025
A META- APPROACH TO BAYESIAN CAUSAL DISCOVERY
Anish Dhir
Matthew Ashman University of Cambridge
James Requeima University of Toronto
Imperial College London anish. dhi r1 r130 30 311 ippria ac..uk uk
Mark der Wilk University of Oxford
ABSTRACT
Discovering unique causal structure is difficult due to both inherent identifiabil- ity issues, and the consequences of finite data. As such, uncertainty over causal structures, such as those obtained from Bayesian posterior, are often necessary for downstream tasks. Finding an accurate approximation to this posterior is chal- lenging, due to the large number of possible causal graphs, as well as the difficulty in the subproblem of finding posteriors over the functional relationships of the causal edges. Recent works have used Bayesian meta learning to view the prob- lem of posterior estimation limited as they cannot reliably sample from the posterior over causal structures and fail encode key properties of the posterior, such correlation between edges and permutation equivariance with respect to nodes. To address these lim- itations, we propose a Bayesian meta learning model that allows for sampling causal structures from the posterior and encodes these key properties. We com- pare our meta- ceta-Bay causal discovery against existing Bayesian causal discov- ery methods, demonstrating the advantages of directly learning a posterior over causal structure.
supervised learning task. Yet, these methods are
INTRODUCTION
Learning causal structure has the potential supercharge areas of science with the ability to un- derstand the impact of interacting with complex systems. Learning causal structures with certainty, requires interventional data (Pearl 2009), which is often impossible makes it preferable to infer as much of the causal structure as possible using observational data, and find way of dealing with the resulting multiple solutions that cannot be distinguished. One way to do this is to use Bayesian inference to express uncertainty over causal structure, which can then be incorporated in downstream tasks, such as effect estimation (Toth al. 2022).
very costly to gather. This
increase
confidence about relations by collecting more data (Agrawal et al. 2019 Zhang et al. 2023 Jain
al. 2023 Tigas al. 2023).
Dhir et al. 2024) showed that the Bayesian posterior can distinguish causal structure, even within Markov equivalence class, using only the implications of priors on functional relationships between variables. This effect occurs because the factorisation assumptions in the prior over functions, im- plied by the Independent Causal Mechanism (ICM) principle (Janzing Schölkopf 2010). can contribute to the inference over the causal structure. Hence, inaccuracies in functional inference can also result in inaccurate inference over causal structure. Current methods that approximate the posterior causal graphs make restrictive assumptions on the functions for accurate inference (An- nadani et al., 2021 Cundy et al. 2021), or rely on approximating the posterior over functions in Bayesian neural networks (Lorch al. 2021 Annadaní et al., 2024), which is notoriously difficult to do accurately (Wenzel et al.; 2020). Moreover, the high dimensionality of graphs with many variables further complicates effective inference.
In contrast, Bayesian meta-learning models such as neural processes (Garnelo et al. 2018) directly learn map from dataset to posterior distribution. The Bayesian prior is implicitly encoded

---

Published as a conference paper at ICLR 2025
through distribution over datasets, while directly predicting the posterior implicitly marginalises out any latent variables (Requeima 2023] Müller et al. 2021). This approach circumvents the need to approximate distributions on functions that present in previous approaches. Such generative models have also shown strong performance in sampling from high-dimensional distributions, mak- ing them well-suited for large graphs. However, directly learning posterior over causal graphs does introduce new challenges from a generative modelling perspective. We need to allow for com- plex distributions with dependencies between edges, while also constraining to the space of acyelic graphs (Zheng et al. 2018). Additionally, properties present in the true posterior, such as permu- tation equivariance with respect to the nodes, must be incorporated. Recent efforts use neural processes for this purpose only targeted the maximum posteriori value, leading to limitations when considering the full posterior. Some methods lack the ability to explicitly generate acyclic samples while ignoring permutation equivariance (Lorch et al. (022), while others fail to capture the dependencies between edges (Ke et al., 2022). resulting in an inability to sample from the correct posterior. Thus, their usefulness for downstream tasks, where the full posterior over causal structures is required, is limited.
We propose the Bayesian Causal Neural Process (BCNP) model that addresses these drawbacks. This takes the form of an encoder-decoder transformer that encodes dependencies between edges, and is permutation equivariant with respect the nodes. Building on recent advances in Charpentier let al. (202T) and Annadani et al. (2024), our decoder allows for direct sampling over directed acyclic graphs (DAGs), by parametrising distribution over permutation and lower triangular Bernoulli ma- trices. The samples are thus acyclic by construction and can allow for sampling multiple valid causal structures that are consistent with given dataset. We show that the BCNP model generates accurate posterior samples compared to previous Bayesian meta-learning approaches (section 4.1) We also demonstrate that the BCNP model outperforms explicit Bayesian models, as well as other meta- learning models when the model is trained on the correct data distribution (section 4.2). and also when the data distribution of a dataset is unknown (section 4.3). Our contributions are as follows: 1) We build on previous insights on meta-learning for causal discovery but focus estimating the full posterior over causal structures. We introduce a decoder and a loss function to achieve this all the while encoding key properties of the posterior (table TT. In contrast with other approaches, this allows for sampling from the correct posterior over causal structures. 2) We show that our changes lead to significant performance increases as well as outperforming explicit Bayesian causal models as well as other meta-learning models on synthetic (section 4.2) and semi-synthetic data (section
We provide code for our method: Causallstructuureeeeurrll....oo.
2 BACKGROUND AND RELATED WORK
2.1 LEARNING CAUSAL STRUCTURE
We denote a given a dataset of N samples with D variables as X. We assume that the dataset is generated according to a Structural Causal Model (SCM), which is specified by directed acyelic graph (DAG) encoded through its acyelic adjacency matrix G E {0, DXD, The variables are generated by following the DAG as
(1) where Xa is the dh variable, fa is a function, PAg(d) is the index set of parents of variable Xa in the DAG g, and ca is some arbitrary noise. Our goal is to infer the DAG (or causal structure) that generated given dataset.
Xa= = fa fa(X Ea), for d E NJ
To do this, we specify Bayesian causal model P(X,f.9) = P(X)F,)) that specifies our belief of the data generation process, with likelihood P(X 9). and priors over functions P(f) and over DAGs P(g). In the SCM, changing variable's functional mechanism does not change any other variable's functional mechanisms. This imposes constraint on the Bayesian causal model in that it must satisfy the Independent Causal Mechanism (ICM) assumption (Janzing & Schölkopf 2010 This is achieved by ensuring that prior over functions factorise over the variables, P(f) 1 P(fa) (ttegle et al. 2010 Dhir et al. 2024). The posterior, which specifies the model's belief
over DAGS can be computed as
X( = P(X
(2)
P((())))))))
2

---

Published as a conference paper at ICLR 2025
where P(X(g) is referred to the marginal likelihood. Crucially, the marginal likelihood is the normalization factor for the posterior distribution over the model's functional mechanisms f and thus relies accurate posterior inference over f. A Bayesian causal model is called identifiable a dataset it generates corresponds uniquely to the correct DAG (Guyon et al. 2019] Ch. 2). In such cases, Dhir al.] (2024) demonstrated that, under the ICM assumption, the marginal likeli- hood-and by extension, the posterior over functions- sufficient to distinguish causal directions. Thus, inaccurate functional inference can lead to incorrect causal structures being inferred. Fur- thermore, if a causal model is not identifiable, multiple causal structures may explain the dataset to varying degrees - challenge that also arises with finite samples. In these cases, the posterior over causal structures in eq. (2) quantifies this uncertainty.
Although the marginal likelihood is tractable for simpler models, this can introduce misspecification issues. For more flexible function approximators, such as neural networks, computing the marginal likelihood becomes challenging due to the complexity of functional inference. Additionally, the large space of possible DAGs further complicates this process, as the normalizer in eq. (2) is in- tractable to compute, and sampling efficiently in high-dimensional spaces is difficult.
2.2 RELATED WORK
A lot of causal discovery focuses on identifying a single causal structure that generated a dataset (Lachapelle et al., 2019, Rolland al. 2022 Zhang et al. 2013). This is prone errors in cases where, due to identifiability or finite sample issues, multiple causal structures can explain a dataset. On the other hand, Bayesian causal discovery provides a distribution over possible causal structures that align with the observed data. As our work is concerned with the latter, we review existing approaches in this area.
The challenges of Bayesian causal discovery lie in accurate estimation of the marginal likelihood, which requires accurate inference over functional mechanisms and the large number of graphs. Geiger & Heckerman (2002) addressed this by limiting their approach to linear Gaussian models and constraining to a single prior, ensuring identical posteriors within a Markov equivalence class, thereby reducing the search space. Other methods for linear models enable the incorporation of ad- ditional priors by using variational inference to tackle the large search space (Cundy et al. [202I]A) An- nadani et al. 202T]. However, even though inference is accurate, considering only simple functional relationships may lead to misspecification. For more expressive models, such as neural networks, methods that combine both sampling and variational inference have been proposed. DiBS (Lorch it al. (202)) uses Bayesian neural networks to model relationships between variables and Stein vari- ational gradient descent (Liu Wang, 2016) to perform inference over both the neural networks and the causal structure. A regulariser is used to ensure that the samples are acyelic (Zheng et al.) 2018). However, this approach is computationally expensive with quadratic cost with the number of samples and further samples requires restarting the optimization process. In contrast BayesDAG (Annadani et al. (024) directly parametrise the space of DAGs using upper triangular and permuta- tion matrices (Charpentier et al. (02II) eliminating the need for regulariser and enabling the use of variational inference. For expressivity, they also use Bayesian neural networks, but use SG-MCMC (Chen et al., 2014) perform inference over the network weights. Despite these advancements, the accuracy of inference mechanisms for Bayesian neural networks remains uncertain (Wenzel et al., 2020), and such the quality of the posterior over graphs. Our approach differs by amortizing the entire inference process, which mitigates inaccuracies in the inference of functional relationships between variables and facilitates rapid sampling from the posterior distribution.
The line of work closest to ours is Bayesian meta-learning based causal discovery, where causal dis- covery is reformulated as a classification problem. [Lpez-Paz et al.) (2015) first proposed this with two variables. The kernel mean embedding of data with known causal relationships (or syntheti- cally generated) was used to train a classifier. [i et al.] (2020) extended the approach using neural network embeddings to output the adjacency matrix for multiple variables. A key observation was that the adjacency matrix must be permutation equivariant with respect to the variables, improving the model's statistical efficiency. However, they limited their input features to the correlation coef- ficients between variables. Recent advances like AVICI (Lorch et al., 2022) and CSIvA (Ke et al., 2022) extend the above approach, using transformers fixed features. Here, attention between nodes and samples of a node are interleaved, ensuring that the final representation is extracted such that it is permutation invariant with respect the samples.
learnt embeddings of the inputs instead of
3

---

Published as a conference paper at ICLR 2025
AVICI uses max pooling operation to achieve this, while CSIvA uses attention operation (Lee al. 2019). They both also differ in their representation of the adjacency matrix. AVICI decodes the representation using linear layer to binary matrix, while CSIvA uses autoregressive decoder to estimate the probability of each element in sequence.
Methods such AVICI and CSIvA aim to approximate the posterior over the causal structure (Ke) et al. 2022, Lorch et al. 2022). but only target the maximum a posteriori value. Thus, when con- sidering the full posterior, these approximations fail capture essential properties, resulting in inaccurate estimates. AVICI only estimates the marginal probabilities of an edge existing, while the use of an acyclicity regulariser directly on the probabilities biases these estimates. Moreover, AVICI can also only provide acyclic samples in expectation and does not provide samples from the actual posterior (section 4.1). CSIvA captures dependencies between edges but lacks permutation equivariance, meaning that reordering nodes can change its belief about causal relationships It also does not guarantee acyclic samples, and its autoregressive approach performs poorly as the number of variables increases. In this work, we allow for sampling acyclic graphs while retaining the per- formance of these methods. The differences between AVICI, CSIvA, and ours (BCNP) can be seen in table
Table 1: Comparison of our approach against existing Bayesian meta- meta-learning methods for Bayesian causal discovery (AVICI and CSIvA).
AVICI
CSIvA
BCNP (ours)
Acyelic Samples Edge Dependency Permutation Equivariance
In expectation
No
Yes
No
Yes
Yes
Yes
No
Yes
3 CAUSAL STRUCTURE LEARNING AS BAYESIAN META LEARNING
The process described in eq. requires positing causal model including prior beliefs over the causal functions, as well as the causal structure. Due to the presence of the marginal likelihood term in eq. (2). the posterior over the causal structure also requires computing the posterior over the functional parameters. For flexible models, such neural networks, good approximations to this posterior are hard to obtain. To address this difficulty, we can use neural processes (Garnelo et al., 2018) to learn direct mapping from datasets to posteriors over causal graphs, bypassing the need to approximate the posterior over functions.
Our goal is to approximate the posterior of the Bayesian causal model of choice, which we refer to as PECM X, f. 9) described section [2.1 To do so, we use a Bayesian Causal Neural Process (BCNP) model P (g with parameters ø. This model is an encoder-decoded transformer network that takes in a dataset and directly approximates the posterior PBCM (g)). To train it to do so, we minimise the KL-divergence between (BcM()]X) and (x(
min Ebccu(X) (X) KL [PccI((IX)[PP(((XXX)
(3)
min EPpcc( (Erist(t) [EPncu(X(1.g) [log (o]])]]]] + +C
(4)
where C' is a constant that does not depend on Φ. During training, the expectation in eq. 3 is computed via Monte Carlo approximation that requires sampling from the Bayesian causal model PReM (X, f, G). Sampling from this model proceeds by first sampling graph G ~ (9), then a functional mechanism for each of the D variables f [f....., fD} ~ (fC and then N samples of each variable generated according to G denoted X ~ PRCM XIf, G). The above proce- dure applies not only to datasets and causal graphs from an explicitly defined Bayesian causal model but also enables posterior approximation from datasets and causal graphs generated from some un- known distribution. Regardless of the data generating procedure, by directly learning the posterior over graphs, the model Po (G) implicitly marginalises out the function f, removing the need to approximate the posterior over functions. The above KL-divergence is minimised if and only if the model recovers the true posterior, P& X POCM (Foong al. 2020
4

---

Published as a conference paper at ICLR 2025
There certain properties of the posterior P that be encoded directly in (o()) Permuting the samples of a dataset does not change the posterior distribution, which implies that Pa(g(X) must be permutation-invariaat with respect to the samples. Furthermore, permuting the nodes in the dataset should not change the model's belief, expressed through the posterior, about the existence of causal relationship. For example, if model predicts X Y with certain probability when given (X, Y) as inputs, it should predict the same relation with the same proba- expressed through a DAG
bility if the inputs are permuted (Y) As causal relationships G {0, 1) 1)Dx D, this requires the probability of sampling a DAG to be ietiuuttttiooquivariiiaa with respect the nodes.
In addition to encoding these properties, the model must also be able to sample DAGs from the posterior to be practically useful. Next, we introduce network that parameterises the distribution (X) to represent distribution over DAGs, allowing us to sample acyelic graphs corresponding valid causal structures. Furthermore, it also ensures that the distribution is permutation invariant with respect to the samples, and permutation equivariant with respect the nodes.
3.1 ENCODER
D
N
R
Append vector of zeros
Attention over samples
Attention nodes
Attention(K
Q
Figure I: Each dataset contains nodes and N samples where each data point is embedded into a vector of size H, giving D N H tensor. A query vector of zeros is then appended along the sample axis. The data is passed through L transformer layers which alternate between attention over samples and attention over nodes. The summary representation It is constructed using attention layer where the samples of each node serve as the keys and values and the query vector acting as the query.
Our encoder follows that of Ke et al. 22022; Lorch al. (2022). Inputs are embedded and passed onto transformer layers (Vaswani et al. (2017), see appendix A. that alternate between attention across samples and attention across nodes. Finally a summary representation vector constructed that summarises the information in each node across all the samples. Crucially, the summary rep- resentation vector constructed such that permutation invariant with respect the samples of each node. Figure T visualises the encoder.
Embedding: Each input node and sample ag is embedded into vector of size H using linear map. As we want the subsequent attention operations to summarise the information across the samples, concatenate vector of Os, which we call the query vector, in the sample dimension. This gives a tensor of size H E R D
Transformer layers: The tensor HD is then passed onto transformer layer that performers atten- tion across samples. That is, parameters are shared across nodes. Following this, a transformer layer applied that performs attention across nodes, with parameters across samples being shared. This procedure is repeated L times.
Summary representation:
The representation of the data is extracted from the output of the trans- former layers H E R DxxN along with the query vector q E RR 1x A final cross-attention operation across samples is carried out using H the key and values, and q the query. The output of this a summary representation vector R E D2x that summarises the information in the samples for each node.
5

---

Published as a conference paper at ICLR 2025
The cross roos-attentiion operation ensures that the summary representation R is permutation invariant with respect to the samples (Kim et al.). As the nodes are only processed using self-attention, the summary representation is also permutation equivariant with respect to the nodes (Lee et al. 2019).
3.2
DECODER
One of the main differences in our architecture from those in Ke et al. (2022); Lorch et al. (2022) lies in the design of our decoder. Unlike the decoders in these prior works, our proposed decoder is de- signed to directly sample from the distribution of interest, requiring us to parameterize distribution over DAGs, Charpentier et al. (202)) show that any DAG G can be represented by a permutation ma- trix Q, and lower triangular binary matrix A through G QAQ", This ensures that samples are acyclic and hence represent valid causal structures. Thus, we can reduce the problem of learning distribution over DAGs to one of learning a distribution over permutations and over lower triangular binary matrices.
Distribution over Permutations:
Permutation matrices, Q, can be parametrised by a matrix θ E DxD by solving the following con-iiferrentible problem
Q arg max QQ' O)
(5)
where (A, trace(A" B) is the Frobenius inner product of matrices. Intuitively, eq. (5 yields the permutation matrix, which when applied to θ, maximises the trace. Changing the values of θ then changes the permutation matrix that yields the maximum. To allow for learning with gradients, Mena et al. (2018) 8) propose modifying the problem in eq. (5) using the differentiable Sinkhorn oper- ator S(-). The Sinkhorn operator consists of repeated normalisations of the rows and columns of the input, converging to solution of eq- (5) in the limit. Mena et al. (2018) show that the parametri- sation of the permutation matrix in eq. (5) can be well approximated by S (e) for T + O, where т is some temperature parameter. Further, adding Gumbel noise G gives differentiable distribution over permutation matrices called the Gumbel-Sinkhore GS == S e + G). This distribution does not have tractable density but allows for sampling permutations parametrised by O. We follow Annadani et al. (2024) and parametrise θ using low rank representation that takes fewer sinkhorn iterations converge solution. The representation from the encoder R" E RDxH is first pro- cessed using transformer layers to give RL,, and then θ is parametrised as (Annadani et al. 2024
θ R Q GS(@)
where o = [1..... Dj. As the rows of R l/1 are permutation equivariant with respect to the nodes, so are the rows of O In practice permutation matrices for the forward pass (Charpentier et al. 2021). In the backward pass, we use the straight through estimator optimise through R. (Bengio al. 2013).
> and the Hungarian algorithm to get discrete
Distribution over Lower Triangular Matrix: We model the presence of edges as lower tri- angular matrix of Bernoulli random variables. Permuting this lower triangular matrix then orients the edges all the while ensuring acyclicity. To ensure permutation equivariance of this matrix with respect to the nodes, we borrow ideas from attention. Further, to ensure dependence between the permutation and lower triangular binary matrices, we ensure that the distributions share parameters. This is done by processing R by further transformer layers to give R R which is then used to parametrise the lower triangular matrices through the following operations
ParameterAAtentioon (Q,
E DxD
head (R4x) ArraantterAteentinn// W Ra ba)) MHPA(R == : stackm (headlm. (R_ W)
(8)
where W. E Ax11 W E Hxha, and W E B ParameterAttentionn takes in two matrices and computes the outer product giving a D x D matrix. MHPA is a multi-head variant of Atteentiorttetioon The matrix outputted by ParameterAttentin permutation equivariant with respect to the input rows (Lee et al.) 2019). As we input a linear map of R into these operations, the output is permutation equivariant with respect to the nodes. The
and thus also by MHPA is

---

Published as a conference paper at ICLR 2025
0)
MHPA
R
T
R o"
R
R
+
Q. ~ GS(()
A. Bern b)
0 1... D
C. Q.A.!
Figure 2: Computational graph of the decoder described in section 3.2 The decoder takes in the summary representation from the encoder R as input. '" denotes a transformer layer, MHPA de- note multi headed parameter attention (eq. 9 and GS is the Gumbel-Sinkhorn distribution (Mena ett al. 2018). The network outputs samples of permutation matrices Qs and lower triangular binary matrices A, that can be used construct samples of DAGs G..
final lower triangular matrix parametrised
MHPA (2) if <
Pij
(10)
0f
The final decoder architecture that allows for sampling acylic DAGs can be seen in fig.
3.3
Loss
Given a permutation matrix Q and lower triangular binary matrix A a DAG can constructed as QAQ" As A is lower triangular matrix with Bernoulli random variables with parameters
4, we can write P& (g Q, 4, X) = Bernoulli Bernouli(Q@Q"). Q&Q"). Since Q is sampled from the distribution defined in section 3.2 the log probability of DAG is
log (x = log
P (
(11)
log Σ Pgl X,X), Q (N((()))
(12)
where Pa (X)). Combining eq. 3 and 12). the following loss approximates posterior given datasets and DAGs from the Bayesian causal model:
min -E E
log 5
1 Σ PaloQo
9, E
Q Px(QI
(13)
EXPERIMENTS
After defining the Bayesian causal neural process (BCNP) model and the loss function in section 3 which approximates the posterior of Bayesian causal model given samples of datasets and DAGs, we move on to testing the practical performance of the BCNP model. Since wel-worldd datasets with established causal relationships are rare and often disputed (Mooij et al.) 2020). we evaluate the BCNP model and baselines using synthetic and semi-synthetic data. With our experiments, we aim to answer the following questions: 1) When the true posterior is known, do the samples from our model match the true posterior? (section[4.) 4.1 2) How does the BCNP model perform in comparison with explicit Bayesian models and other meta-learning models? (section (42). 3) How does the BCNP model and other meta- leatnnln models perform on realistic datasets, where the datasets do not directly inform Bayesian causal model and hence the training data? (section 4.3
7

---

Published as a conference paper at ICLR 2025
Metrics We follow prior work in using the Area Under the Receiver Operating Curve (AUC) to assess edge prediction at various probability thresholds for the existence of an edge (Friedman & Koller 2000p. Additionally, we evaluate the sum of the log Bernoulli probabilities for each edge under the model. The AUC reflects the ranking of edge probabilities, while the log probability metric considers the actual probability values. Both metrics focus on marginal edge probabilities. Previous work also considers metries like the expected expected Structural Hamming Distance (expected SHD) and expected edge Fl score over samples. However, we argue that these are not necessarily suitable for evaluating Bayesian methods because they average metrics over the posterior, which doesn't 't reflect how good prediction based on the uncertainty the posterior is. Nonetheless, we include them as they offer insights into the expected utility of the samples. Lower is better for expected SHD, while higher is better for the rest.
4.1 COMPARING BAYESIAN META-LEARNING METHODS ON TWO VARIABLE NORMALISED LINEAR GAUSSIAN DATA
We consider the two variable X, Y case where datasets are either generated from the DAG X Y or Y + X. The causes are generated from a normal distribution and the cause to effect relationship linear with additive Gaussian noise. After generating, both variables are normalised. All the meta-lcarning models are trained on this data. This is known unidentifiable case where the data cannot identify the causal graph at all (Peters et al. 2017 Dhir et al. 2024). sound Bayesian model should thus generate samples of X + Y and Y X with probability 0.5 each. To do so will require ensuring that the samples are acyclic, and terms of the adjacency matrix are correlated.
The details of the data generation and model used are in appendix B.1 and 500 samples from each of the AVICI, CSIvA, and BCNP models. (ahlee]ssoww the proportion of graph types output by each of the models. It shows that while CSIvA and BCNP output reasonable samples, AVICI does not, Although the data is clearly correlated, AVICI outputs samples with no causal relation 32% of the time, and cyclic samples 19% of the time. This is because the decoder in AVICI does not take into account edge dependencies. Even though CSIvA outputs the correct graphs here, it does not explicitly enforce acyclicity or permutation equivariance. In section 4.2 will see that with more variables its performance suffers.
We generate 100 test datasets
4.2 COMPARISON ON VARIOUS BAYESIAN CAUSAL MODELS
To analyse the performance of the Bayesian meta-learning methods as well as explicit Bayesian models, on larger number of nodes, we compare the models on data from 20 node Bayesian causal models with varying functional distributions. Causal discovery methods tend to perform worse on denser graphs. Therefore, following prior work, we generate graphs from Erdos Renyi (ER) dis- tributions (Erdos et al.) 1960) with varying densities, with expected edge counts of 20, 40, and 60. To showcase the advantage of the meta- meta-learning Bayesian approach, we choose functional relation- ships whose posterior may be hard to approximate with explicit Bayesian models: 1) Linear: Linear functions with heteroscedastic noise, 2) NeuralNet: Randomly initialised neural networks with a normally distribution latent variable included an input, 3) GPCDE: Random function drawn from a Gaussian process with a latent variable included as an input. The exact details of the data generat- ing procedure is in appendix B.2 For each of the function types and graph densities, we sample 25 datasets to test the methods on.
We compare against the explicit Bayeisan models DiBS (Lorch al. 202T), BayesDAG (Annadani et al. 2024), as well as the Bayesian meta-learning models AVICI and CSIvA. Each meta-learning model trained on datasets from each of the functional and graph distributions. Details of all the models are in appendix B.3 Due to memory constraints in the autoregression, the CSIvA model has a lower width than AVICI and BCNP. However we do show that AVICI and BCNP outperform CSIvA with similar widths in appendix C.2 To see if single model can perform well across mul- tiple distributions, we also include a single BCNP model trained on all synthetic datasets (labelled BCNP All Data).
Results for the densest graphs, ER60, are in table The rest of the results for the Linear datasets in table 7 table 8 table 9 for the NeuralNet datasets are in table [0] table []] table [2] and for the GPCDE datasets are in table totable Iablle [2] Meta-learning models generally outperform explicit Bayesian models like DiBS and BayesDAG, especially with denser graphs (ER60). Among
8

---

Published as a conference paper at ICLR 2025
meta-learning models, CSIvA underperforms, likely due to the challenges of autoregressive adja- cency matrix generation with increased variables, as well as missing permutation equivariance with respect to nodes. The challenge in autoregression is also noted by the authors in Ke et al.] (2022), who resort to adding an auxiliary loss. As the AUC and Log Probability only consider marginal edge probabilities, AVICI and BCNP show comparable performance in these metries. However, as demonstrated in Section section 4.1 AVICI faces difficulties in DAG sampling, an issue not captured by these metrics but partially reflected in the expected edge Fl score. Training on a mixture of all datasets (BCNP All Data) maintains performance across all datasets (more results in appendix O suggesting that training on a mixture of distributions can help mitigate model misspecification with- out a loss of performance on each distribution.
4.3 COMPARISON ON SYNTREN
Since real data for causal discovery tasks is scarce, we follow prior work and use data generated from simulators. A commonly used simulator is the Syntren generator (Van den Bulcke et al. 2006)
which generates gene expression data that matches known experimental data. Each dataset consists of 20 nodes with 500 samples each. We use the 10 datasets generated by Lachapelle et al. (2019) to test all the models. To train the meta-learning models, we use mixture of all the possible function and graph distributions in section 4.2 Exact details about the training data used and baselines is in appendix B.3 Table 2 shows that the BCNP is competitive even in the case when the model has not been trained on the exact distribution from which the data is sampled. We found that training on wide range of distributions helped model performance as well as improving the capacity of the model.
Table 2: Results for the Syntren dataset (mean + std of mean). Results are over 10 datasets.
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
DiBS
0.81 0.02
-83.31 # 1.68
59.83 # 1.22
0.15 0.01
BayesDAG
0.52 # 0.04
-195.53 # 16.78
99.09 # 0.90
0.10 + 0.01
AVICI
0.66 0.03
-90.82 4.82
48.94 # 2.37
0.09 0.01
CSIvA
0.37 # 0.03
-100.59 + 2.49
67.34 # 1.11
0.07 + 0.00
BCNP
0.84 = 0.02
-76.35 7
50.87 # 1.92
0.17 = 0.01
5 DISCUSSION AND LIMITATIONS
Our objective minimises the KL-divergence between the posterior of the Bayesian causal model from which training data is drawn PBCM( and the BCNP model ()))). Datasets and their corresponding causal graphs may also come from an unknown distribution PBCM instead of an explicit causal model. Nevertheless, given dataset from ( X). if our model achieves KL-divergence of zero, we can expect it to perform well on downstream tasks requiring the posterior from PBCM (9) However, given datasets from separate distribution II, what can we say about the performance of our approximate posterior (g) Note that
X)] + KL[PacM ( (G) (G [ = = KLIG(X)) = O.
If the KL-divergence between our model and the BCM is zero, then the performance on datasets drawn from II will depend on how closely PECM (G|X) matches the target distribution [I) the KL-divergence is not zero, the approximation is likely perform even worse tasks requiring (()))). Thus, to improve performance, either the BCNP training should be enhanced through more data and higher capacity, align with the target distribution. Although results from section 4.3 conform to this intuition a more rigorous treatment of this question would be necessary to make stronger claims.
the training data distribution FBCM ( X) should be adjusted to better
Unlike other meta-learning approaches (Ke et al. 2022 Lorch et al., 2022), we do not incorporate interventional data in this work. However, similar to these studies, interventional data be in- tegrated into the BCNP model by appending an indicator vector to the node axis in section 3.1 signaling whether a node has been intervened upon.

---

Published as a conference paper at ICLR 2025
CONCLUSION
In this work, we address challenges causal discovery by employing a Bayesian framework to quantify uncertainty. Meta-learning approaches useful here they circumvent complex func- tional inference and enable efficient sampling from high-dimensional spaces. Building on prior research, we incorporate key properties of the posterior distribution such as permutation equívari- ance with respect to the nodes and dependency between edges. In contrast to other meta-lcarning methods, the BCNP model also provides samples of DAGs from the posterior. The BCNP model demonstrates competitive performance compared to both explicit Bayesian and other meta-learning models. Our work suggests that meta-leariing holds significant potential for a tractable and accurate Bayesian treatment of causal discovery.
Table 3: Proportion of different graphs output in the two-variable umidentifiable case tested with 50% X1 X2 and 50% X2 X1. The table shows that AVICI only estimates the marginal probability of each edge, sampling from it does not output graphs from the correct posterior.
DAG Type
AVICI
CSIvA
BCNP
No edge between X1 and X2
0.3194
0.0000
0.0124
X1 + X2
0.2587
0.4851
0.4979
X2 + X1
0.2342
0.5147
0.4897
X1 + X₂ (bidirectional)
0.1877
0.0000
0.0000
Other
0.0000
0.0002
0.0000
Table 4: Results for the 20 variable Linear ER60, NeuralNet ER60, and GPCDE ER60 datasets (mean of mean). The BCNP ER60 model has been trained on datasets from the specific function type and graph density, while the BCNP All Data is trained mixture of all datasets.
Linear ER60 Dataset
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
DIBS
0.40 0.01
-231.40 # 3,69
106.64 1.25
0.11 # 0.00
BayesDAG
0.51 0.01
-1509.27 + 48.40
63.51 + 0.40
0.04 + 0.01
AVICI
0.01
-126.31 + 1.71
80.90 0.71
0.31 = 0.01
CSIvA
0.49
-169.49 # 0.19
102.00 + 0.05
0.15 + 0.00
BCNP ER60
+
-121.87 + 2.29
76.52 0.92
0.36 - 0.01
BCNP All Data
0.86 + 0.01
-122.02 + 2.66
72.15 + 1.17
0.36 + 0.01
NeuralNet ER60 Dataset
Model
AUC
Log Probability
Expected SHD
Expected Edge Fl
DiBS
0.64
-208.22 + 1.82
65.72 0.17
0.13 + 0.01
BayesDAG
0.58+000
-170.73 = 0.26
103.95 0.09
0.19 + 0.01
AVICI
0.79 0.01
-138.04 # 1.36
85.15 $5.15+0.55 0.59
0.29 + 0.01
CSIvA
0.01
-169.30 # 0.20
101.89 0.05
0.15 0.00
BCNP ER60
0.85 - 0.00
-121.09 + 1.08
70.40 0.35
0.40 + 0.00
BCNP All Data
0.83 + 0.01
-127.48 + 2.71
68.34 + 0.91
0.39 + 0.01
GPCDE ER60 Dataset
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
DiBS
0.67 + 0.01
-199.87 + 2.03
64,22 0.22
0.13 + 0.00
BayesDAG
0.66
-184.48 5.12
107.46 0.31
0.24 0.00
AVICI
0.00
-151.70 1.28
91.33 0.36
0.21 + 0.01
CSIvA
0.49 0.01
1169.53
101.62 0.05
0.15 0.00
BCNP ER60
0.72
-149.48 # 1.05
93.07 + 0.34
0.22 + 0.01
BCNP All Data
0.77 + 0.01
-142.97 1.80
80.28 0.77
0.25 + 0.01
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Raj Agrawal, Chandler Squires, Karren Yang, Karthikeyan Shanmugam, and Caroline Uhler. Abed- strategy: Budgeted experimental design for targeted causal structure discovery. In The 22nd International Conference Artificial Intelligence and Statistics. PMLR, 2019.
Yashas Annadani, Jonas Rothfuss, Alexandre Lacoste, Nino Scherrer, Anirudh Goyal, Yoshua Ben- gio, and Stefan Bauer. Variational causal networks: Approximate bayesian inference over causal structures. arXiv preprint arXiv:2/06.07633, 2021.
Yashas Annadani, Nick Pawlowski, Joel Jennings, Stefan Bauer, Cheng Zhang, and Wenbo Gong- Bayesdag: gradient-based posterior inference for causal discovery. Advances in Neural Informa- tion Processing Systems, 36, 2024.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffre E Hinton. Layer normalization. stat, 1050, 2016.
Llbert-Lászle Barabási and Réka Albert. Emergence of scaling in random networks. science, 286 (5439), 1999.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprini arXiv: 301.3308..3
arXiv:1308.3332. 3432, 2013.
Bertrand Charpentier, Simon Kibler, and Stephan Günnemann. Differentiable dag sampling. In
International Conference Learning Representations, 2021.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning. PMLR, 2014.
Chris Cundy, Aditya Grover, and Stefano Ermon. Bed nets: Scalable variational approaches for bayesian causal discovery. Advances in Neural Information Processing Systems, 34, 2021.
Anish Dhir, Samuel Power, and Mark Van Der Wilk. Bivariate causal discovery using Bayesian model selection. In Proceedings of the 41st International Conference Machine Learning. volume 235 of Proceedings of Machine Learning Research. PMLR, 2024.
Paul Erdos, Alfréd Rényi, et al. On the evolution of random graphs. Publ. math. inst. hung. acad. sci, 5(1):17-60, 1960.
Andrew Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard Tumner Meta-leaming stationary stochastic process prediction with convolutional neural pro- cesses. Advances in Neural Information Processing Systems, 33, 2020.
Nir Friedman and Daphne Koller. Being bayesian about network structure. In Proceedings of the Sixteenth conference on Uncertainty artificial intelligence, 2000.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01622. 2018.
Dan Geiger and David Heckerman. Parameter priors for directed acyclic graphical models and the characterization of several probability distributions. The Annals of Statistics, 30(5), 2002.
Isabelle Guyon, Olivier Goudet, and Diviyan Kalainathan. Evaluation methods of cause-effect pairs. In Cause Effect Pairs in Machine Learning, chapter 2. Springer, 2019.
Moksh Jain, Tristan Deleu, Jason Hartford, Cheng- Hao Liu, Alex Sermandez-Garciie and Yoshua Bengio. Gflownets for ai-driven scientific discovery. Digital Discovery, 2(3), 2023.
Dominik Janzing and Bernhard Schölkopf. Causal inference using the algorithmic markov condi- tion. IEEE Transactions on Information Theory, 56(10), 2010.
Nan Rosemary Ke, Silvia Chiappa, Jane X Wang, Jorg Bornschein, Anirudh Goyal, Melanie Rey, Theophane Weber, Matthew Botvinick, Michael Curtis Mozer, and Danilo Jimenez Rezende. Learning to induce causal structure. In International Conference on Learning Representations, 2022.
11

---

Published as a conference paper at ICLR 2025
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gamelo,, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient-based neural dag learning. In International Conference on Learning Representations, 2019.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation- invvariin neural networks. In Interna- tional conference on machine learning. PMLR, 2019.
Hebi Li, Qi Xiao, and Jin Tian. Supervised whole dag causal discovery. arXiv preprint arXiv:2006. 04697, 2020.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances neural information processing systems, 29, 2016.
David Lopez-Paz, Krikamol Muandet, Bernhard Schölkopf, and lliya Tolstikhin. Towards learning theory of cause-effect inference. In International Conference on Machine Learning. PMLR, 2015.
Lars Lorch, Jonas Rothfuss, Bernhard Schölkopf, and Andreas Krause. Dibs: Differentiable bayesian structure learning. Advances Neural Information Processing Systems, 34, 2021.
Lars Lorch, Scott Sussex, Jonas Rothfuss, Andreas Krause, and Bernhard Schölkopt. Amortized inference for causal structure learning. Advances in Neural Information Processing Systems, 35, 2022.
Yi-An Ma, Tianqi Chen, and Emily Fox. A complete recipe for stochastic gradient meme. Advances in neural information processing systems, 28, 2015.
Gonzalo Mena, David Belanger, Scott Linderman, and Jasper Snoek. Learning latent permutations with gumbel-sinkhorn networks. In International Conference on Learning Representations, 2018.
Joris M Mooij, Sara Magliacane, and Tom Claassen. Joint causal inference from multiple contexts. The Journal of Machine Learning Research, 21(1), 2020.
Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Trans- formers can do bayesian-inference by meta-learning on prior-data. In Fifth Workshop on Meta- Learning the Conference on Neural Information Processing Systems, 2021.
Judea Pearl. Causality. Cambridge university press, 2009.
Jonas Peters, Dominik Janzing, and Berhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning. Springer, 2003.
Alexander Reisach, Christof Seiler, and Sebastian Weichwald. Beware of the simulated dag! causal discovery benchmarks may be easy to game. Advances in Neural Information Processing Systems, 34, 2021.
James Requeima. The Neural Processes Family: Translation Equivariance and Output Dependen- cies. PhD thesis, 2023.
Paul Rolland, Volkan Cevher, Matthäus Kleindessner, Chris Russell, Dominik Janzing, Bernhard Schölkopf, and Francesco Locatello. Score matching enables causal discovery of nonlinear addi- tive noise models. In International Conference Machine Learning. PMLR, 2022.
Oliver Stegle, Dominik Janzing, Kun Zhang, Joris M Mooij, and Bernhard Schölkopf. Probabilistic latent variable models for distinguishing between cause and effect. Advances in neural informa- fion processing systems, 23, 2010.
12

---

Published as a conference paper at ICLR 2025
Panagiotis Tigas, Yashas Annadani, Desi R Ivanova, Andrew Jesson, Yarin Gal, Adam Foster, and Stefan Bauer. Differentiable multi-target causal bayesian experimental design. In International Conference Machine Learning. PMLR, 2023.
Christian Toth, Lars Lorch, Christian Knoll, Andreas Krause, Franz Pernkopf, Robert Peharz, and Julius Von Kügelgen. Active bayesian causal inference. Advances in Neural Information Pro- cessing Systems, 35, 2022.
Tim Van den Bulcke, Koenraad Van Leemput, Bart Naudts, Piet van Remortel, Hongwu Ma, Alain Verschoren, Bart De Moor, and Kathleen Marchal. Syntren: generator of synthetic gene ex- pression data for design and analysis of structure learning algorithms. BMC bioinformatics, 7, 2006.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. 2017. https://aaxiv
org/pdf/1705. 03764 727.66p.. pdf
Florian Wenzel, Kevin Roth, Bastiaan Veeling, Jakub Swiatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the bayes posterior deep neural networks really? International Conference on Machine Learning. PMLR, 2020.
Jiaqi Zhang, Louis Cammarata, Chandler Squires, Themistoklis P Sapsis, and Caroline Uhler. Active learning for optimal intervention design in causal models. Nature Machine Intelligecce 5(10), 2023.
Kun Zhang, Zhikun Wang, Jiji Zhang, and Bernhard Schölkopf. On estimation of functional causal models: general results and application the post-nonlinear causal model. ACM Transactions on Intelligent Systems and Technology (TIST), 7(2), 2015.
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric Xing. Dags with no tears: Continuous optimization for structure learning. Advances in Neural Information Processing Systems, 31, 2018.
A ADDITIONAL BACKGROUND
A.1 TRANSFORMERS
Here we describe transformer layer Vaswani al. (017).
Given an input from a previous layer X E R xdaxt the key K, query Q, and value V matrices are computed as
K = XW*
(14)
Q XW"
(15)
V = X
(16)
where W W9, W are the key, query, and value weights with K E Nxhe, Q E Nxhe and VE Nxh. We define the dot-product attention these matrices
Att K, V) = softmax QK? //hh)
(17)
where the softmax is over the last dimension. Multi-head attention is where multiple query, key, and value pairs are used (indexed by m), their results concatenated and the result projected down using W R MxR.
MHSA(X) stackm= (headm WX) where head, (X) = Attn Qm. Km, Vn).
(18)
(19)
where Qm XV K XW my and Vm = XWe m The final transformer layer as follows
m3
R(X) = X (X)),
(20)
T(X) R(X) ML))R
(21)
13

---

Published as a conference paper at ICLR 2025
where LN(.) is layer- normaliomaisonionn operation (Ba al. 2016). and MLP is feed-forward layer.
key property of the attention operation that permuting the rows X results in the rows of the output of the attention being permuted the same way. That is, applying permutation P matrix to the input of the layer X PX results in,
K XXW
(22)
Q XW"
(23)
XW",
(24)
and
QK" PQK" PT,
(25)
Softmax QK? V P (ooftax((Q""))
(26)
P Attnt(Q.... Q. V).
(27)
The transformer architecture thus encodes permutation equivariance with respect rows of the inputs Lee et al. (2019).
B EXPERIMENT DETAILS
B. TWO VARIABLE LINEAR GAUSSIAN EXPERIMENTS DETAILS
We generate the data by first sampling a graph for two variables X, Y that either corresponds to X + or X. For example graph of X + Y, we generate the cause and effects as
X ~ ~O, 1),
(28)
(O, 10),
(29)
22,,2(2.5,255).
(30)
Y Y~ V~N(xX,,'s
(31)
After generation, we normalise both X and Y ensure that the causal direction is not identifiable from data. We generate 200, 000 datasets in total with 1, 000 samples each.
For all the models, we use Adam (Kingma & Ba 2014) with a learning rate of 10-4 and batch size of 64. We train for 2 epochs and use a linear warmup for 10% of the total iterations. To reduce the memory footprint, we train in bfloat 16. All models used layers their encoder, while BCNP and CSIvA used 4 decoder layers. AVICI directly decodes from the summary representation. For BCNP, we used 100 permutation samples to approximate the loss, and maximum of 1000 sinkhorn iterations.
B.2 BAYESIAN CAUSAL MODEL DATA GENERATION DETAILS
Here, we provide details about the data generation process used to generate training and test data for results section 4.2 Following recommendations from Reisach al. (202T), we normalise all variables after generation.
All datasets contain 1000 samples with 20 nodes. For training the Bayesian meta-learning models, we generate 500, 000 datasets. Each test set contains 25 datasets. We generate graphs from the ER distribution Erdos al. (1960), and generate datasets for varying densities 20, 40, 60.
Linear: Given graph, we sample all causes from N/ 1). Samples for each variable X₂ are then generated as follows
NN(0, 10),
(32)
σ ~ Gamma(2.5, 2.5).
(33)
(Xa): = N' (w" (Xrac(n), o.),
(34)
where (XpAo(d)); is the 8th sample of the parents of variable X, in graph G.
14

---

Published as a conference paper at ICLR 2025
NeuralNet: We sample each variable as follows
~ NO0, 1),
(35)
(Xa); = NNo ((XPAc());
(36)
where (Xp) is the ht sample of the parents of variable Xa in graph ς and NN₂ is randomly initialised neural network. We use a two layer network with Leaky Relu activation and width of
32. When node has no parents, we simply pass the noise through the neural network to generate the variable.
GPCDE: Given samples of parents of a node (P) and scalar noise term NN(0,)) 1), we first sample hyperparameters
7 (111, 1),
(37)
1-1,1-
(38)
where one À is generated per input (including the scalar noise term), A = ( Apxengwolt)
Ay....,ApxeAguull
The exponential Gamma kernel is then constructed (Rasmussen) 2003)
K (((XPAc(e)) ( ((XpAo(e))) (5))
(39)
- ((XPAo(e)) .5) 11:
exp
(40)
where one À is generated per input. Sampling the 7 parameters ensures that we sample functions with varying smoothness. Sampling a different À for each input variable changes how quickly the function changes with respect to an input. Both together ensure that we sample a large variety of functions. We then sample the variable as follows
o Gamma(1, 10). Xa N (0, K + o²1).
(41)
(42)
B.3 BAYESIAN CAUSAL MODEL AND REAL DATA EXPERIMENT DETAILS
Here, provide the hyperparameter settings and baseline details for the experiments carried out in section 4.2 We compare against the explicit Bayesian models BayesDAG (Annadani et al.) (2024) and DiBS Lorch al., ) Here, the authors provide different linear and non-linear estimators. The non-linear estimators are neural networks. BayesDAG performs inference over the neural net- work weights by using SG-MCMC (Ma al. 2015), whereas DiBS using Stein variational gradient descent (Liu & Wang 2016). We also compare against the Bayesian meta-learning models AVICI (Lorch et al. 2022) and CSTvA (Ke et al. 2022). The differences between our model BCNP and the
other Bayesian meta-learning models can be seen in table [
DIBS:
Table 5: Hyperparameters for the DiBS model for the Linear, NeuralNet, GPCDE, and Syntren datasets
Hyperparameters
Linear
NeuralNet
GPCDE
Syntren
a
0.2
0.02
0.02
0.2
Y
5
5
5
Gi Number of particles
500
1000
1000
500
64
32
32
32
DiBS provides a linear and a non-linear model. The hyperparameters include the annealing rate of the acyclicity regulariser term o, lengthscale of 72 for the latent kernel, and lengthscale of yo for
15

---

Published as a conference paper at ICLR 2025
the parameter kernel For both models, we use the tuned hyperparameters from Lorch al. (2021)
for 20 variables for the Bayesian causal model datasets. For the Syntren dataset, we use values tuned in Annadanii et al. (2024). All the hyperparameters used can be seen in table We provide all the models with the correct graph density, number of variables as well as the graph generating distribution (ER) to construct its prior. For Syntren we used the scale-free (Barabási & Albert 1999) graph prior, as it closely matches the structure of the Syntren datasets, Due to the high memory cost, we optimised 64 particles for the linear model and 32 for the non-linear model. As the number of samples are quite low, when these were averaged to find the probability of each edge, the probability of certain edges was This lead an infinite log Bernoulli probability value. To deal with this, during testing, we add constants to the log probability from 10 8 to 10 and pick the highest resultant log probability. We keep all other hyperparameters to the preset value by the authors.
BayesDAG: For the Syntren dataset, we used the values tuned by the authors. For the Bayesian causal model datasets, following Amnadani et al. (2024). for each function type. we generate 5 vali- dation sets with graphs drawn from the ER distribution (Erdos et al., 1960) with 20 expected edges. For the linear and the non-linear models, and for each validation set we sample 20 hyperparameter configurations from the recommended hyperparameter range settings in the authors' the code. In contrast to the settings used by authors, we normalise the all variables as recommended by Reisach cct al. (202T). We note that the authors recommend choosing hyperparameters based on the lowest expected SHD score. However, in experience this lead nearly empty graphs on the test set. Hence, we choose hyperparameters based on the highest AUC score. The chosen hyperparameters for each function type can be seen in table 6 We keep all other hyperparameters to their preset values.
Table Hyperparameters for the BayesDAG model for the Linear, NeuralNet, GPCDE and Syntren datasets
Hyperparameters
Linear
NeuralNet
GPCDE
Syntren
X.
500.0
10.0
1.0
300.0
Number of Chains
20
10
10
10
Scale θ Scale
0.001
0.001
0.1
0.01
1.0
0.01
0.001
0.1
Bayesian Meta-Learning Models: For the models AVICI, CSIvA, and ours BCNP, we ensure the same architecture here we highlight key differences. AVICI uses max pooling to construct the summary representation, instead of attention using query vector. The summary representation decoded using a linear layer (Lorch et al., 2022). AVICI also uses an acyelic regulariser on it's marginal edge probabilities which forces the probabilities to be acyelic. Following the values used by the authors, we initialise the weight of the acyclic regulariser to 0 and update it every 500 iterations using a regulariser learning rate (Lorch et al. 2022). This regulariser learning rate is kept at value of 10 after warming up for 10% of the total iterations. In contrast, the only difference with CSIvA is that it uses an autoregressive decoder where each D elements of the adjacency are autoregressively generated. All the models have 2 encoder layers (4 transformer layers in total, but attention over samples and nodes is done twice each), while CSIvA, and BCNP also have decoder layers. For AVICI and BCNP, we use a width of 512 for the attention layers and width of 1024 for the feedforward layers. Due to memory constraints of autoregressive generation, for CSIvA we use width of 256 for the attention and 512 for the feedforward layers. We use 8 attention heads for each model. We use Adam (Kingma & Ba 2014) with a learning rate of 10 with a linear warmup of 10% of the total iterations. We use batch size of 32 for AVICI and BCNP, and a batch size of 8 for CSIvA. For BCNP, we used 100 permutation samples to approximate the loss, and maximum of 1000 sinkhorn iterations.
Training details for Bayesian meta-learning models for Syntren: To train the meta-learning models, we generate data from a mixture of all possible choices used in appendix B.2] We generate
HayeaDAA
16

---

Published as a conference paper at ICLR 2025
graphs from the ER distribution as well the scale-free distribution (Barabási & Albert 1999). The densities of the graphs are sampled from 20 to 60. Each edge function is sampled from the NeuralNet the GPCDE (described in appendix [3.] distributions with equal probability. As the Syntren datasets have 500 samples per dataset, we sample the same amount. We use 500, 000 datasets to train in total. We the same networks for the rest of the experiments, but increase the number of attention heads to 16 and increase the feedforward width of the AVICI and BCNP model to 2048. For CSIvA kept the same width as the other experiments due memory constraints.
C ADDITIONAL RESULTS
C.I RESULTS ON DATA FROM BAYESIAN CAUSAL MODELS
In this section we present the full results for each of the function type and graph density (section 4.2 Higher is better for AUC, Log Probability, and Expected Edge F1, while lower is better for Expected SHD.
For the BCNP model, those labelled ER20, ER40, ER60 are trained on datasets from the same distri- bution the test dataset. Those labelled ER20-60 are trained on mixture of the graph densities for a particular function type (Linear, NeuralNer, GPCDE), hence there one model per function type. The model labelled All Data, is single model that is trained and function types. The fact that the single model performs as well (sometimes better, c.g. GPCDE datasets), than training on single distribution type, shows that training mixture of datasets can be a useful strategy.
mixture of all graph densities
Table 7: Results for the 20 variable Linear ER20 dataset (mean std of mean).
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
DiBs
0.73 0.01
-70.53 # 1.64
25.41 0.74
0.16 0.01
BayesDAG
0.52
-515.58 # 18.76
23.02 # 0.54
0.03 # 0.01
AVICI
0.89 0.01
-51.27 1.47
26.61 0.59
0.31 0.01
CSIvA
0.51 0.01
0.11
38.36
0.00
BCNP ER20
0.90 0.01
-52.11
21.53 0.75
0.44
BCNP ER20-60
0.89 + 0.01
+2..772..4
23.47 # 0.74
0.41
BCNP All Data
0.89 0.01
-52.07 + 2.90
23.35 + 0.75
0.44
Table 8: Results for the 20 variable Linear ER40 dataset (mean std of mean).
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
DIBS
0.53 0.02
-160.71 + 4.13
72.43 2.08
0.11 0.01
BayesDAG
0.51 + 0.01
-1016.56 + 47.11
44.07 0.53
0.03 = 0.01
AVICI
0.86 1860
.94.12 1.82
56.22 0.48
0.28 # 0.01
CSIVA
0.51 0.01
-130.16 0.20
71.24 0.03
0.10
BCNP ER40
0.89 + 0.01
-89.38 # 2.39
49.68 0.56
0.37
BCNP ER20-60
0.88 0.01
-90.63 # 2.36
51.17 + 1.04
0,36 + 0.01
BCNP All Data
0.90 0.01
-86.49 + 3.03
49.01 1.05
0.40 0.01
17

---

Published as a conference paper at ICLR 2025
Table Results for the 20 variable Linear ER60 dataset (mean std of mean).
Model
AUC
Neg. Log Prob.
Expected SHD
Expected Edge F1
DiBS
0.40 + 0.01
-231.40 + 3.69
106.64 + 1.25
0.11 0.00
BayesDAG
0.51 + 0.01
-1509.27 + 48.40
63.51 + 0.40
0.04 + 0.01
AVICI
0.83 + 0.01
-126.31 + 1.71
80.90 + 0.71
0.31 0.01
CSIvA
0.49 - 0.01
-169.49 0.19
102.00 0.05
0.15 0.00
BCNP ER60
0.85 + 0.01
-121.87 + 2.29
76.52 0.92
0.36 - 0.01
BCNP ER20-60
0.84 = 0.01
122..22
74.19 + 1.08
0.33 = 0.01
BCNP All Data
0.86 = 0.01
-122.02 + 2.66
72.15 + 1.17
0.36 = 0.01
Table 10: Results for the 20 variable NeuralNet ER20 dataset (mean + std of mean).
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
DiBS
0.69 0.01
-75.31 1.19
28.48 0.20
0.12 + 0.01
BayesDAG
0.59 +.02
-19.99
90.21 = 0.20
0.09 + 0,00
AVICI
0.84 0.01
-59.21 1.15
29.51 0.37
0.24 + 0.01
CSIvA
0.51 + 0.01
38.01 = 0.02
0.05 + 0.00
BCNP ER20
0.88 0.00
-56.07
0.76
26.73 0.16
0.26 + 0.00
BCNP ER20-60
0.88 + 0.00
-53.92 + 0.98
27.55
= 0.36
0.38 + 0.01
BCNP All Data
0.88 0.01
-54.00
1.94
30.05 0.65
0.36 + 0.01
Table 11: Results for the 20 variable NeuralNet ER40 dataset (mean + std of mean).
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
DIBS
0.67 + 0.01
-139.47 + 1.96
46.81 0.21
0.13 + 0.01
BayesDAG
0.59 + 0.01
-145.35 + 0.18
97.25
0.14
0.15 + 0.00
AVICI
0.79 0.01
+
61.11 0.46
0.24 1 0.01
CSIvA
0.00
-..0.44
71.92 = 0.04
0.10 0.00
BCNP ER40
0.86 0.00
933.22
47.38 0.28
0.39 + 0.00
BCNP ER20-60
0.86 + 0.00
.95.39 1.31
49.71 = 0.38
0.37 + 0.00
BCNP All Data
0.84 0.01
43.3.
53.44 0.71
0.37 + 0.01
Table 12: Results for the 20 variable NeuralNet ER60 dataset (mean + std of mean).
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
DIBS
0.64 + 0.01
-208.22 + 1.82
65.72 = 0.17
0.13 + 0,01
BayesDAG
0.58 + 0.01
-170.73 + 0.26
103.95 + 0.09
0.19 + 0.01
AVICI
0.79 + 0.01
-138.04 + 1.36
85.15 = 0.59
0.29 + 0.01
CSIVA
0.50 0.01
169.330+:.23
101.89 + 0.05
0.15 - 0.00
BCNP ER60
-121.09 + 1.08
70.40 0.35
0.40 + 0.00
BCNP ER20-60
4.00
-133.36
+ 1.59
68.34 + 0.36
0.38 4: 0.00
BCNP All Data
0.83 + 0.01
-127.48 2.71
68.34 0.91
0.39 + 0.01
18

---

Published as a conference paper at ICLR 2025
Table 13: Results for the 20 variable GPCDE ER20 dataset (mean std mean).
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
DiBS
0.80 0.01
-63.3
1.24
26.33 0.17
0.18 + 0.01
BayesDAG
0.67 + 0.02
-137.93 + 2.63
99.69 + 0.22
0.11 + 0.00
AVICI
0.74
-71.67 0.64
36.27 0.20
0.08 + 0.01
CSIvA
0,48 =
37,93 0.01
0.05 + 0,00
BCNP ER20
0.75
-72.29 0.87
37.31 0.10
0.09 + 0.00
BCNP ER20-60
0.74
-73.29 = 0.92
41.23 = 1.01
0.00
BCNP All Data
0.83 - 0.01
-63.99 1.42
38.24 0.85
0.20 0.01
Table 14: Results for the 20 variable GPCDE ER40 dataset (mean + std of mean).
Model
AUC
Log Probability
Expected SHD
Expected Edge
DiBS
0,74 + 0.01
-125.15 + 1.52
45.04 + 0.26
0.16 + 0.00
BayesDAG
0.67 0.02
-154.09 2.25
103.09 + 0.33
0.19 + 0.01
AVICI
0.72 0.01
-116.67 1.21
67.68 = 0.33
0.15 + 0.01
CSIvA
-130.29 0.22
71.65 0.04
0.10 + 0.00
BCNP ER40
0.73
-116.11
69.68 0.30
0.16 + 0.00
BCNP ER20-60
-116.91 + 1.33
66.95 1.10
0.15 + 0.00
BCNP All Data
0.78 0.01
-00.550 + 1.58
62.05 1.17
0.22 + 0.01
Table 15: Results for the 20 variable GPCDE ER60 dataset (mean + std of mean).
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
DIBS
0.67 + 0.01
-199.87 + 2.03
64.22 0.22
0.13 + 0.00
BayesDAG
0,66 0.01
-184.48 5.12
107.46 + 0.31
0.24 + 0.00
AVICI
0.71 0.01
-151.70 + 1.28
91.33 0.36
0.21 + 0.01
CSIvA
0.49 0.01
-169.53
0.19
101.62 + 0.05
0.15 + 0.00
BCNP ER60
0.72 0.01
-149.48 1.05
93.07 0.34
0.22 1 0.01
BCNP ER20-60
0.71 0.01
1.3.97
87.03 = 0.67
0.19 + 0.01
BCNP All Data
0.77 0.01
-142.97 + 1.80
80.28 0.77
0.25 + 0.01
C.2 RESULTS FOR META-LEARNING MODELS WITH THE SAME WIDTH
Due to memory constraints, we used smaller width network for CSIvA than AVICI and BCNP. Here, we train the same width networks for all three to allow for fair comparison, The results show that CSIvA struggles with larger number of nodes.
Table 16: Results for the same width meta-learning models for Linear ER20 (mean + std of mean).
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
CSIvA
0.51 0.01
-79.79 0.13
38.36 0.01
0.05 # 0.00
AVICI
0.85 # 0.01
-61.28 # 1.54
31.91 + 0.33
0.18 # 0.01
BCNP
0.87 = 0.01
-54.52 1.56
26.68 0.62
0.31 + 0.02
19

---

Published as a conference paper at ICLR 2025
Table 17: Results for the same width meta-learning models for Linear ER40 (mean std of mean).
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
CSIvA
0.51 + 0.01
-130.16 0.20
71.24 0.03
0.10 + 0.00
AVICI
0.82 + 0.01
-102.57 0.96
60.25 + 0.34
0.21 = 0.01
BCNP
0.85 + 0.01
-98.64 1.70
57.44 0.45
0.27 + 0.01
Table 18: Results the same width meta learning models for Linear ER60 (mean + variance of mean).
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
CSIvA
0.49 + 0.00
-169.49 0.03
102.00 + 0.00
0.15 + 0.00
AVICI
0.80 + 0.00
-134.23 1.87
84.83 0.25
0.27 + 0.00
BCNP
0.82 = 0.00
-130.28 5.09
82.19 + 0.30
0.31 = 0.00
Table 19: Results for the same width meta-leaminn models for NeuralNet ER20 (mean std of mean).
Model
AUC
Log Probability
Expected SHD
Expected Edge
CSIvA AVICI
0.51 + 0.01
-79.82 + 0.21
38.01 = 0,02
0.05 + 0.00
0.78 + 0.01
-68.97 1.13
33.96 0.30
0.11 0.00
BCNP
0.70 # 0.01
-73.86 + 0.65
37.46 + 0.19
0.08 # 0.00
Table 20: Results for the same width meta-learning models for NeuralNet ER40 (mean + std of mean).
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
CSIvA
0.50 0.01
-130.44 0.22 -115.88 # 0.88
71.92 + 0.04
0.10 0.00
AVICI
0.73 # 0.01
64.97 # 0.37
0.16 # 0.00
BCNP
Model diverged
Model diverged
Model diverged
Model diverged
Table 21: Results for the same width meta-learning models for NeuralNet ER60 (mean = std of mean).
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
CSIvA
0.50 = 0.01
-169.30 0.20
101.89 + 0.05
0.15 + 0.00
AVICI
0.75 # 0.01
-146.07 1.25
88.62 + 0.48
0.23 0.00
BCNP
0.77 0.01
-141.54 1.51
87.53 + 0.49
0.26 + 0.00
Table 22: Results for the same width meta learning models for GPCDE ER20 (mean + std of mean).
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
CSIvA
0.48 0.01
-80.19 0.13
37.93 0.01
0.05 0.00
AVICI
0.74 # 0.01
-72.94 0.64
36,26 # 0.20
0.08 + 0.00
BCNP
0.74 0.01
-72.85 0.87
35.38 + 0.10
0.08 0.00
20

---

Published as a conference paper at ICLR 2025
Table 23: Results for the same width meta learning models for GPCDE ER40 (mean + std of mean).
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
CSIvA
0.51 0.01
-130.29 0.22
71.65 + 0.04
0.10 0.00
AVICI
0.72 = 0.01
-116.70 1.21
67.11 = 0.33
0.15 + 0.00
BCNP
0.72 0.01
-116.72 1.22
67.00 + 0.30
0.15 0.00
Table 24: Results for the same width meta learning models for GPCDE ER60 (mean + std of mean).
Model
AUC
Log Probability
Expected SHD
Expected Edge F1
CSIvA
0.49 = 0.01
-169.53 0.19
101.62 = 0.05
0.15 + 0.00
AVICI
0.71 # 0.01
-151.76 1.28
92.79 = 0.36
0.21 0.00
BCNP
0.70 0.01
-152.24 0.76
94.27 + 0.34
0.21 + 0.00
21


Reviews:
[{"review_id": "x3i90zKXPz", "reviewer": "Reviewer_xBXp", "summary": "This paper proposes the Bayesian Causal Neural Process (BCNP), a Neural Process-based framework for learning a posterior distribution over causal graphs given a dataset $P(G \\mid X)$ . The proposed method is scalable and captures distinctions between graphs within the same Markov equivalence class. Since BCNP learns causal structures across multiple (synthetic) datasets, it is considered a meta-learning approach.", "questions": "1. Is it nontrivial to consider the setting where latent confounders between variables exist? \n\n2. Is the proposed method computationally efficient when a function class is chosen as neural networks? \n\n3. How sensitive the proposed framework is regarding to the sparcity of the graph? \n\n4. Are there any interesting example where the Bayesian approach beats the Markov-equivalence-based method?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This paper provides a well-structured and self-contained summary of prior works in Bayesian causal discovery, allowing readers to clearly follow the evolution of Bayesian approaches for learning causal graphs. As a reviewer, this summary enables me to track the advancements in the field and understand the recent progress in addressing challenges like scalability and uncertainty in causal inference.\n\n2. The paper clearly outlines its unique contributions in Table 1, highlighting significant advancements over existing methods. The proposed model addresses critical aspects such as acyclicity, permutation invariance, and edge dependencies—all of which are critical for accurate causal graph learning. These improvements over prior approaches directly enhance the reliability and precision of inferred causal structures, making the contributions both clear and impactful.\n\n3. The proposed BCNP is particularly attractive due to its innovative use of Neural Processes to directly sample the posterior distribution $ P(G | X) $. This approach enables BCNP to achieve scalability and uncertainty-aware inference by learning across tasks (= datasets), allowing the model to efficiently adapt to new datasets while maintaining a robust representation of causal uncertainty. This combination of scalability, direct posterior sampling, and adaptability positions BCNP as a strong advancement over existing Bayesian causal discovery methods.", "weaknesses": "1. One of the main limitations of this paper is the assumption that there are no latent confounders between variables. This assumption may limit the applicability of the method to real-world datasets where unobserved confounders are common. \n\n2. I think this method is heavily relying on the assumptions on the choice of modeling class of functions F, the types of graphs and the noise model. Unlike fully nonparametric approaches like the PC algorithm, which do not impose strict functional or noise assumptions, BCNP’s effectiveness depends on these choices being well-suited to the data.\n\n3. The proposed framework may be sensitive to hyperparameter settings and architecture choices, such as the hyperparameters for the encoder-decoder network, or the types of prior distributions. Fine-tuning these parameters is often data-dependent, which could reduce the model's robustness and generalization capabiltiy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MmOq64xuXM", "reviewer": "Reviewer_CqB9", "summary": "This paper introduces a Bayesian meta-learning model for causal discovery, named the Bayesian Causal Neural Process. This \"meta-learning\" algorithm learns a model from pairs of datasets and their associated causal graphs. Specifically, the model incorporates a neural process to address model uncertainty and employs an encoder-decoder architecture carefully designed to achieve desirable properties for causal discovery in a meta-learning context. These properties include permutation invariance to sample order, permutation equivariance to nodes, and acyclicity, achieved by decomposing the adjacency matrix into a lower triangular matrix and a permutation matrix—some of which build on existing approaches. The authors demonstrate the empirical superiority of this model over other explicit Bayesian and Bayesian meta-learning models", "questions": "- For DiBS and BayesDAG, the authors mention that they “keep all other hyperparameters to their preset values.” What if these hyperparameters were tuned? Would the results differ significantly? For instance, DiBS and BayesDAG perform well on the Syntren dataset for certain metrics, so tuning may further widen this performance gap.\n- In the caption, it says, “Each dataset contains D nodes and N samples.” Does this imply that all datasets must be of the same size? What if the sample sizes vary, such as 100, 200, or 500? Should N be set to 100, or can larger datasets (e.g., 200 and 500 samples) be split into multiple parts (e.g., 2 or 5 parts) to train the model? In that case, should there be weighting adjustments (1/2, 1/5), also training perhaps only for data-related parameters and not for those related to the graph prior?\n- If there is no training involved, does your model function like a standard Bayesian causal discovery method with a uniform prior?\n- Each variable might have distinct functional priors, such as some being linear, others non-linear, discrete, or having different means. Does the encoder “implicitly” reorder these variables (perhaps using attention mechanisms) to align them by their functional type? I’m trying to conceptually grasp how the encoder handles such diverse functional characteristics under the hood.\n- What if permutation equivariance among nodes was not enforced? Would this potentially improve performance, especially if nodes are ordered consistently between training and test data?\n- In Line 382, could you clarify what is meant by “correlations between edges” when dealing with just two variables, X and Y, with at most a single edge?\n\nMinor comments\n- It would be better to have some annotations in Figure 2 (given that the figure takes lots of space)\n- Not quite sure whether it is sufficient to say the dependence in permutation and lower triangular binary matrices is modeled because they share the same representation R0.\n- 047, 137 citep for Wenzel et al. \n- 115 citing PC, GES might be good (given you have some space for 119)\n- 141 Bayesian metal → meta\n- 305 Is the grammar correct? (The matrix … and thus … )\n- 340 space before Mena\n- 389 In section section 4.2\n- 389 its \n- 414 space after comma\n- 480 enable\n- 485 period\n- BayesDAG is not highlighted in Table 4 for Linear data/SHD\n- Figure 2 shouldn’t o be the input to Theta? or onto the edge between R^L1 to Theta?\n- Sorry for suggesting my preference here but can we align Theta and Phi (parameters, vertically) and align Qs and As with the same horizontal line as Theta and Phi, respectively, and put Gs at far right.\n\n(I can change to weak accept (or accept) depending on the response and others' reviews.)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- As summarized in Table 1, this approach implements several key desiderata for a Bayesian meta-learning model more effectively than existing alternatives.\n- The paper is well-written, allowing readers to easily follow the motivation and desiderata of Bayesian meta-learning models.", "weaknesses": "- While I appreciate the overall architecture of the model, it is challenging to identify which components provide truly novel technical contributions. For instance, the results on pages 4 and 5 appear to largely rely on existing findings, and, if I'm not mistaken, page 6 includes results from Annadani et al. (2024). The contributions of this paper are not clearly or explicitly articulated.\n- The Bayesian prior is learned through pairs of datasets and directed acyclic graphs (DAGs). From my perspective, learning P(X,G) is likely very challenging. Indeed, Appendix B.1 notes that, for a case involving two variables, the model was trained on 200,000 datasets. I understand that Section 4.1 is intended to evaluate the posterior, but in Appendix B.2, 500,000 datasets are used for training. This information would be better suited for the main text, and details on computational resources should be transparently reported there as well (they are not provided even in the appendix).\n- I find it difficult to be fully convinced by the meta-learning approach to causal discovery. While I understand the importance of establishing a strong Bayesian prior, can we realistically expect access to hundreds of thousands of dataset-graph pairs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mVOgyCUePB", "reviewer": "Reviewer_fWq7", "summary": "In Bayesian causal discovery, the goal is to approximate the posterior over graphs. Since computing the posterior over graphs requires computing posterior over functional parameters, previous approaches have tried to approximate the posterior over function parameters.  This paper uses neural processes to approximate the posterior over graphs directly by learning a map from the dataset to the set of distributions. The neural process is an encoder-decoder network that claims to sample, permutation-invariant, acyclic graphs that capture edge dependencies.", "questions": "1) The ideas that the paper uses to impose acyclicity and permutation invariance seem applicable in general. Can existing meta-learning approaches be modified to get the same? While this is not the focus of the paper. It would make it stronger to highlight it if that's the case.\n\n2) The experimental validation for the case when the true posterior is known, is done only for the two-variable case. Is it possible to scale this up? \n\n3) How can the posterior over the graphs be used for inference on that of the functional parameters? \n\n4) How is the performance on Gene Regulatory Network simulated data like SERGIO? I believe it would be useful to add that in given its importance. \n\n5) The paper says that the metrics average over the posterior. I believe this is a problem with any metric. Are there alternatives that the authors propose?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper proposes a technique that can directly sample from the posterior over graphs. The main contribution of the paper is in the architecture of the encoder-decoder network that captures necessary properties like the permutation invariance by cross-attention, and ayclic DAGs and edge dependencies by sampling the decomposition of a DAG into permutation and lower-triangular matrices. The comparison of the proposed model is done with existing meta-learning approaches and Bayesian approaches using multiple metrics. The proposed model is shown to be competitive on synthetic and simulated datasets. The paper is written clearly and flows well.", "weaknesses": "My main concern is that of this being an incremental contribution. The framework is not new and the only contributions I see are a change in how permutation invariance and acyclicity is incorporated. Please let me know if I am missing something else. I have a detailed list of questions below whose answers might help strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H441SdGqu5", "reviewer": "Reviewer_DiiJ", "summary": "The paper introduces the Bayesian Causal Neural Process (BCNP), a novel approach for Bayesian causal discovery that leverages meta-learning to address challenges in traditional methods. BCNP learns a mapping from datasets to posterior distributions over causal graphs, thereby bypassing the need to explicitly infer the posterior over causal mechanisms and allowing for efficient sampling from high-dimensional spaces. By encoding key properties like permutation equivariance and edge dependencies, BCNP accurately approximates the true posterior and generates acyclic graph samples. The paper demonstrates BCNP's effectiveness through experiments on synthetic and semi-synthetic datasets, showcasing its superior performance compared to existing Bayesian and meta-learning models for causal discovery.", "questions": "1. As my main concerns, I would appreciate the authors to elaborate on the abovementioned weaknesses.\n2. Given that the proposed method does not utilize interventional data, using CPDAG E-SHD (as used in previous papers such as in BayesDAG) might be more reflective of the performance of the models compared with E-SHD.\n3. Have the authors tried any experiments on low data regime where the number of samples is close to the number of nodes? This is particularly important since Bayesian Causal Discovery is most desirable in low data regime where $n \\approx d$. Do you have an intuition about how the proposed model performs in these particular settings?\n4. In several experiments, other baselines such as BayesDAG and DiBS achieve a better E-SHD compared to the proposed method. Could you elaborate on why this happens?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written and clear and includes a range of well-designed experiments.", "weaknesses": "For me, there are two major concerns regarding the novelty and contributions of the proposed method to the community:\n\n1. The paper presents the direct mapping from data to a posterior over graphs ($P(G|D)$) as a main advantage, bypassing the need to model causal mechanisms. However, learning a distribution over causal mechanisms is crucial for many applications, such as causal inference and treatment effect estimation [1, 2]. Could the authors elaborate on potential applications of the proposed approach beyond the discovery of the causal graph?\n2. The proposed method builds on several modules from existing work. For instance, the representation of the posterior distribution as permutation and upper triangular matrices is also similar to [3]. Although the method outperforms baselines in graph-based metrics in causal discovery, the specific novelty remains unclear, particularly given the first concern.\n\n[1] Emezue, C. C., et al. (2023). Benchmarking Bayesian Causal Discovery Methods for Downstream Treatment Effect Estimation. arXiv, 2307.04988. https://arxiv.org/abs/2307.04988v3\n\n[2] Schölkopf, B., et al. (2021). Towards Causal Representation Learning. arXiv, 2102.11107. https://arxiv.org/abs/2102.11107v1\n\n[3] Cundy, C., et al. (2021). BCD Nets: Scalable Variational Approaches for Bayesian Causal Discovery. arXiv, 2112.02761. https://arxiv.org/abs/2112.02761v1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)