SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For (b), we show that for any DAG  $G$ , there exists a  $(W, p)$  pair s.t.  $\tau(W, p) = G$ . To see this, we can observe that  $p$  implicitly defines a topological order in the mapping  $\tau$ . For any  $p_i > p_j$ , we have  $j \to i$  after the mapping Step(grad  $p$ ). Thus, by leveraging Theorem 3.7 in [72], we obtain that there exists a potential vector  $p \in \mathbb{R}^d$  for any DAG  $G$  such that
$$
(\operatorname {g r a d} \boldsymbol {p}) (i, j) > 0 \quad \text {w h e n} G _ {i j} = 1
$$
Thus, we can choose  $\mathbf{W}$  in the following way:
$$
\boldsymbol {W} = \left\{ \begin{array}{l l} W _ {i j} = 0 & \text {i f} G _ {i j} = 0 \\ W _ {i j} = 1 & \text {i f} G _ {i j} = 1 \end{array} \right.
$$
Next, let's prove the Theorem 3.1.
Proof of Theorem 3.1. From Lemma B.1, we see that the mapping is complete. Namely, the  $(W,p)$  space can represent the entire DAG space. Next, we show that performing Bayesian inference in  $(W,p)$  space can also correspond to the inference in DAG space.
Assume we have the prior  $p_{\tau}(\pmb{W}, \pmb{p})$ . Then through mapping  $\tau$ , we implicitly define a prior over the DAG  $\pmb{G}$  in the following:
$$
p _ {\tau} (\boldsymbol {G}) = \int p _ {\tau} (\boldsymbol {W}, \boldsymbol {p}) \mathbb {1} (\boldsymbol {G} = \tau (\boldsymbol {W}, \boldsymbol {p})) d \boldsymbol {W} d \boldsymbol {p} \tag {20}
$$
This basically states that the corresponding prior over  $G$  is an accumulation of the corresponding probability associated with  $(W, p)$  pairs.
Similarly, we can define a corresponding posterior  $p_{\tau}(\boldsymbol {G}|\boldsymbol {D})$  ..
$$
p _ {\tau} (\boldsymbol {G} | \boldsymbol {D}) = \int p _ {\tau} (\boldsymbol {W}, \boldsymbol {p} | \boldsymbol {D}) \mathbb {1} (\boldsymbol {G} = \tau (\boldsymbol {W}, \boldsymbol {p})) d \boldsymbol {W} d \boldsymbol {p} \tag {21}
$$
Now, let's show that this posterior  $p_{\tau}(\pmb{G}|\pmb{D}) = p(\pmb{G}|\pmb{D})$  if prior matches, i.e.  $p(\pmb{G}) = p_{\tau}(\pmb{G})$ . From Bayes's rule, we can easily write down
$$
p _ {\tau} (\boldsymbol {W}, \boldsymbol {p} | \boldsymbol {D}) = \frac {p (\boldsymbol {D} | \boldsymbol {G} = \tau (\boldsymbol {W} , \boldsymbol {p})) p (\boldsymbol {p} , \boldsymbol {W})}{\sum_ {\boldsymbol {G} ^ {\prime} \in \mathbb {D}} p (\boldsymbol {D} , \boldsymbol {G} ^ {\prime})} \tag {22}
$$
Then, by substituting Equation (22) into Equation (21), we have
$$
\begin{array}{l} p _ {\tau} (\boldsymbol {G} | \boldsymbol {D}) = \int \frac {p (\boldsymbol {D} | \boldsymbol {G}) p _ {\tau} (\boldsymbol {W} , \boldsymbol {p})}{\sum_ {\boldsymbol {G} ^ {\prime} \in \mathbb {D}} p (\boldsymbol {D} , \boldsymbol {G} ^ {\prime})} \mathbb {1} (\boldsymbol {G} = \tau (\boldsymbol {W}, \boldsymbol {p})) d \boldsymbol {W} d \boldsymbol {p} \\ = \frac {\int p (\boldsymbol {D} | \boldsymbol {G}) p _ {\tau} (\boldsymbol {W} , \boldsymbol {p}) \mathbb {1} (\boldsymbol {G} = \tau) d \boldsymbol {W} d \boldsymbol {p}}{\sum_ {\boldsymbol {G} ^ {\prime} \in \mathbb {D}} p (\boldsymbol {D} , \boldsymbol {G} ^ {\prime})} (23) \\ = \frac {p (\boldsymbol {D} | \boldsymbol {G}) \int p _ {\tau} (\boldsymbol {W} , \boldsymbol {p}) \mathbb {1} (\boldsymbol {G} = \tau) d \boldsymbol {W} d \boldsymbol {p}}{\sum_ {\boldsymbol {G} ^ {\prime} \in \mathbb {D}} p (\boldsymbol {D} , \boldsymbol {G} ^ {\prime})} (24) \\ = \frac {p (\boldsymbol {D} | \boldsymbol {G}) p _ {\tau} (\boldsymbol {G})}{\sum_ {\boldsymbol {G} ^ {\prime} \in \mathbb {D}} p (\boldsymbol {D} | \boldsymbol {G} ^ {\prime}) p _ {\tau} (\boldsymbol {G} ^ {\prime})} \\ = p (\boldsymbol {G} | \boldsymbol {D}) (25) \\ \end{array}
$$
where Equation (23) is from the fact that  $\sum_{\pmb{G}^{\prime}\in \mathbb{D}}p(\pmb {D},\pmb{G}^{\prime})$  is independent of  $(W,p)$  due to marginalization. Equation (24) is obtained because  $p(D|G)$  is also independent of  $(W,p)$  due to (1)  $\mathbb{1}(\pmb {G} = \tau (\pmb {W},\pmb {p}))$  and (2)  $p(\pmb {D}|\pmb {G})$  is a constant when fixing  $\pmb{G}$ . Equation (25) is obtained by applying Bayes's rule and  $p_{\tau}(G) = p(G)$ .
Theorem 3.2 (Equivalence of NoCurl formulation). Assuming the conditions in Theorem 3.1 are satisfied. Then, for a given  $(\mathbf{W},\mathbf{p})$ , we have
$$
\boldsymbol {G} = \boldsymbol {W} \odot \operatorname {S t e p} (\operatorname {g r a d} \boldsymbol {p}) = \boldsymbol {W} \odot \left[ \boldsymbol {\sigma} ^ {*} (\boldsymbol {p}) \boldsymbol {L} \boldsymbol {\sigma} ^ {*} (\boldsymbol {p}) ^ {T} \right]
$$
where  $G$  is a DAG and  $\sigma^{*}(\pmb{p})$  is defined in Equation (7).
To prove this theorem, we need to first prove the following lemma.
Lemma B.2. For any permutation matrix  $M \in \Sigma_d$ , we have
$$
\operatorname {g r a d} (M \boldsymbol {p}) = M ^ {T} \operatorname {g r a d} (p) M
$$
where grad is the operator defined in Equation (3).
Proof. By definition of  $\operatorname{grad}(\cdot)$ , we have
$$
\begin{array}{l} \operatorname {g r a d} (M p) = (M p) _ {i} - (M p) _ {j} \\ = \mathbf {1} (i) ^ {T} \boldsymbol {M} \boldsymbol {p} - \mathbf {1} (j) ^ {T} \boldsymbol {M} \boldsymbol {p} \\ = M _ {i,:} \boldsymbol {p} - M _ {j,:} \boldsymbol {p} \\ \end{array}
$$
where  $\mathbf{1}(i)$  is a one-hot vector with  $i^{\mathrm{th}}$  entry 1, and  $M_{i,\cdot}$  is the  $i^{\mathrm{th}}$  row of matrix  $M$ . The above is equivalent to computing the grad with new labels obtained by permuting  $\pmb{p}$  with  $M$ . Therefore, we can see that  $\operatorname{grad}(Mp)$  can be computed by permuting the original  $\operatorname{grad}(p)$  by matrix  $M$ .
$$
\operatorname {g r a d} (M \boldsymbol {p}) = M ^ {T} \operatorname {g r a d} (\boldsymbol {p}) M
$$
Proof of Theorem 3.2. Since  $\mathbf{W}$  plays the same role in both formulations, we focus on the equivalence of Step(grad(·)).
Define a sorted  $\tilde{\pmb{p}} = \pmb{\sigma}\pmb{p}$ , where  $\pmb{\sigma} \in \Sigma_{d}$ , such that for  $i < j$ , we have  $\tilde{p}_i > \tilde{p}_j$ . Namely,  $\pmb{\sigma}$  is a permutation matrix. Thus, we have
$$
\operatorname {g r a d} (\boldsymbol {p}) = \operatorname {g r a d} \left(\boldsymbol {\sigma} ^ {T} \tilde {\boldsymbol {p}}\right).
$$
By Lemma B.2, we have
$$
\operatorname {g r a d} \left(\boldsymbol {\sigma} ^ {T} \tilde {\boldsymbol {p}}\right) = \boldsymbol {\sigma} \operatorname {g r a d} (\tilde {\boldsymbol {p}}) \boldsymbol {\sigma} ^ {T}.
$$
Since  $\tilde{\pmb{p}}$  is an ordered vector. Therefore,  $\operatorname {grad}(\tilde{\pmb{p}})$  is a skew-symmetric matrix with a positive lower half part.
$$
\operatorname {S t e p} (\operatorname {g r a d} (\boldsymbol {p})) = \operatorname {S t e p} (\boldsymbol {\sigma} \operatorname {g r a d} (\tilde {\boldsymbol {p}}) \boldsymbol {\sigma} ^ {T}) = \boldsymbol {\sigma} \operatorname {S t e p} (\operatorname {g r a d} (\tilde {\boldsymbol {p}})) \boldsymbol {\sigma} ^ {T} = \boldsymbol {\sigma} \boldsymbol {L} \boldsymbol {\sigma} ^ {T}
$$
This is true because  $\sigma$  is just a permutation matrix that does not alter the sign of  $\operatorname{grad}(\tilde{p})$ .
Since  $\sigma$  is a permutation matrix that sort  $p$  value in a ascending order, from Lemma 1 in [8], we have
$$
\boldsymbol {\sigma} = \underset {\boldsymbol {\sigma} ^ {\prime} \in \boldsymbol {\Sigma} _ {d}} {\arg \max } \boldsymbol {p} ^ {T} (\boldsymbol {\sigma} ^ {\prime} \boldsymbol {o})
$$
B.3 Proof of Proposition A.1
$$
\begin{array}{l} \mathbb {E} _ {p (\boldsymbol {p}, \boldsymbol {W}, \boldsymbol {\Theta} | D)} [ f (\boldsymbol {G} = \tau (\boldsymbol {p}, \boldsymbol {W}), \boldsymbol {\Theta}) ] \\ = \int p (\boldsymbol {p}, \boldsymbol {W}, \boldsymbol {\Theta}, \tilde {\boldsymbol {W}} | \boldsymbol {D}) f (\boldsymbol {G}, \boldsymbol {\Theta}) d \boldsymbol {p} d \boldsymbol {W} d \boldsymbol {\Theta} d \tilde {\boldsymbol {W}} \\ = \int p (\boldsymbol {p}, \tilde {\boldsymbol {W}}, \Theta | D) p (\boldsymbol {W} | \boldsymbol {p}, \Theta , \tilde {\boldsymbol {W}}, D) f (\boldsymbol {G}, \Theta) d \boldsymbol {p} d \boldsymbol {W} d \Theta d \tilde {\boldsymbol {W}} \\ = \mathbb {E} _ {p (\boldsymbol {p}, \tilde {\boldsymbol {W}}, \boldsymbol {\Theta} | D)} \left[ \frac {\int p (\boldsymbol {D} | \boldsymbol {p} , \boldsymbol {\Theta} , \boldsymbol {W}) p (\boldsymbol {p}) p (\tilde {\boldsymbol {W}}) p (\boldsymbol {W} | \tilde {\boldsymbol {W}}) p (\boldsymbol {\Theta} | \boldsymbol {p} , \boldsymbol {W}) f (\boldsymbol {G} , \boldsymbol {\Theta}) d \boldsymbol {W}}{\int p (\boldsymbol {D} | \boldsymbol {p} , \boldsymbol {\Theta} , \boldsymbol {W}) p (\boldsymbol {p}) p (\tilde {\boldsymbol {W}}) p (\boldsymbol {W} | \tilde {\boldsymbol {W}}) p (\boldsymbol {\Theta} | \boldsymbol {p} , \boldsymbol {W}) d \boldsymbol {W}} \right] \\ = \mathbb {E} _ {p (\boldsymbol {p}, \tilde {\boldsymbol {W}}, \boldsymbol {\Theta})} \left[ \frac {\mathbb {E} _ {p (\boldsymbol {W} | \tilde {\boldsymbol {W}})} [ f (\boldsymbol {G} , \boldsymbol {\Theta}) p (\boldsymbol {D} , \boldsymbol {\Theta} | \boldsymbol {p} , \boldsymbol {W}) ]}{\mathbb {E} _ {p (\boldsymbol {W} | \tilde {\boldsymbol {W}})} [ p (\boldsymbol {D} , \boldsymbol {\Theta} | \boldsymbol {p} , \boldsymbol {W}) ]} \right] \\ \end{array}
$$
B.4 Proof of Proposition A.2
$$
\begin{array}{l} \nabla_ {\boldsymbol {p}} U (\boldsymbol {p}, \tilde {\boldsymbol {W}}, \boldsymbol {\Theta}) = - \nabla_ {\boldsymbol {p}} \log p (\boldsymbol {p}, \tilde {\boldsymbol {W}}, \boldsymbol {\Theta}, \boldsymbol {D}) \\ = - \nabla_ {\boldsymbol {p}} \log p (\boldsymbol {p}) - \nabla_ {\boldsymbol {p}} \log p (\bar {\boldsymbol {W}}, \Theta , D | \boldsymbol {p}) \\ = - \nabla_ {\boldsymbol {p}} \log p (\boldsymbol {p}) - \frac {\nabla_ {\boldsymbol {p}} \int p (\boldsymbol {D} | \boldsymbol {W} , \boldsymbol {p} , \boldsymbol {\Theta}) p (\boldsymbol {\Theta} | \boldsymbol {p} , \boldsymbol {W}) p (\boldsymbol {W} | \tilde {\boldsymbol {W}}) p (\tilde {\boldsymbol {W}}) d \boldsymbol {W}}{\int p (\boldsymbol {D} | \boldsymbol {W} , \boldsymbol {p} , \boldsymbol {\Theta}) p (\boldsymbol {\Theta} | \boldsymbol {p} , \boldsymbol {W}) p (\boldsymbol {W} | \tilde {\boldsymbol {W}}) p (\tilde {\boldsymbol {W}}) d \boldsymbol {W}} \\ = - \nabla_ {\boldsymbol {p}} \log p (\boldsymbol {p}) - \frac {\nabla_ {\boldsymbol {p}} \mathbb {E} _ {p (\boldsymbol {W} | \tilde {\boldsymbol {W}})} [ p (\boldsymbol {D} | \boldsymbol {W} , \boldsymbol {p} , \boldsymbol {\Theta}) ]}{\mathbb {E} _ {p (\boldsymbol {W} | \tilde {\boldsymbol {W}})} [ p (\boldsymbol {D} | \boldsymbol {W} , \boldsymbol {p} , \boldsymbol {\Theta}) ]} \\ = - \nabla_ {\boldsymbol {p}} \log p (\boldsymbol {p}) - \nabla_ {\boldsymbol {p}} \log \mathbb {E} _ {p (\boldsymbol {W} | \tilde {\boldsymbol {W}})} [ p (\boldsymbol {D} | \boldsymbol {W}, \boldsymbol {p}, \boldsymbol {\Theta}) ] \\ \end{array}
$$
Other gradient  $\nabla_{\tilde{\boldsymbol{W}}}U$  and  $\nabla_{\Theta}U$  can be derived using the similar approach, which concludes the proof.
B.5 Proof of Proposition 4.1
Proof of Proposition 4.1. By definition, we have easily have
$$
\begin{array}{l} \nabla_ {\boldsymbol {p}} U = - \nabla_ {\boldsymbol {p}} \log p (\boldsymbol {p}, \boldsymbol {W}, \boldsymbol {\Theta}, \boldsymbol {D}) \\ = - \nabla_ {\boldsymbol {p}} \log p (\boldsymbol {p}, \boldsymbol {W}) - \nabla_ {\boldsymbol {p}} \log p (\boldsymbol {D}, \boldsymbol {\Theta} | \tau (\boldsymbol {W}, \boldsymbol {p})) \\ = - \nabla_ {\boldsymbol {p}} \log p (\boldsymbol {p}, \boldsymbol {W}) - \nabla_ {\boldsymbol {p}} \log p (\boldsymbol {D} | \boldsymbol {\Theta}, \tau (\boldsymbol {W}, \boldsymbol {p})) + \underbrace {\nabla_ {\boldsymbol {p}} \log p (\boldsymbol {\Theta} | \tau (\boldsymbol {p} , \boldsymbol {W}))} _ {0} \\ \end{array}
$$
$$
\begin{array}{l} \nabla_ {\Theta} U = - \nabla_ {\Theta} \log p (\boldsymbol {p}, \boldsymbol {W}, \Theta , \boldsymbol {D}) \\ = - \nabla_ {\Theta} \log p (\boldsymbol {D} | \Theta , \tau (\boldsymbol {W}, \boldsymbol {p})) - \nabla_ {\Theta} \log p (\Theta | \boldsymbol {p}, \boldsymbol {W}) - \underbrace {\nabla_ {\Theta} \log p (\boldsymbol {p} , \boldsymbol {W})} _ {0} \\ = - \nabla_ {\Theta} \log p (\boldsymbol {D} | \Theta , \tau (\boldsymbol {W}, \boldsymbol {p})) - \nabla_ {\Theta} \log p (\Theta) \\ \end{array}
$$
B.6 Derivation of ELBO
$$
\begin{array}{l} \log p (\boldsymbol {p}, \boldsymbol {\Theta}, \boldsymbol {D}) = \log \int p (\boldsymbol {p}, \boldsymbol {\Theta}, \boldsymbol {D}, \boldsymbol {W}) d \boldsymbol {W} \\ = \log \int \frac {q _ {\phi} (\boldsymbol {W} | \boldsymbol {p})}{q _ {\phi} (\boldsymbol {W} | \boldsymbol {p})} p (\boldsymbol {p}, \boldsymbol {\Theta}, \boldsymbol {D}, \boldsymbol {W}) d \boldsymbol {W} \\ \geq \int q _ {\phi} (\boldsymbol {W} | \boldsymbol {p}) \log p (\boldsymbol {p}, \boldsymbol {\Theta}, \boldsymbol {D} | \boldsymbol {W}) d \boldsymbol {W} + \int q _ {\phi} (\boldsymbol {W} | \boldsymbol {p}) \log \frac {p (\boldsymbol {W})}{q _ {\phi} (\boldsymbol {W} | \boldsymbol {p})} d \boldsymbol {W} \tag {26} \\ = \mathbb {E} _ {q _ {\phi} (\boldsymbol {W} | \boldsymbol {p})} [ \log p (\boldsymbol {p}, \boldsymbol {\Theta}, \boldsymbol {D} | \boldsymbol {W}) ] - D _ {\mathrm {K L}} [ q _ {\phi} (\boldsymbol {W} | \boldsymbol {p}) \| p (\boldsymbol {W}) ] \\ \end{array}
$$
where the Equation (26) is obtained by Jensen's inequality.
C SG-MCMC Update
Assume we want to draw samples  $\pmb{p} \sim p(\pmb{p}|\pmb{D},\pmb{W},\Theta) \propto \exp(-U(\pmb{p},\pmb{W},\Theta))$  with  $U(\pmb{p},\pmb{W},\Theta) = -\log p(\pmb{p},\pmb{W},\Theta)$ , we can compute  $U$  by
$$
U (\boldsymbol {p}, \boldsymbol {W}, \boldsymbol {\Theta}) = - \sum_ {n = 1} ^ {N} \log p (\boldsymbol {x} _ {n} | \boldsymbol {G} = \tau (\boldsymbol {W}, \boldsymbol {p}), \boldsymbol {\Theta}) - \log p (\boldsymbol {p}, \boldsymbol {W}, \boldsymbol {\Theta}) \tag {27}
$$
In practice, we typically use mini-batches  $S$  instead of the entire dataset  $D$ . Therefore, an approximation is
$$
\tilde {U} (\boldsymbol {p}, \boldsymbol {W}, \boldsymbol {\Theta}) = - \frac {| \boldsymbol {D} |}{| \mathcal {S} |} \sum_ {n \in \mathcal {S}} \log p (\boldsymbol {x} _ {n} | \boldsymbol {G} = \tau (\boldsymbol {W}, \boldsymbol {p}), \boldsymbol {\Theta}) - \log p (\boldsymbol {p}, \boldsymbol {W}, \boldsymbol {\Theta}) \tag {28}
$$
where  $|\mathcal{S}|$  and  $|D|$  are the minibatch and dataset sizes, respectively.
[28] uses the preconditioning technique on stochastic gradient Hamiltonian Monte Carlo(SG-HMC), similar to the preconditioning technique in [41]. In particular, they use a moving-average approximation of diagonal Fisher information to adjust the momentum. The transition dynamics at step  $t$  with EM discretization is
$$
B = \frac {1}{2} l
$$
$$
\boldsymbol {V} _ {t} = \beta_ {2} \boldsymbol {V} _ {t - 1} + (1 - \beta_ {2}) \nabla_ {\boldsymbol {p}} \tilde {U} (\boldsymbol {p}, \boldsymbol {W}, \boldsymbol {\Theta}) \odot \nabla_ {\boldsymbol {p}} \tilde {U} (\boldsymbol {p}, \boldsymbol {W}, \boldsymbol {\Theta})
$$
$$
g _ {t} = \frac {1}{\sqrt {1 + \sqrt {\boldsymbol {V} _ {t}}}}
$$
$$
\boldsymbol {r} _ {t} = \beta_ {1} \boldsymbol {r} _ {t - 1} - l g _ {t} \nabla_ {\boldsymbol {p}} \tilde {U} (\boldsymbol {p}, \boldsymbol {W}, \boldsymbol {\Theta}) + l \frac {\partial g _ {t}}{\partial \boldsymbol {p} _ {t}} + s \sqrt {2 l \left(\frac {1 - \beta_ {1}}{l} - B\right)} \eta
$$
$$
\boldsymbol {p} _ {t} = \boldsymbol {p} _ {t - 1} + l g _ {t} \boldsymbol {r} _ {t} \tag {29}
$$
where  $l^2$  is the learning rate;  $(\beta_1, \beta_2)$  controls the preconditioning decay rate,  $\eta$  is the Gaussian noise with 0 mean and unit variance, and  $s$  is the hyperparameter controlling the level of injected noise to SG-MCMC. Throughout the paper, we use  $(\beta_1, \beta_2) = (0.9, 0.99)$  for all experiments.
D Experimental Settings
D.1Baselines
For all the experimental settings, we compare with the following baselines:
- Bootstrap GES (BGES) [20, 14] is a bootstrap based quasi-Bayesian approach for linear Gaussian models which first resamples with replacement data points at random and then estimates a linear SCM using the GES algorithm [14] for each bootstrap set. GES is a score based approach to learn a point estimate of a linear Gaussian SCM. For all the experimental settings, we use 50 bootstrap sets.
- Differentiable DAG Sampling (DDS) is a VI based approach to learn distribution over DAGs and a point estimate over the nonlinear functional parameters. DDS performs inference on the node permutation matrices, thus directly generating DAGs. Gumbel-sinkhorn [47] is used for obtaining valid gradients and Hungarian algorithm is used for the straight-through gradient estimator [7]. In the author provided implementation, for evaluation, a single permutation matrix is sampled and the logits of the edge beliefs are directly thresholded. In this work, in order to make the comparison fair to Bayesian learning methods, we directly sample the binary adjacency matrix based on the edge logits.
- BCD Nets [16] is a VI based fully Bayesian structure learning approach for linear causal models. BCD performs inference on both the node permutations through the Gumbel-sinkhorn [47] operator as well as the model parameters through a VI distribution. Both DDS and BCD nets operate directly on full rank initializations to the Gumbel-sinkhorn operator, unlike our rank-1 initialization, which saves computations in practice.
- DIBS [43] uses SVGD [42] with the DAG regularizer [77] and bilinear embeddings to perform inference over both linear and nonlinear causal models. As our data generation process involves SCM with unequal noise variance, we extend DIBS framework with an inference over noise variance using SVGD, similar to the original paper.
While DIBS and DDS can handle nonlinear parameterization, approaches like BGES and BCD, which are primarily designed for linear models still give competitive results when applied on nonlinear data. Given that there are limited number of baselines in the nonlinear case, and DIBS being the only fully Bayesian nonlinear baseline, we compare with BGES and BCD for all settings despite their model misspecification.
D.2 Evaluation Metrics
For higher dimensional settings with nonlinear models, the true posterior is intractable. While in general it is hard to evaluate the posterior inference quality in high dimensions, prior work has suggested to evaluate on proxy metrics which we adopt in this work as well [43, 22, 4]. In particular, we evaluate the following metrics:
- E-SHD: Structural Hamming Distance (SHD) measures the hamming distance between graphs. In particular, it is a measure of number of edges that are to be added, removed or reversed to get the ground truth from the estimated graph. Since we have a posterior distribution  $q(\pmb{G})$  over graphs, we measure the expected SHD:
$$
\mathbb {E} \text {- S H D} := \mathbb {E} _ {\boldsymbol {G} \sim q (\boldsymbol {G})} [ \mathrm {S H D} (\boldsymbol {G}, \boldsymbol {G} ^ {G T}) ] \approx \frac {1}{N _ {e}} \sum_ {i = 1} ^ {N _ {e}} [ \mathrm {S H D} (\boldsymbol {G} ^ {(i)}, \boldsymbol {G} ^ {G T}) ], \text {w i t h} \boldsymbol {G} ^ {(i)} \sim q (\boldsymbol {G})
$$
where  $G^{GT}$  is the ground-truth causal graph.
- Edge F1: It is F1 score of each edge being present or absent in comparison to the true edge set, averaged over all edges.
- NLL: We also measure the negative log-likelihood of the held-out data, which is also typically used as a proxy metric on evaluating the posterior inference quality [26, 44, 63].
The first two metrics measure the goodness of the graph posterior while the NLL measures the goodness of the joint posterior over the entire causal model.
For  $d = 5$  with linear models (unequal noise variance, identifiable upto MEC [53, 34]), we evaluate the following metrics:
- MMD True Posterior: Since the true posterior is tractable, we compare the approximation with the ground truth using a Maximum Mean Discrepancy (MMD) metric [31]. If  $\mathrm{P} \coloneqq p(\boldsymbol{G} \mid \boldsymbol{D})$  is the marginalized true posterior over graphs and  $\mathrm{Q}$  is the approximated posterior over graphs, then the MMD between these two distributions is defined as:
$$
\operatorname {M M D} ^ {2} (\mathrm {P}, \mathrm {Q}) = \mathbb {E} _ {\boldsymbol {G} \sim \mathrm {P}} [ k (\boldsymbol {G}, \boldsymbol {G}) ] + \mathbb {E} _ {\boldsymbol {G} \sim \mathrm {Q}} [ k (\boldsymbol {G}, \boldsymbol {G}) ] - 2 \mathbb {E} _ {\boldsymbol {G} \sim \mathrm {P}, \boldsymbol {G} ^ {\prime} \sim \mathrm {Q}} [ k (\boldsymbol {G}, \boldsymbol {G} ^ {\prime}) ]
$$
where  $k(\pmb{G},\pmb{G}^{\prime}) = 1 - \frac{H(\pmb{G},\pmb{G}^{\prime})}{d^{2}}$  is the Hamming kernel, and  $H$  is the Hamming distance between  $\pmb{G}$  and  $\pmb{G}^{\prime}$ . This requires just the samples from the true posterior and the model. For calculating the true posterior which involves marginalization of the model parameters, appropriate prior over these parameters are required. This is ensured by using BGe score [23, 39] which places a Gaussian Wishart prior on the parameters. This leads to closed form marginal likelihood which is distribution equivalent, i.e. all graphs within the MEC will have the same likelihood. In addition, due to the low dimensionality ( $d = 5$ ), we can enumerate all possible DAGs and compute the normalizing constant  $p(D)$ . We refer to [23] for details. This metric has been used in prior work [3].
- E-CPDAG SHD: An MEC can be represented by a Completed Partially Directed Acyclic Graph (CPDAG) [54] which contains both directed edges and arcs (undirected edges). When causal relations between certain set of variables can be established, a directed edge is present. If there is an association between a certain set of variables for which causal direction is not identifiable, an undirected edge is present. For any graph, it has a corresponding CPDAG associated to the MEC which it belongs to. Since the ground truth graph is identifiable only upto MEC, we compare the (structural) Hamming distance between the graph posterior and the CPDAG of the ground truth. This is done by computing the E-CPDAG SHD:
$$
\mathbb {E} \text {- C P D A G S H D} := \mathbb {E} _ {\boldsymbol {G} \sim q (\boldsymbol {G})} [ \mathrm {S H D} (\boldsymbol {G} _ {\mathrm {C P D A G}}, \boldsymbol {G} _ {\mathrm {C P D A G}} ^ {G T}) ] \approx \frac {1}{N _ {e}} \sum_ {i = 1} ^ {N _ {e}} [ \mathrm {S H D} (\boldsymbol {G} _ {\mathrm {C P D A G}} ^ {(i)}, \boldsymbol {G} _ {\mathrm {C P D A G}} ^ {G T}) ]
$$
with  $G^{(i)}\sim q(G)$  and  $G_{CPDAG}^{GT}$  is the ground-truth CPDAG.
D.3 Synthetic Data
As knowledge of ground truth graph is not possible in many real world settings, it is standard across causal discovery to benchmark in synthetic data settings. Following prior work, we generate synthetic data by first sampling a DAG at random from either Erdos-Renyi (ER) [19] or Scale-Free (SF) [5] family. For  $d = 5$ , we ensure that the graphs have  $d$  edges in expectation and  $2d$  edges for  $d > 5$ . The ground truth parameters for linear functions are drawn at random from a fixed range of [0.5, 1.5]. For nonlinear models, the nonlinear functions are defined by randomly initialized Multi-Layer Perceptrons (MLP) with a single hidden layer of 5 nodes and ReLU nonlinearity. The variance of the exogenous Gaussian noise variable is drawn from an Inverse Gamma prior with concentration  $\alpha = 1.5$  and rate  $\beta = 1$ . For  $d = 5$  linear case, we sample at random  $N = 500$  samples from the SCM for training and  $N = 100$  for held-out evaluation. For higher dimensional settings, we consider  $N = 5000$  random samples for training and  $N = 1000$  samples for held-out evaluation. For all settings, we evaluate on 30 random datasets.
D.4 Hyperparameter Selection
In this section, we will give the details our how to select the hyperparameters for our method and all the baseline models.