SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We employ a cross-validation-like procedure for hyperparameter tuning in BayesDAG and DIBS to optimize MMD true posterior (for  $d = 5$  linear setting) and  $\mathbb{E} - \mathrm{SHD}$  value (for nonlinear setting). For each ER and SF dataset with varying dimensions, we initially generate five tuning datasets. After determining the optimal hyperparameters, we fix them and evaluate the models on 30 test datasets. For DDS, we adopt the hyperparameters provided in the original paper [11]. BCD and BGES do not necessitate hyperparameter tuning since BCD already incorporates the correct prior graph for ER and SF datasets. For semi-synthetic Syntren and real world Sachs protein cells datasets, we assume the
BayesDAG
λs	Scale p	Scale Θ	Sparse Init.
linear ER d = 5	50	0.001	0.001	False
linear SF d = 5	50	0.01	0.001	False
nonlinear ER d = 20	300	0.01	0.01	False
nonlinear SF d = 20	200	0.1	0.1	False
nonlinear ER d = 30	500	1	0.01	False
nonlinear SF d = 30	300	0.01	0.01	False
nonlinear ER d = 50	500	0.01	0.01	True
nonlinear SF d = 50	300	0.1	0.01	False
nonlinear ER d = 70	700	0.1	0.01	True
nonlinear SF d = 70	300	0.01	0.01	False
nonlinear ER d = 100	700	0.1	0.01	False
nonlinear SF d = 100	700	0.1	0.01	False
SynTren	300	0.1	0.01	False
Sachs Protein Cells	1200	0.1	0.01	False
Table 3: The hyperparameter selection for BayesDAG for each setting.
DIBS
α	h latent	hθ	hσ
linear ER d = 5	0.02	5	1000	1
linear SF d = 5	0.02	15	500	1
nonlinear ER d = 20	0.02	5	1500	10
nonlinear SF d = 20	0.2	5	1500	10
nonlinear ER d = 30	0.2	5	500	1
nonlinear SF d = 30	0.2	5	1000	1
nonlinear ER d = 50	0.2	5	500	10
nonlinear SF d = 50	0.2	5	1500	1
SynTren	0.2	5	500	10
Sachs Protein Cells	0.2	5	500	10
Table 4: The hyperparameter selection for DIBS for each setting.
number of edges in the ground truth graphs are known and we tune our hyperparameters to produce roughly correct number of edges. BCD and DIBS also assume access to the ground truth edge number and use the graph prior to enforce the number of edges.
Network structure We use one hidden layer MLP with hidden size of  $\max(4*d,64)$  for the nonlinear functional relations, where  $d$  is the dimensionality of dataset. We use LeakyReLU as the activation function. We also enable the LayerNorm and residual connections in the network. In particular, for variational network  $\mu_{\phi}$  in BayesDAG, we apply the LayerNorm on  $p$  before inputting it to the network. We use 2 hidden layer MLP with size 48, LayerNorm and residual connections for  $\mu_{\phi}$ .
Sparse initialization for BayesDAG For BayesDAG, we additionally allow sparse initialization by sampling a sparse  $\mathbf{W}$  from the  $\mu_{\phi}$ . This can be achieved by subtracting a constant 1 from the existing logits (i.e. the output from  $\mu_{\phi}$ ).
Other hyperparameters For BayesDAG, we run 10 parallel SG-MCMC chains for  $p$  and  $\Theta$ . We implement an adaptive sinkhorn iteration where the iteration automatically stops when the sum of rows and columns are closed to 1 within the threshold 0.001 (upto a maximum of 3000 iterations). Typically, we found this to require only around 300 iterations. We set the sinkhorn temperature  $t$  to be 0.2. For the reparametrization of  $W$  matrix with Gumbel-softmax trick, we use temperature 0.2. During evaluation, we use 100 SG-MCMC particles extracted from the particle buffer. We use 0.0003 for SG-MCMC learning rate  $l$  and batch size 512. We run 700 epochs to make sure the model is fully converged.
Table 5: Walltime results (in minutes, rounded to the nearest minute) of the runtime of different approaches on a single 40GB A100 NVIDIA GPU. The N/A fields indicate that the corresponding method cannot be run within the memory constraints of a single GPU.
d=30	d=50	d=70	d=100
BaDAG (Ours)(Bayesian, Nonlinear)	171	238	261	448
DIBS (Bayesian, Nonlinear)	187	350	N/A	N/A
BGES (Quasi-Bayesian, Linear)	2	3	6	11
BCD (Bayesian, Linear)	252	328	418	600
DDS (Quasi-Bayesian, Nonlinear)	92	130	174	N/A
Figure 6: Posterior inference of both graph and functional parameters on synthetic datasets of nonlinear causal models with  $d = 20$  variables. BayesDAG gives best results across all metrics.  $\downarrow$  denotes lower is better and  $\uparrow$  denotes higher is better. For the sake of clarity, DDS has been omitted for E-SHD due to its significantly inferior performance on this metric.
For DIBS, we can only use 20 SVGD particles for evaluation due to the quadratic scaling with the number of particles. We use 0.1 for Gumbel-softmax temperature. We run 10000 epochs for convergence. The learning rate is selected as 0.01.
Table 3 shows the hyperparameter selection for BayesDAG. Table 4 shows the hyperparameter selection for DIBS.
E Additional Results
E.1 Walltime Comparison
Table 5 presents walltime comparison of different methods. Our method converges faster while being scalable w.r.t. DIBS, the nonlinear Bayesian causal discovery baseline. Other methods like BGES and DDS, while faster, perform much worse in terms of uncertainty quantification. In addition BGES is limited to linear model and DDS is not a fully Bayesian method.
E.2 Performance with higher dimensional datasets
Full results for all the metrics for settings  $d = 20$ ,  $d = 70$  and  $d = 100$  for nonlinear settings are presented in Figure 6, Figure 7 and Figure 8.
Figure 7: Posterior inference of both graph and functional parameters on synthetic datasets of nonlinear causal models with  $d = 70$  variables. BayesDAG gives best results across most metrics.  $\downarrow$  denotes lower is better and  $\uparrow$  denotes higher is better. As DIBS and DDS are computationally prohibitive to run for this setting, it has been omitted. BCD has been omitted for NLL as we observed that it performs significantly worse.
Figure 8: Posterior inference of both graph and functional parameters on synthetic datasets of nonlinear causal models with  $d = 100$  variables. BayesDAG gives best results across E-SHD, comparable across NLL but slightly worse for Edge F1.  $\downarrow$  denotes lower is better and  $\uparrow$  denotes higher is better. As DIBS and DDS are computationally prohibitive to run for this setting, it has been omitted. BCD has been omitted for NLL as we observed that it performs significantly worse.
Figure 9: Performance comparison of SG-MCMC+VI v.s. fully SG-MCMC with  $\tilde{W}$  for  $d = 10$  variables.
E.3 Performance of SG-MCMC with Continuous Relaxation
We compare the performance of SG-MCMC+VI and SG-MCMC with  $\tilde{W}$  on  $d = 10$  ER and SF graph settings. Figure 9 shows the performance comparison. We can observe that SG-MCMC+VI generally outperforms its counterpart in most of the metrics. We hypothesize that this is because VI network  $\mu_{\phi}$  couples  $\pmb{p}$  and  $\pmb{W}$ . This coupling effect is crucial since the changes in  $\pmb{p}$  results in the change of permutation matrix, where the  $\pmb{W}$  can immediately respond to this change through  $\mu_{\phi}$ . On the other hand,  $\tilde{W}$  can only respond to this change through running SG-MCMC steps on  $\tilde{W}$  with fixed  $\pmb{p}$ . In theory, this is the most flexible approach since this coupling do not requires parametric form like  $\mu_{\phi}$ . However in practice, we cannot run many SG-MCMC steps with fixed  $\pmb{p}$  for convergence, which results in the inferior performance.
F Code and License
For the baselines, we use the code from the following repositories:
- BGES: We use the code from [2] from the repository https://github.com/agrawalraj/active_learning (No license included).
DDS: We use the code from the official repository https://github.com/sharpenb/Differentiable-DAG-Sampling (No license included).
- BCD: We use the code from the official repository https://github.com/ermongroup/BCD-Nets (No license included).
- DIBS: We use the code from the official repository https://github.com/larslorch/dibs (MIT license).
Additionally for the Syntren [67] and Sachs Protein Cells [59] datasets, we use the data provided with repository https://github.com/kurowasan/GraN-DAG (MIT license).
G Broader Impact Statement
This work is concerned with understanding cause and effects from data, which has potential applications in empirical sciences, economics and machine perception. Understanding causal relationships
can improve fairness in decision making, understand biases which might be present in the data and answering causal queries. As such, we envision this line of work to not have any significant negative impact.