SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition 1. Given any  $d$  dimensional ODE system with  $\mathcal{G}^*$  identifiable under  $f\in \mathcal{F}$ , the  $D = d + a$  dimensional system  $\frac{dx}{dt} = Ax$ , denote the vector of multiplicities  $m\in \mathbb{N}^d$  with  $m_{i}$  as the number of repetitions of each variable. Then this construction creates an admissible family of graphs  $\mathcal{G}'$  where  $|\mathcal{G}'| = \prod_{i\in d}(2^{m_i} - 1)^{\mathcal{G}_i\mathbf{1}}$ . Furthermore, under an  $L^0$  penalty on  $G$ , this reduces to  $\prod_{i}(m_{i})^{\mathcal{G}_{i}\mathbf{1}}$ .
See Appendix A for full proof. The intuition behind this proposition can be seen from the case of adding a single copied variable. This corresponds to  $\mathbf{A} = [\delta_v\mathbf{I}_d]$  where  $\delta_v$  is a vector with a 1 on
2We discuss training dynamics when using  $h_\phi$  in Appendix B.7.
Figure 2: Visualization of modeling cyclic dependencies over time (left). To create a family of admissible graphs (right), we add a new variable  $v_3'$  which has the same values as  $v_3$  and creates three possible explanations for the data (green arrows). If we apply a sparsity penalty, then we can eliminate the last possibility (which has two additional edges) for only two possible graphs.  $G^*$  denotes the ground truth graph while  $G'$  denotes the admissible family of graphs induced by  $v_3'$ .
node  $v$  and zeros elsewhere, and  $I_d$  is the  $d$ -dimensional identity matrix. Let  $v$  have  $c$  children, such that  $v \in Pa(c)$  in the identifiable system, then any of those  $c$  child nodes could depend either on  $v$  or on the new node  $v'$  or both. This creates  $3^c$  possible explanatory graphs. If we restrict ourselves to the set of graphs with minimal  $L^0$  norm, then we eliminate the possibility of a child node depending on both  $v$  and  $v'$ , this gives  $2^c$  possible graphs, choosing either  $v$  or  $v'$  as a parent.
In this section we evaluate the performance of DynGFN against counterpart Bayesian structure learning methods (see Appendix B.2 for details). Since our primary objective is to learn Bayesian posteriors over discrete structure  $G$ , we compare to Bayesian methods that can also accomplish this task, i.e. versions of BCD-Nets [11] and DiBS [32]. We show in certain cases, DynGFN is able to better capture the true posterior when there are a large number of modes. We evaluate methods according to four metrics: Bayes-SHD, area under the receiver operator characteristic curve (AUC), Kullback-Leibler (KL) divergence between learned posteriors  $Q(G)$  and the distribution over true graphs  $P(G^{*})$ , and the negative log-likelihood (NLL)  $P(D|G,\theta)$  (in our setting this reduces to the mean squared error between  $\widehat{dx}$  and  $dx$ , given  $\theta$  and sampled  $G^{\prime}s$ ). Since the analytic linear solver requires data at run-time to compute optimal parameters for the structural model, we include the NLL metric only for models using the HyperNetwork solver. Bayes-SHD measures the average distance to the closest structure in the admissible set of graphs according to the structural hamming distance, which in this case is simply the hamming distance of the adjacency matrix representation to the closest admissible graph. We assume  $P(G^{*})$  is uniform over  $G^{*}$  and include further details about evaluating the quality of learned posteriors in Appendix B.8.
6.1 A Toy Example with Synthetic Data
To validate the ability of DynGFN to learn cyclic dependencies we consider identifiable acyclic and cyclic 3 variable toy systems and provide a comparison with a DAG structure learning method (NOTEARS [54]). We show results for this toy example in Figure 3. NOTEARS does not model cyclic dependencies and therefore struggles to yield accurate predictions of  $A$  in the cyclic setting. We can also verify this result by considering a conditional independence test over cyclic dependencies. It is easy to see that the conditional independence test fails in the cyclic setting: in the acyclic case, we can identify the v-structure by observing that  $x_{1} \perp x_{3}$  and  $x_{1} \not\perp x_{3}|x_{2}$ , which implies that  $x_{2}$  is a collider (i.e.  $x_{1}$  and  $x_{3}$  are marginally independent and conditionally dependent); while in the cyclic example, we introduce time dependencies such that there are cycles in the summary graph that render these variables marginally dependent. We show that DynGFN is able to identify the true cyclic dependencies in this toy example. We note that in this toy example DynGFN exhibits a degree
Figure 3: Toy example for learning DAG structure (top) and cyclic graph structure (bottom) from observational data. In this example we compare NOTEARS and DynGFN for learning acyclic and cyclic dependencies from observational data. Data is generated from a linear system  $dx = Ax$  with a corresponding acyclic and cyclic  $A$  as defined in the figure. We use 500 observational samples for each system. NOTEARS is implemented using the CausalNex [5] library.
Table 1: Bayesian dynamic structure learning of linear and non-linear systems with  $d = 20$  variables. The graphs representing the structural dynamic relationships of the linear and non-linear systems have 50 edges out of possible 400. The ground truth discrete distribution  $P(G^{*})$  contains 1024 admissible graphs for each respective system. The  $\ell$  and  $h$  pre-fix denote usage of the analytic linear solver and HyperNetwork solver for structural model parameters, respectively. Results are reported on held out test data over 5 model seeds.
Model	Bayes-SHD ↓	Linear System
AUC ↑	KL ↓	NLL ↓
\( \ell-DynBCD \)	32.0 ± 0.27	0.71 ± 0.0	1707.45 ± 9.66	—
\( \ell-DynDiBS \)	29.2 ± 0.78	0.71 ± 0.0	6622.43 ± 171.67	—
\( \ell-DynGFN \)	22.8 ± 1.4	0.75 ± 0.01	1091.60 ± 35.72	—
\( h-DynBCD \)	5.5 ± 1.1	0.89 ± 0.04	701.19 ± 46.99	(9.83 ± 0.59)E - 5
\( h-DynDiBS \)	28.5 ± 4.2	0.51 ± 0.07	7934.90 ± 381.80	(8.17 ± 1.30)E - 6
\( h-DynGFN \)	6.7 ± 0.0	0.94 ± 0.0	350.92 ± 30.15	(8.35 ± 0.02)E - 3
Model	Bayes-SHD ↓	Non-linear System	NLL ↓
AUC ↑	KL ↓
l-DynBCD	77.5 ± 8.3	0.42 ± 0.03	3814.86 ± 354.56	—
l-DynDiBS	75.7 ± 7.7	0.59 ± 0.01	5893.65 ± 59.66	—
l-DynGFN	45.7 ± 0.6	0.55 ± 0.0	226.25 ± 6.58	—
h-DynBCD	192.9 ± 0.7	0.50 ± 0.0	9108.69 ± 51.34	(3.83 ± 0.32)E - 4
h-DynDiBS	48.1 ± 9.0	0.53 ± 0.10	8716.64 ± 265.29	(4.06 ± 0.10)E - 6
h-DynGFN	32.6 ± 0.9	0.67 ± 0.01	193.28 ± 8.53	(1.47 ± 0.11)E - 3
of convergence sensitivity on a per-run basis. In the following sections we provide comprehensive results over 5 random seeds and consider larger systems.
6.2 Experiments with Synthetic Data
We generated synthetic data from two systems using our indeterminacy model presented in section 5: (1) a linear dynamical system  $dx = Ax$ , and (2) a non-linear dynamical system  $dx = \text{sigmoid}(Ax)$ . We consider  $\ell$ -DynGFN and  $h$ -DynGFN, i.e. DynGFN with the linear analytic parameter solver as shown in (5), and DynGFN with the HyperNetwork parameter solver  $h_\phi$ . Likewise, we compare  $\ell$ -DynGFN and  $h$ -DynGFN to counterpart Bayesian baselines which we call  $\ell$ -DynBCD,  $\ell$ -DynDiBS,  $h$ -DynBCD, and  $h$ -DynDiBS. To constrain the discrete search procedure, we assume a sparse prior on the structure (i.e. the graphs  $G$ ), using the  $L^0$  prior. Due to challenging iterative optimization dynamics present when using  $\theta = h_\phi(G)$  for DynGFN, to train initialize the forward policy  $P_F(s_i | s_{i-1}; \psi)$  using the  $\psi$  learned in  $\ell$ -DynGFN to provide a more admissible starting point for learning  $h_\phi$  (we
discuss further details in Appendix B.7). We do not need to do this for  $h$ -DynBCD and  $h$ -DynDiBS as we are able to train both models end-to-end without iterative optimization. In Table 1 we show results of our synthetic experiments for learning posteriors over multi-modal distributions of cyclic graphs. We observe the DynGFN is most competitive on both synthetic systems for modelling the true posterior over structure. Details about DynGFN, baselines, and accompanying hyper-parameters can be found in Appendix B.
6.3 Ablations Over Sparsity and Linearity of Dynamic Systems
We conduct two ablations: (1) ablation over sparsity of the dynamic system structure, and (2) ablation over  $\Delta t$ , the time difference between data points of dynamic simulation. For a sparsity level of 0.9, the ground truth graphs have 50 edges out of  $d^2$  possible edges. In these experiments,  $P(G^{*})$  for  $d = 20$  and sparsity 0.9 has 1024 modes. We conduct the ablations over 5 random seeds for each set of experiments.
Sparsity: DynGFN uses the  $L^0$  prior on  $G$  throughout training. Under this setting, system sparsity carries significant weight on the ability to learn posteriors over the structured dynamics of a system. We show this trend in Table 2. We note that computing the KL-divergence for DynGFN, specifically computing the probability of generating a true  $G$ , becomes computationally intractable as  $G$  is less sparse<sup>3</sup>. For systems of 0.9 and 0.95 sparsity, we observe a decreasing trend in KL and Bayes-SHD, and an increasing trend in AUC. This result is expected as DynGFN can better traverse sparse graphs as the combinatorial space over possible trajectories is smaller relative to denser systems.
Linearity: Training DynGFN via the linear solver for the structural model parameters is an easier objective due to simplified
training dynamics. Because of this, we explore the performance of  $\ell$ -DynGFN assuming  $f_{\theta}$  for modelling equation (1) to be linear in the non-linear system. We do this by conducting an ablation over  $\Delta t$  and find that the performance of  $\ell$ -DynGFN on the non-linear system improves as  $\Delta t \to 0$ . We show a portion of this trend in Table 3.
Table 2: Ablation for  $\ell$  -DynGFN on  $d = {20}$  systems with varying levels of sparsity and fixed  ${\Delta t} = {0.05}$  .
Sparsity	Bayes-SHD ↓	AUC ↑	KL ↓
0.95	16.4 ± 1.71	0.79 ± 0.0	889.57 ± 31.24
0.90	22.8 ± 1.41	0.75 ± 0.01	1091.60 ± 35.72
0.85	32.8 ± 0.72	0.71 ± 0.0	—
0.80	39.2 ± 0.69	0.71 ± 0.0	—
0.75	60.2 ± 1.17	0.66 ± 0.01	—
Table 3: Ablation for  $\ell$  -DynGFN on  $d = {20}$  non-linear systems with varying  ${\Delta t}$  and fixed sparsity at 0.9 .
Δt	Bayes-SHD ↓	AUC ↑	KL ↓
0.001	38.7 ± 0.80	0.61 ± 0.0	202.41 ± 9.95
0.005	39.0 ± 0.81	0.60 ± 0.0	206.83 ± 11.55
0.01	40.6 ± 1.13	0.59 ± 0.0	202.71 ± 7.74
0.05	45.7 ± 0.62	0.55 ± 0.0	226.25 ± 6.58
0.1	51.8 ± 0.18	0.50 ± 0.0	264.86 ± 2.17
6.4 Experiments on Single-Cell RNA-velocity Data
To show how DynGFN can be applied to single cell data we use a cell cycle dataset of human Fibroblasts [46]. As a motivating example we show the correlation structure of single-cell RNA-seq data from human Fibroblast cells [46] Figure 4. We show both the raw correlation and the correlation over cell cycle time, which is significantly higher. With such a pure cell population whose primary axis of variation is state in the cell cycle by aggregating over cell cycle time we expect observation noise to be averaged out, leading to a "truer" view of the correlation between latent variables. Further details for this experimental set-up are provided in Appendix C.1.3. Since there are many genes which are affected by the cell cycle phase, there are many correlated variables that are downstream of the true cell cycle regulators. This provides a natural way of using cell cycle data to evaluate a model's ability to capture the Bayesian posterior. In Table 4 we show results for learning posteriors over an undetermined GRN using RNA velocity data. We find that  $\ell$ -DynGFN and  $h$ -DynGFN yield low KL
For example, since DynGFN constructs one object  $G$  sequentially over a state space distribution, we must compute probabilities of all combinatorial state trajectories for constructing  $G = (s_i, \dots, s_n)$ . The space of combinatorial state trajectories is  $n!$  in nature, hence this computation is only possible for small graphs and/or sparse graphs.
(b)
Figure 4: (a) Correlation structure in the raw single cell data over 5000 cells and 2000 genes selected by scVelo [8] pre-processing. (b) Correlation structure among genes over (inferred) cell cycle times. This stronger correlation structure is more reflective of the correlation in the underlying system. (c) Histogram of pairwise Pearson correlations between all genes passing pre-processing, comparing the absolute values of the elements in (a) and (b). (d) Shows the ground truth GRN extracted as a subset of the KEGG cell cycle pathway. Cdc25A is known to inhibit Cdk1 which is known to inhibit Cdc25C, while the Mcm complex is highly correlated with Cdc25A, they do not directly interact with Cdk1 [24].
Table 4: Bayesian dynamic structure learning 5-D cellular system using scRNA velocity data. The dynamics of this system are unknowns, however we identify 81 admissible graphs between variables (genes) that describe the data. We train models over 5 seeds. The graphs of this system contain of 7 true edges.
Model	Cellular System - RNA Velocity
Bayes-SHD ↓	AUC ↑	KL ↓	NLL ↓
\( \ell-DynBCD \)	2.6 ± 0.1	0.56 ± 0.01	321.95 ± 3.34	—
\( \ell-DynDiBS \)	6.5 ± 0.4	0.47 ± 0.01	550.17 ± 16.63	—
\( \ell-DynGFN \)	3.3 ± 0.4	0.59 ± 0.03	44.98 ± 18.60	—
h-DynBCD	10.1 ± 0.8	0.53 ± 0.03	587.41 ± 24.00	0.094 ± 0.003
h-DynDiBS	9.6 ± 4.2	0.51 ± 0.13	560.85 ± 83.83	0.084 ± 0.0
h-DynGFN	5.1 ± 1.2	0.58 ± 0.05	39.82 ± 28.05	0.109 ± 0.001
and moderate Bayes-SHD. While  $\ell$ -DynBCD performs well in terms of identify a small distribution of true  $G$ 's, it falls short in modelling the true posterior (this can be seen from low Bayes-SHD, high KL).
We presented DynGFN, a method for Bayesian dynamic structure learning. In low dimensions we found that DynGFN is able to better model the distribution over possible explanatory structures than counterpart Bayesian structure learning baseline methods. As a proof of concept, we presented an example of learning the distribution over likely explanatory graphs for linear and non-linear synthetic systems where complex uncertainty over explanatory structure is prevalent. We demonstrate the use of DynGFN for learning gene regulatory structure from single-cell transcriptomic data where there are many possible graphs, showing DynGFN can better model the uncertainty over possible explanations of this data rather than capturing a single explanation.
Limitations and Future Work: We have demonstrated a degree of efficacy when using DynGFN for Bayesian structure learning with dynamic observational data. A key limitation of DynGFN is scaling to larger systems. To effectively model  $P(G, \theta, D)$ , DynGFN needs to search over an environment state space of possible graphs. This state space grows exponentially with the number of possible edges, i.e.  $2^{d^2}$  or  $d2^d$  for per-node-GFN where  $d$  is the number of variables in the system. Therefore, DynGFN is currently limited to smaller systems. Nevertheless, there are many applications where Bayesian structure learning, even over 5-20 dimensional examples that we explore here, could be extraordinarily useful. We include further discussion of scaling DynGFN in Appendix C.3 with some ideas on how to approach this challenge. We found that training DynGFN requires some selection of hyper-parameters and in particular parameters that shape the reward function. Selecting hyper-parameters for the baseline methods prove more difficult for this task.
This research was enabled in part by the computational researches provided by Mila (mila.quebec) and Compute Canada (ccdb.computecanada.ca). In addition, resources used in preparing this research were in part provided by the Province of Ontario and companies sponsoring the Vector Institute (vectorinstitute.ai/partners/). All authors are funded by their primary academic institution. We also acknowledged funding from the Natural Sciences and Engineering Research Council of Canada, Recursion Pharmaceuticals, CIFAR, Samsung, and IBM. We are grateful to Cristian Dragos Manta for catching some typos in the manuscript.
[1] Aliee, H., Theis, F. J., and Kilbertus, N. Beyond predictions in neural odes: Identification and interventions. arXiv preprint 2106.12430, 2021.
[2] Aliee, H., Richter, T., Solonin, M., Ibarra, I., Theis, F., and Kilbertus, N. Sparsity in continuous-depth neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[3] Améndola, C., Dettling, P., Drton, M., Onori, F., and Wu, J. Structure learning for cyclic linear causal models. Uncertainty in Artificial Intelligence (UAI), 2020.
[4] Annadani, Y., Rothfuss, J., Lacoste, A., Scherrer, N., Goyal, A., Bengio, Y., and Bauer, S. Variational causal networks: Approximate bayesian inference over causal structures. arXiv preprint, 2021.
[5] Beaumont, P., Horsburgh, B., Pilgerstorfer, P., Droth, A., Oentaryo, R., Ler, S., Nguyen, H., Ferreira, G. A., Patel, Z., and Leong, W. CausalNex, 2021. URL https://github.com/quantumblacklabs/causalnex.
[6] Bellot, A. and Branson, K. Neural Graphical Modelling in Continuous Time: Consistency Guarantees and Algorithms. International Conference on Learning Representations (ICLR), 2022.
[7] Bengio, E., Jain, M., Korablyov, M., Precup, D., and Bengio, Y. Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[8] Bengio, Y., Deleu, T., Hu, E. J., Lahlou, S., Tiwari, M., and Bengio, E. GFlowNet Foundations. arXiv preprint 2111.09266, 2022.
[9] Bergen, V., Lange, M., Peidli, S., Wolf, F. A., and Theis, F. J. Generalizing RNA velocity to transient cell states through dynamical modeling. BioRxiv preprint 820936, 2019.
[10] Chen, R. T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. Neural Ordinary Differential Equations. Advances in Neural Information Processing Systems (NeurIPS), 2018.
[11] Cundy, C., Grover, A., and Ermon, S. BCD Nets: Scalable Variational Approaches for Bayesian Causal Discovery. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[12] Deleu, T., Góis, A., Emezue, C., Rankawat, M., Lacoste-Julien, S., Bauer, S., and Bengio, Y. Bayesian Structure Learning with Generative Flow Networks. Uncertainty in Artificial Intelligence (UAI), 2022.
[13] Friedman, N., Murphy, K., and Russell, S. Learning the structure of dynamic probabilistic networks. Uncertainty in Artificial Intelligence (UAI), 1998.
[14]Glymour, C., Zhang, K., and Spirtes, P. Review of Causal Discovery Methods Based on Graphical Models. Frontiers in Genetics, 2019.
[15] Granger, C. W. Investigating causal relations by econometric models and cross-spectral methods. *Econometrica: journal of the Econometric Society*, 1969.
[16] Ha, D., Dai, A. M., and Le, Q. V. Hypernetworks. International Conference on Learning Representations (ICLR), 2017.
Proposition 1 calculates the number of admissible structure graphs for a linear ODE system with correlated variables. We will first show the general case this is  $\prod_{i\in d}(2^{m_i} - 1)^{\mathcal{G}_i\mathbf{1}}$ , then analyze the case of an  $L^0$  penalty on the edges of  $G$ , which reduces the size of the set of admissible graphs to  $\prod_{i}(m_{i})^{\mathcal{G}_{i}\mathbf{1}}$ .
Proof. Consider an identifiable linear system  $\frac{dx}{dt} = Ax$  where we directly observe  $(x, \frac{dx}{dt})$  with  $\mathcal{G}^*$  identifiable. Then the system with  $m = 1^d$  has exactly one admissible graph by definition. For each node, we analyze its set of child nodes in  $\mathcal{G}$ , i.e.  $c(u) = \{v \in \mathcal{V} \text{ s.t. } u \to v \in \mathcal{G}\}$ . For an identifiable system, each child  $v$  must have an incoming edge from its parent.
Next, we consider the process of adding a correlated variable, i.e. consider the situation of w.l.o.g. consider  $m = (s, 1, 1, \dots, 1)$  for some  $s > 1$ . Then for each child of  $c_{j}(v_{1})$ , there are now  $s$  possible parents. This has multiplied the number of possible graphs by  $2^{s} - 1$ . Since each element of  $m$  is independent, this leads to the first statement, i.e.  $|\mathcal{G}'| = \prod_{i \in d} (2^{m_i} - 1)^{\mathcal{G}_i\mathbf{1}}$ .
Under an  $L^0$  penalty, then we constrain the possible graphs to  $s$  different graphs, where each child node picks exactly one of the  $s$  possible parents. This leads to the second statement,  $|\mathcal{G}^{\prime}| = \prod_{i}(m_{i})^{\mathcal{G}_{i}\mathbf{1}}$
B.1 Single Cell Dataset Preprocessing
We start with the processed data from [46]. We first filter it applying steps from the ScVelo tutorial. We then sub-select the genes of interest and use the "Ms" and "velocity" layers, which we normalize to mean zero standard deviation one for the states and scale the  $dx$  with the same parameters.
B.2 Baselines for Bayesian Dynamic Structure Learning
Existing Bayesian structure learning methods are typically constrained to learning DAGs. Temporal information about the dynamic relationships amongst variables in a system can help alleviate this constraint. DiBS and BCD-Nets are two state-of-the-art Bayesian structure learning approaches for static systems. We apply versions of DiBS and BCD-Nets such that they are applicable in our Bayesian dynamic structure learning framework for learning cyclic graph structure from dynamic data. We use the approach taken in DiBS and parameterize the distribution over graphs as  $P_{\alpha_t}(G|Z) = \prod_i \prod_j P_{\alpha_t}(G_{ij}|Z_{ij})$ , where  $Z = U^T V$ ,  $U, V \in \mathbb{R}^{k \times d}$ . Here  $P_{\alpha_t}(G_{ij} = 1|Z_{ij}) = \sigma(\alpha_t Z_{ij})$ ,  $\sigma(x) = 1/(1 + e^{-\alpha_tx})$ , and  $\alpha_t = \alpha c(t)$  ( $t$  denotes the training iteration. We use  $c(t) = \sqrt{t}$ ). As  $t \to \infty$ ,  $P_{\alpha_t}(G|Z) \to \delta(G|Z)$ . In DiBS, Stein variational gradient descent (SVGD) [30] is used to iteratively transport particles  $Z$  to learn the target distribution. Following from the above parameterization, we implement a version of BCD-Nets by treating  $U \sim \mathcal{N}(\mu_{\mathbf{u}}, \sigma_{\mathbf{u}}^2)$ ,  $V \sim \mathcal{N}(\mu_{\mathbf{v}}, \sigma_{\mathbf{v}}^2)$ , and learning  $\mu_{\mathbf{u}}, \mu_{\mathbf{v}}, \sigma_{\mathbf{u}}$ , and  $\sigma_{\mathbf{v}}$ . Since our framework uses dynamic data, we incorporate DiBS and BCD-Nets within the framework (labeled DynDiBS and DynBCD, respectively) to leverage dynamic information for Bayesian structure learning of cyclic graphs.
B.3 Hyper-parameters for Baselines
For both DynBCD and DynDiBS we use  $k = d$  across datasets. Since DynDiBS is an ensemble based method, we use 1024 samples for the linear and non-linear synthetic systems and 1000 samples for the cellular system (both training and evaluation). Since DynBCD is a variational approach and doesn't require parallelized model ensembles, we use a large quantity of samples for training and evaluation. In the case of the cellular system, since there is a significantly smaller quantity of admissible graphs, we use less samples for DynBCD. We use graph sparsity regularization denoted by  $\lambda_0$  and a temperature parameter  $T$  that scales the magnitude of the likelihood (e.g.  $\frac{1}{T^2} \mathrm{MSE}(dx, \widehat{dx})$ ).
In Table 5 and Table 6 we outline the hyper-parameters we found to yield the most competitive results. We use grid search to tune DynBCD and DynDiBS. All baselines are trained for 1000 epochs.
Table 5: Hyper-parameters for DynBCD. We define learning rate as  $\epsilon$  .
Model	Linear System
ε	λ0	T	α	samples
l-DynBCD	0.0001	0.001	0.01	0.1	5000
h-DynBCD	0.0001	0.0025	0.01	2	2000
Model	Non-linear System
ε	λ0	T	α	samples
ℓ-DynBCD	0.00005	0.01	0.01	2	5000
h-DynBCD	0.0001	0.001	0.01	1	2000
Model	Cellular System
ε	λ0	T	α	samples
l-DynBCD	0.0001	0.001	0.05	0.05	1000
h-DynBCD	0.00001	0.0005	0.1	2	1000
Table 6: Hyper-parameters for DynDiBS. We define learning rate as  $\epsilon$  .
Linear System
Model	ε	λ0	T	α	γ
ℓ-DynDiBS	0.0025	500	0.01	0.0001	3000
h-DynDiBS	0.0001	3	0.01	0.0001	10000
Non-linear System
Model	ε	λ0	T	α	γ
ℓ-DynDiBS	0.001	10	0.01	0.0001	3000
h-DynDiBS	0.0001	0.1	0.01	0.0001	10000
Cellular System
Model	ε	λ0	T	α	γ
ℓ-DynDiBS	0.0025	1	0.05	0.0001	3000
h-DynDiBS	0.00001	0.1	0.01	0.01	3000