SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
R (G, \theta) P _ {F} \left(G ^ {\prime} \mid G\right) P _ {F} \left(\theta^ {\prime} \mid G ^ {\prime}\right) = R \left(G ^ {\prime}, \theta^ {\prime}\right) P _ {B} \left(G \mid G ^ {\prime}\right) P _ {F} (\theta \mid G), \tag {C.7}
$$
which is the subtrajectory balance condition in (6).
C.3.2 Integrating undirected paths of length 2
Similar to Appendix C.3.1, we consider here an undirected of length 2 of the form  $(G,\theta)\gets (G,\cdot)\to$ $(G,\tilde{\theta})$  (see Figure C.2). Since  $(G,\cdot)$  is a common ancestor (a common parent in this case) of both complete states  $(G,\theta)$  and  $(G,\tilde{\theta})$ , we can write the subtrajectory balance conditions (C.4) as
$$
F (G, \theta) P _ {B} (G \mid \theta) P _ {F} (\tilde {\theta} \mid G) = F (G, \tilde {\theta}) P _ {B} (G \mid \tilde {\theta}) P _ {F} (\theta \mid G). \tag {C.8}
$$
Using the same simplifications as in Appendix C.3.1 ( $P_B(G \mid \theta) = P_B(G \mid \tilde{\theta}) = 1$ ), we get the following subtrajectory balance conditions for the undirected paths of length 2
$$
R (G, \theta) P _ {F} (\tilde {\theta} \mid G) = R (G, \tilde {\theta}) P _ {F} (\theta \mid G). \tag {C.9}
$$
Note that these conditions are effectively redundant if the SubTB conditions over undirected paths of length 3 (6) are satisfied for all possible pairs of complete states  $(G, \theta)$  and  $(G', \theta')$ . Indeed, if we write these conditions between  $(G, \theta)$  and  $(G', \theta')$  on the one hand, and between  $(G, \tilde{\theta})$  and  $(G', \theta')$  on the other hand (with a fixed  $G'$  and  $\theta'$ )
$$
R \left(G ^ {\prime}, \theta^ {\prime}\right) P _ {B} (G \mid G ^ {\prime}) P _ {F} (\theta \mid G) = R (G, \theta) P _ {F} \left(G ^ {\prime} \mid G\right) P _ {F} \left(\theta^ {\prime} \mid G\right) \tag {C.10}
$$
$$
R \left(G ^ {\prime}, \theta^ {\prime}\right) P _ {B} (G \mid G ^ {\prime}) P _ {F} (\tilde {\theta} \mid G) = R (G, \tilde {\theta}) P _ {F} \left(G ^ {\prime} \mid G\right) P _ {F} \left(\theta^ {\prime} \mid G\right), \tag {C.11}
$$
we get the same subtrajectory balance conditions over undirected paths of length 2 as in (C.9):
$$
\frac {R (G , \theta)}{P _ {F} (\theta \mid G)} = \frac {R \left(G ^ {\prime} , \theta^ {\prime}\right) P _ {B} \left(G \mid G ^ {\prime}\right)}{P _ {F} \left(G ^ {\prime} \mid G\right) P _ {F} \left(\theta^ {\prime} \mid G ^ {\prime}\right)} = \frac {R (G , \tilde {\theta})}{P _ {F} (\tilde {\theta} \mid G)}. \tag {C.12}
$$
However, since the SubTB conditions (6) are only satisfied approximately in practice, it might be advantageous to also satisfy (C.9) in addition to those in (6). The equation above provides an alternative way to express (C.9). Indeed, (C.12) shows that the function
$$
f _ {G} (\theta) \triangleq \log R (G, \theta) - \log P _ {F} (\theta \mid G) \tag {C.13}
$$
is constant, albeit with a constant that depends on the graph  $G$ . Since this function is differentiable, this is equivalent to  $\nabla_{\theta}f_{G}(\theta) = 0$ , and therefore we get the differential form of the subtrajectory balance conditions in (7)
$$
\nabla_ {\theta} \log P _ {F} (\theta \mid G) = \nabla_ {\theta} \log R (G, \theta). \tag {C.14}
$$
As we saw in Section 3.3, one way to enforce the SubTB conditions over undirected paths of length 3 is to create a learning objective that encourages these conditions to be satisfied, and optimizing it
using gradient methods. The learning objective has the form  $\mathcal{L}(\phi) = \mathbb{E}_{\pi}[\tilde{\Delta}^2 (\phi)]$ , where  $\tilde{\Delta} (\phi)$  is a non-linear residual term
$$
\tilde {\Delta} (\phi) = \log \frac {R \left(G ^ {\prime} , \theta^ {\prime}\right) P _ {B} \left(G \mid G ^ {\prime}\right) P _ {\phi} \left(\theta \mid G\right)}{R \left(G , \theta\right) P _ {\phi} \left(G ^ {\prime} \mid G\right) P _ {\phi} \left(\theta^ {\prime} \mid G ^ {\prime}\right)}. \tag {C.15}
$$
Suppose that the parameters  $\phi$  of the GFlowNet are such that the subtrajectory balance conditions in (C.14) are satisfied for any  $(G,\theta)$ . Although this assumption is unlikely to be satisfied in practice, they will eventually be approximately satisfied over the course of optimization, given the discussion above about the relation between (C.9) and (C.7). Since  $\theta$  and  $\theta^{\prime}$  depend on  $\phi$  (via the reparametrization trick since they are sampled on-policy, see Section 5), taking the derivative of  $\tilde{\Delta}^{2}(\phi)$ , we get
$$
\begin{array}{l} \frac {d}{d \phi} \tilde {\Delta} ^ {2} (\phi) = \tilde {\Delta} (\phi) \cdot \frac {d}{d \phi} \left[ \log R \left(G ^ {\prime}, \theta^ {\prime}\right) + \log P _ {\phi} (\theta \mid G) \right. \tag {C.16} \\ \left. - \log R (G, \theta) - \log P _ {\phi} \left(G ^ {\prime} \mid G\right) - \log P _ {\phi} \left(\theta^ {\prime} \mid G ^ {\prime}\right) \right]. \\ \end{array}
$$
Using the law of total derivatives, we have
$$
\begin{array}{l} \frac {d}{d \phi} [ \log P _ {\phi} (\theta \mid G) - \log R (G, \theta) ] = \underbrace {\left[ \frac {\partial}{\partial \theta} \log P _ {\phi} (\theta \mid G) - \frac {\partial}{\partial \theta} \log R (G , \theta) \right]} _ {= 0} \frac {d \theta}{d \phi} + \frac {\partial}{\partial \phi} \log P _ {\phi} (\theta \mid G) (C.17) \\ = \frac {\partial}{\partial \phi} \log P _ {\phi} (\theta \mid G), (C.18) \\ \end{array}
$$
and similarly for the terms in  $(G^{\prime},\theta^{\prime})$ . The derivative of the objective then becomes
$$
\frac {d}{d \phi} \tilde {\Delta} ^ {2} (\phi) = \tilde {\Delta} (\phi) \left[ \frac {\partial}{\partial \phi} \log P _ {\phi} (\theta \mid G) - \frac {\partial}{\partial \phi} \log P _ {\phi} \left(\theta^ {\prime} \mid G ^ {\prime}\right) - \frac {d}{d \phi} \log P _ {\phi} \left(G ^ {\prime} \mid G\right) \right]. \tag {C.19}
$$
An alternative way to obtain the same derivative in (C.17) is to take  $d\theta / d\phi = 0$  instead, meaning that we would not differentiate through  $\theta$  (and  $\theta'$ ). Using the stop-gradient operation  $\perp$ , this shows that the following objective
$$
\mathcal {L} (\phi) \triangleq \mathbb {E} _ {\pi} \left[ \Delta (\phi) ^ {2} \right] = \mathbb {E} _ {\pi} \left[ \left(\log \frac {R \left(G ^ {\prime} , \perp \left(\theta^ {\prime}\right)\right) P _ {B} \left(G \mid G ^ {\prime}\right) P _ {\phi} \left(\perp (\theta) \mid G\right)}{R \left(G , \perp (\theta)\right) P _ {\phi} \left(G ^ {\prime} \mid G\right) P _ {\phi} \left(\perp \left(\theta^ {\prime}\right) \mid G ^ {\prime}\right)}\right) ^ {2} \right] \tag {C.20}
$$
takes the same value and has the same gradient (C.19) as the objective in (C.15) when the subtrajectory balance conditions (in differential form) over undirected paths of length 2 are satisfied.
While optimizing (C.20) alone leads to eventually satisfying the subtrajectory balance conditions over undirected paths of length 2, it may be advantageous to explicitly encourage this behavior, especially in cases where  $d$  is larger and/or for non-linear models. We can incorporate some penalty to the loss function, such as
$$
\begin{array}{l} \tilde {\mathcal {L}} (\phi) = \mathcal {L} (\phi) + \frac {\lambda}{2} \mathbb {E} _ {\pi} \left[ \left\| \nabla_ {\theta} \log P _ {\phi} (\theta \mid G) - \nabla_ {\theta} \log R (G, \theta) \right\| ^ {2} \right. \tag {C.21} \\ \left. + \left\| \nabla_ {\theta^ {\prime}} \log P _ {\phi} \left(\theta^ {\prime} \mid G ^ {\prime}\right) - \nabla_ {\theta^ {\prime}} \log R \left(G ^ {\prime}, \theta^ {\prime}\right) \right\| ^ {2} \right] \\ \end{array}
$$
C.3.3 Marginal distribution over complete states
where  $P_{\phi}(G \mid G_0)$  is the marginal probability of reaching  $G$  from the initial state  $G_0$  with any (complete) trajectory  $\tau = (G_0, G_1, \dots, G_{T-1}, G)$ :
$$
P _ {\phi} (G \mid G _ {0}) \triangleq \sum_ {\tau : G _ {0} \sim \rightarrow G} \prod_ {t = 0} ^ {T - 1} P _ {\phi} (G _ {t + 1} \mid G _ {t}),
$$
using the conventions  $G_{T} = G$ , and  $P_{\phi}(G_0 \mid G_0) = 1$ .
Proof. We assume that the SubTB conditions are satisfied for all undirected paths of length 3 between any  $(G, \theta)$  and  $(G', \theta')$ , that is
$$
R \left(G ^ {\prime}, \theta^ {\prime}\right) P _ {B} (G \mid G ^ {\prime}) P _ {\phi} (\theta \mid G) = R (G, \theta) P _ {\phi} \left(G ^ {\prime} \mid G\right) P _ {\phi} \left(\theta^ {\prime} \mid G ^ {\prime}\right). \tag {C.22}
$$
Let  $G \neq G_0$  be a fixed DAG different from the initial state, and  $\theta$  a set of corresponding parameters. Let  $\tau = (G_0, \dots, G_{T-1}, G)$  be an arbitrary trajectory from  $G_0$  to  $G$ , where we use the convention  $G_T = G$ . For any  $t < T$ , if  $\theta_t$  is a fixed set of parameters associated with  $G_t$ , then the SubTB conditions above can written for every timestep as
$$
R \left(G _ {t + 1}, \theta_ {t + 1}\right) P _ {B} \left(G _ {t} \mid G _ {t + 1}\right) P _ {\phi} \left(\theta_ {t} \mid G _ {t}\right) = R \left(G _ {t}, \theta_ {t}\right) P _ {\phi} \left(G _ {t + 1} \mid G _ {t}\right) P _ {\phi} \left(\theta_ {t + 1} \mid G _ {t + 1}\right), \tag {C.23}
$$
again, using the convention  $\theta_T = \theta$ . Taking the product of the ratio between  $P_{\phi}$  and  $P_B$  over the trajectory  $\tau$ , we get
$$
\begin{array}{l} \prod_ {t = 0} ^ {T - 1} \frac {P _ {\phi} \left(G _ {t + 1} \mid G _ {t}\right)}{P _ {B} \left(G _ {t} \mid G _ {t + 1}\right)} = \prod_ {t = 0} ^ {T - 1} \frac {P _ {\phi} \left(\theta_ {t} \mid G _ {t}\right) R \left(G _ {t + 1} , \theta_ {t + 1}\right)}{R \left(G _ {t} , \theta_ {t}\right) P _ {\phi} \left(\theta_ {t + 1} \mid G _ {t + 1}\right)} (C.24) \\ = \frac {P _ {\phi} \left(\theta_ {0} \mid G _ {0}\right) R (G , \theta)}{R \left(G _ {0} , \theta_ {0}\right) P _ {\phi} (\theta \mid G)} (C.25) \\ \end{array}
$$
Moreover, the backward transition probability  $P_B$ , defined only over the transitions of the GFlowNet, induces a distribution over the trajectories from  $G_0$  to  $G$  (Bengio et al., 2023), meaning that
$$
\sum_ {\tau : G _ {0} \rightsquigarrow G} \prod_ {t = 0} ^ {T - 1} P _ {B} \left(G _ {t} \mid G _ {t + 1}\right) = 1. \tag {C.26}
$$
$$
\begin{array}{l} P _ {\phi} (G \mid G _ {0}) P _ {\phi} (\theta \mid G) = P _ {\phi} (\theta \mid G) \left(\sum_ {\tau : G _ {0} \sim \rightarrow G} \prod_ {t = 0} ^ {T - 1} P _ {\phi} \left(G _ {t + 1} \mid G _ {t}\right)\right) (C.27) \\ = \frac {P _ {\phi} \left(\theta_ {0} \mid G _ {0}\right)}{R \left(G _ {0} , \theta_ {0}\right)} R (G, \theta) \left(\sum_ {\tau : G _ {0} \sim \sim G} \prod_ {t = 0} ^ {T - 1} P _ {B} \left(G _ {t} \mid G _ {t + 1}\right)\right) (C.28) \\ = \frac {P _ {\phi} \left(\theta_ {0} \mid G _ {0}\right)}{R \left(G _ {0} , \theta_ {0}\right)} R (G, \theta) (C.29) \\ \end{array}
$$
We saw in Appendix C.3.2 that  $P_{\phi}(\theta_0 \mid G_0) / R(G_0, \theta_0)$  is independent of the value of  $\theta_0$  if the SubTB conditions are satisfied for all undirected paths of length 3 (see (C.12)). This concludes the proof:  $P_{\phi}(G \mid G_0)P_{\phi}(\theta \mid G) \propto R(G, \theta)$ .
C.3.4 Mini-batch training
Proof. We will first show that  $\log \widehat{R}_{\mathcal{B}}(G,\theta)$  defined in (13) is an unbiased estimate of the log-reward  $\log R(G,\theta)$  under a uniform distribution of the mini-batches  $\mathcal{B}$ . We can observe that by conditional independence of the observations  $\pmb{x}^{(n)}$  given  $G$  and  $\theta$ , we have
$$
\log P (\mathcal {D} \mid \theta , G) = \sum_ {n = 1} ^ {N} \log P \left(\boldsymbol {x} ^ {(n)} \mid \theta , G\right) = N \mathbb {E} _ {\boldsymbol {x}} \left[ \log P (\boldsymbol {x} \mid \theta , G) \right], \tag {C.30}
$$
where the expectation is taken wrt. the uniform distribution over the observations in  $\mathcal{D}$ . It is important to note that we can decompose the likelihood term as in (C.30) because the observations are mutually independent given  $G$  and  $\theta$ ; if we were only conditioning on  $G$  (i.e., using the marginal likelihood,
as in (Deleu et al., 2022)), then those observations would not be conditionally independent in general. Similarly, we have
$$
\mathbb {E} _ {\mathcal {B}} \left[ \sum_ {\boldsymbol {x} ^ {(m)} \in \mathcal {B}} \log P (\boldsymbol {x} ^ {(m)} \mid \theta , G) \right] = M \mathbb {E} _ {\boldsymbol {x}} \left[ \log P (\boldsymbol {x} \mid \theta , G) \right]. \tag {C.31}
$$
Therefore, it shows that the estimate of the log-reward is unbiased:
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {B}} \left[ \log \widehat {R} _ {\mathcal {B}} (G, \theta) \right] = \frac {N}{M} \mathbb {E} _ {\mathcal {B}} \left[ \sum_ {\boldsymbol {x} ^ {(m)} \in \mathcal {B}} \log P (\boldsymbol {x} ^ {(m)} \mid \theta , G) \right] + \log P (\theta \mid G) + \log P (G) (C.32) \\ = \log P (\mathcal {D} \mid \theta , G) + \log P (\theta \mid G) + \log P (G) = \log R (G, \theta). (C.33) \\ \end{array}
$$
Recall that the estimate of the loss is defined as  $\widehat{\mathcal{L}}_{\mathcal{B}}(\phi) = \mathbb{E}_{\pi}\big[\widehat{\Delta}_{\mathcal{B}}^{2}(\phi)\big]$ , where the residual is defined by
$$
\widehat {\Delta} _ {\mathcal {B}} (\phi) = \log \frac {\widehat {R} _ {\mathcal {B}} (G , \bot (\theta)) P _ {B} (G \mid G ^ {\prime}) P _ {\phi} (\bot (\theta) \mid G)}{\widehat {R} _ {\mathcal {B}} (G ^ {\prime} , \bot (\theta^ {\prime})) P _ {\phi} (G ^ {\prime} \mid G) P _ {\phi} (\bot (\theta^ {\prime}) \mid G ^ {\prime})}. \tag {C.34}
$$
Taking the expectation of this estimated loss wrt. a random mini-batch  $\mathcal{B}$ , we get
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {B}} \left[ \widehat {\mathcal {L}} _ {\mathcal {B}} (\phi) \right] = \mathbb {E} _ {\pi} \left[ \mathbb {E} _ {\mathcal {B}} \left[ \widehat {\Delta} _ {\mathcal {B}} ^ {2} (\phi) \right] \right] (C.35) \\ \geq \mathbb {E} _ {\pi} \left[ \mathbb {E} _ {\mathcal {B}} \left[ \widehat {\Delta} _ {\mathcal {B}} (\phi) \right] ^ {2} \right] (C.36) \\ = \mathbb {E} _ {\pi} \left[ \Delta^ {2} (\phi) \right] (C.37) \\ = \mathcal {L} (\phi), (C.38) \\ \end{array}
$$
where we used the convexity of the square function and Jensen's inequality in (C.36), and the unbiasedness of  $\log \widehat{R}_{\mathcal{B}}(G,\theta)$  (as well as  $\log \widehat{R}_{\mathcal{B}}(G',\theta')$ ) in (C.37); recall that  $\Delta (\phi)$  is given in (C.20) (see also (8)).
Proposition C.1. The mini-batch gradient estimator is unbiased, i.e.,  $\nabla_{\phi}\mathcal{L}(\phi) = \mathbb{E}_{\mathcal{B}}\big[\nabla_{\phi}\widehat{\mathcal{L}}_{\mathcal{B}}(\phi)\big]$ . Therefore, the local and global minima of the expected mini-batch loss coincide with those of the full-batch loss.
Proof. We now show that the gradient estimator is unbiased. We observe that
$$
\nabla_ {\phi} \Delta (\phi) = \nabla_ {\phi} \widehat {\Delta} _ {\mathcal {B}} (\phi) \tag {C.39}
$$
since only the terms corresponding to the rewards differ between  $\Delta (\phi)$  and  $\widehat{\Delta}_{\mathcal{B}}(\phi)$ , and they do not depend on  $\phi$ . Therefore,
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {B}} \big [ \nabla_ {\phi} \widehat {\mathcal {L}} _ {\mathcal {B}} (\phi) \big ] = \mathbb {E} _ {\mathcal {B}} \big [ \nabla_ {\phi} [ \widehat {\Delta} _ {\mathcal {B}} (\phi) ^ {2} ] \big ] \\ = 2 \cdot \mathbb {E} _ {\mathcal {B}} \left[ \widehat {\Delta} _ {\mathcal {B}} (\phi) \nabla_ {\phi} \widehat {\Delta} _ {\mathcal {B}} (\phi) \right] \\ = 2 \cdot \mathbb {E} _ {\mathcal {B}} \left[ \widehat {\Delta} _ {\mathcal {B}} (\phi) \right] \nabla_ {\phi} \Delta (\phi) \\ = 2 \Delta (\phi) \nabla_ {\phi} \Delta (\phi) \\ = \nabla_ {\phi} [ \Delta (\phi) ^ {2} ] \\ = \nabla_ {\phi} \mathcal {L} (\phi), \\ \end{array}
$$
as desired. This implies the expected mini-batch loss and full-batch loss differ by a constant and have the same set of local and global minima. This constant happens to equal
$$
\mathrm {V a r} _ {\mathcal {B}} \left[ \log \frac {\widehat {R} _ {\mathcal {B}} (G , \theta)}{\widehat {R} _ {\mathcal {B}} (G ^ {\prime} , \theta^ {\prime})} \right],
$$
and showing the difference equals this constant yields an alternative proof.
Algorithm 1 JSP-GFN training procedure
1: Initialize the trajectory at  $G_{0}$  the empty graph   
2: repeat   
3: Interactions with the environment   
4: for  $K$  steps do   
5: Sample whether the trajectory continues or not:  $a\sim P_{\phi}(\mathrm{stop}\mid G_t)$    
6: if  $a$  is the "stop" action then   
7: Reset the trajectory:  $G_{t + 1} = G_0$  is the empty graph   
8: else   
9: Sample a new graph  $G_{t + 1}\sim P_{\phi}(G_{t + 1}\mid G_t,\neg \mathrm{stop})$    
10: Store the transition  $G_{t}\rightarrow G_{t + 1}$  in the replay buffer   
11: Update of the parameters  $\phi$    
13: Sample a mini-batch  $\mathcal{B}$  of transitions  $G\to G^{\prime}$  from the replay buffer   
14: Sample  $\theta \sim P_{\phi}(\theta \mid G,\mathrm{stop})$  (and similarly for  $\theta^{\prime}$  ) for each transition in  $\mathcal{B}$    
15: Evaluate the rewards  $R(G,\theta)$  and  $R(G^{\prime},\theta^{\prime})$ $\triangleright$  (5)   
16: Evaluate the loss  $\mathcal{L}(\phi)$  based on  $\mathcal{B}$ $\triangleright$  (8)   
17:  $\phi \gets \phi -\alpha \nabla_{\phi}\mathcal{L}(\phi)$  Update the parameters with stochastic gradient methods   
18: until convergence criterion
D Additional experiments & experimental details
In addition to Bayesian structure learning methods based on variational inference (Lorch et al., 2021; Cundy et al., 2021) or GFlowNets (Nishikawa-Toomey et al., 2023), we also consider 2 baseline methods based on MCMC, and 2 methods based on bootstrapping (Friedman et al., 1999), as introduced in (Lorch et al., 2021). Metropolis-Hastings  $\mathrm{MC}^3$  (MH-MC $^5$ ) samples both graphical structures and parameters jointly at each move, whereas Metropolis-within-Gibbs  $\mathrm{MC}^3$  (Gibbs-MC $^3$ ) alternates between updates of the structure, and updates of the parameters; note that  $\mathrm{MC}^3$  here refers to the Structure MCMC algorithm (Madigan et al., 1995). In terms of bootstrapping methods, we consider a variant (called B-GES $^*$ ) based on GES (Chickering, 2002), and another (called B-PC $^*$ ) based on PC (Spirtes et al., 2000). However, since this would yield an approximation of the marginal posterior  $P(G \mid \mathcal{D})$  only, the parameter sample  $\theta$  corresponding to a DAG  $G$  correspond to the parameter inferred by  $P(\theta \mid G, \mathcal{D})$  (see Appendix D.5.1).
D.1 Sampling distribution
In Section 3.3, we saw that the learning objective of JSP-GFN can be written as  $\mathcal{L}(\phi) = \mathbb{E}_{\pi}[\Delta^2 (\phi)]$  where  $\pi$  is a sampling distribution over  $(G,\theta)$  and  $(G^{\prime},\theta^{\prime})$  with full support. We use a combination of on-policy  $(\pi = P_{\phi})$  and off-policy (  $\pi$  is different from  $P_{\phi}$ ) in order to train the GFlowNet: taking inspiration from (Deleu et al., 2022), transitions  $G\rightarrow G^{\prime}$  are sampled off-policy from a replay buffer, whereas their corresponding parameters  $\theta$  and  $\theta^\prime$  are sampled on-policy using our current  $P_{\phi}(\theta \mid G)$ . Therefore, a key difference with Deleu et al. (2022) is that the reward  $R(G,\theta)$  in (5) is calculated "lazily" when the loss is evaluated (i.e., only once  $\theta$  and  $\theta^\prime$  are known), as opposed to being computed during the interaction with the state space and stored in the replay buffer alongside the transitions. Algorithm 1 gives a description of the training algorithm of JSP-GFN.
D.2 Joint posterior over small graphs
D.2.1 Data generation & modeling
Data generation. We follow the same data generation process as in (Deleu et al., 2022; Lorch et al., 2021). More precisely, we first sample a graph from an Erdős-Rényi model (Erdős and Rényi, 1960) over  $d = 5$  nodes, with  $d$  edges on average (a setting typically referred to as ER1). Once the structure of the graph  $G^{\star}$  is known, we sample the parameters  $\theta^{\star}$  of the linear Gaussian model randomly from a standard Normal distribution  $\mathcal{N}(0,1)$ . The linear Gaussian model is defined as
$$
X _ {i} = \sum_ {X _ {j} \in \operatorname {P a} _ {G} \left(X _ {i}\right)} \theta_ {i j} ^ {\star} X _ {j} + \varepsilon_ {i}, \tag {D.1}
$$
(a) Edge features
(b) Path features
(c) Markov features
(d) Edge features
(e) Path features
(f) Markov features
Figure D.1: Comparison of the marginals over graphs in terms of features computed with the exact posterior (x-axis) and the approximation given by JSP-GFN (y-axis). (a-c) Comparison with JSP-GFN (diag); (d-f) comparison with JSP-GFN (full). Each point corresponds to a pair of variables  $(X_{i},X_{j})$  for each of the 20 datasets.
where  $\theta_{ij}^{\star}\sim \mathcal{N}(0,1)$ , and  $\varepsilon_{i}\sim \mathcal{N}(0,0.01)$ ; this defines all the CPDs necessary for (1). Finally, we use ancestral sampling to generate  $N = 100$  observations to create the dataset  $\mathcal{D}$ .
Modeling. We use a linear Gaussian Bayesian Network to model the data, where the CPD for the variable  $X_{i}$  can be written as  $P\big(X_{i}\mid \mathrm{Pa}_{G}(X_{i});\theta_{i}\big) = \mathcal{N}(\mu_{i},\sigma^{2})$ , where
$$
\mu_ {i} = \sum_ {j = 1} ^ {d} \mathbb {1} \left(X _ {j} \in \operatorname {P a} _ {G} \left(X _ {i}\right)\right) \theta_ {i j} X _ {j}, \tag {D.2}
$$
and where  $\sigma^2 = 0.01$  is a fixed variance across variables, matching the variance used for data generation. We place a unit Normal prior over the parameters  $\theta_{ij}$  of the model:  $P(\theta_{ij} \mid G) = \mathcal{N}(0,1)$ . This model differs from the widely used BGe score (Geiger and Heckerman, 1994) in that  $\sigma^2$  is treated as a hyperparameter here, instead of a parameter of the model, and therefore the resulting log-reward is not score equivalent (i.e., placing the same reward for Markov equivalent DAGs; Koller and Friedman, 2009). We used a uniform prior over graphs. Under this model, the posterior distribution  $P(\theta \mid G, \mathcal{D})$  can be computed analytically, and the proof is available in Appendix D.5.1.
D.2.2 Comparing JSP-GFN with the exact posterior against features
In Section 5.1, we evaluated the accuracy of the posterior approximation returned by JSP-GFN by comparing them on the edge features, i.e., the marginal distribution of an edge  $X_{i} \rightarrow X_{j}$  is present in the graph:
$$
P \left(X _ {i} \rightarrow X _ {j} \mid \mathcal {D}\right) = \mathbb {E} _ {P (G \mid \mathcal {D})} \left[ \mathbb {1} \left(X _ {i} \rightarrow X _ {j} \in G\right)\right], \tag {D.3}
$$
where the expectation in (D.3) is either over the true (marginal) posterior  $P(G \mid \mathcal{D})$  (for the x-axis of Figure 2), or over the distribution  $P_{\phi}^{\top}$  induced by the GFlowNet (discarding  $\theta$ , see Section 3.4, for the y-axis). Besides edge features, there exists other marginals of interest (Friedman and Koller, 2003; Deleu et al., 2022), such as the path feature and the Markov feature. The path feature corresponds to