On Divergence Measures for Training GFlowNets
Authors: Tiago Silva
Eliezer de Souza da Silva
Diego Mesquita
Venue: NeurIPS 2024 poster (2024)
Keywords: GFlowNets
Variational Inference
Primary area: generative_models
TL;DR: We show for the first time that f-divergence-based VI can be used to speed up GFlowNet training by employing appropriate variance reduction techniques

Abstract: Generative Flow Networks (GFlowNets) are amortized samplers of unnormalized distributions over compositional objects with applications to causal discovery, NLP, and drug design. Recently, it was shown that GFlowNets can be framed as a hierarchical variational inference (HVI) method for discrete distributions. Despite this equivalence, attempts to train GFlowNets using traditional divergence measures as learning objectives were unsuccessful. Instead, current approaches for training these models rely on minimizing the log-squared difference between a proposal (forward policy) and a target (backward policy) distributions. In this work, we first formally extend the relationship between GFlowNets and HVI to distributions on arbitrary measurable topological spaces. Then, we empirically show that the ineffectiveness of divergence-based learning of GFlowNets is due to large gradient variance of the corresponding stochastic objectives. To address this issue, we devise a collection of provably variance-reducing control variates for gradient estimation based on the REINFORCE leave-one-out estimator. Our experimental results suggest that the resulting algorithms often accelerate training convergence when compared against previous approaches. All in all, our work contributes by narrowing the gap between GFlowNet training and HVI, paving the way for algorithmic advancements inspired by the divergence minimization viewpoint.

On Divergence Measures for Training GFlowNets
Tiago da Silva (tiago. henrique,
Eliezer de Souza da Silva eliezer. cliezer.silva, Saiego.esqque @fggv gv..r... br
Diego Mesquita
School of Applied Mathematics Getulio Vargas Foundation Rio de Janeiro, Brazil
Abstract
Generative Flow Networks (GFlowNets) are amortized samplers of unnormalized distributions over compositional objects with applications to causal discovery, NLP, and drug design. Recently, it shown that GFlowNets hierarchical variational inference (HVI) method for discrete distributions. Despite this equivalence, attempts to train Gets using traditional divergence measures as learning objectives were unsuccessful. Instead, current approaches for training these models rely minimizing the log-squared difference between a proposal (forward policy) and a target (backward policy) distribution. In this work, we first formally extend the relationship between GFlowNets and HVT to distributions on arbitrary measurable topological spaces. Then, we empirically show that the ineffectiveness of divergence- based learning of GFlowNets due the large gradient variance of the corresponding stochastic objectives. To address this issue, we devise a collection of provably pariance-reduciio control variates for gradient estimation based on the REINFORCE leave-one- estimator. Our experimental results suggest that the resulting algorithms often accelerate training convergence when compared against previous approaches. All in all, our work contributes by narrowing the gap between GFlowNet training and HVI, paving the way for algorithmic advancements inspired by the divergence minimization viewpoint.
be framed
Introduction
The approximation of intractable distributions one the central issues in machine learning and modern statisties [7, 35]. In reinforcement learning (RL), recurring goal is to find diverse set of high-valued state-action trajectories according to a reward function. This problem may be cast sampling trajectories proportionally the reward, which is generally intractable distribution over the environment [3, 9, 38, 50]. Similarly, practical Bayesian inference and probabilistic models computations involve assessing intractable posterior distributions [36, 78, 102]. In the variational inference (VI) approach, circumventing this intractability involves searching for tractable approxi- mation the target distribution within family of parametric models. Conventionally, the problem reduces to minimizing a divergence measure, such as Bellback-Leibler (KL) divergence [7, 36, 92] or Renyi-a divergence [51, 70]. between the variational approximation and the target.
In particular, Generative Flow Networks (etsswNets) [3, 4, 48] are recently proposed family of variational approximations well-suited for distribution over compositional objects (e.g., graphs and texts). GFlowNets have found empirical success within various applications from causal discovery [15, 16), NLP [30], and chemical and biological modeling [3, 32]. In nutshell, GFlowNet learns an iterative generative process (IGP) [26] over an extension the target's support, which, for sufficiently expressive parameterizations of transition kernels, yields independent and correctly distributed samples [3, 48]. Remarkably, training GFlowNets typically consists of minimizing the log-squared difference between proposal and target distributions over the extended space via SGD [4, 55], contrasting with divergence-minimizin. algorithms commonly used in VI [7, 72].
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Indeed, Malkin et al. [56] suggests that trajectory balance (TB) loss training for GFlowNets leads to better upproximations of the target distribution than directly minimizing the reverse and forward KL divergence, particularly in setups with sparser rewards. Nevertheless, as we highlight in Section 3, these results are a potential consequence of biases and high variance in gradient estimates for the divergence's 's estimates, which can be overlooked in the evaluation protocol reliant upon sparse target distributions. Therefore, in Section 5, we present a comprchensive empirical investigation of the minimization of well-known fell-knownf-divereee f-divergence measures (including reverse and forward KL), showing is an effective procedure that often accelerates the training convergence of GetowNets relative to alternatives. To achieve these results, we develop in Section 4 collection of control variates (CVs) [63, 71] to reduce the variance without introducing bias on the estimated gradients, improving the efficiency of the optimization algorithms [77, 85]. In summary, our main contributions are:
We evaluate the performance of forward and reverse KI- [47], Renyi-a [74] and Tsallis-o [91] divergences as learning objectives for GFFloWets through an extensive empirical campaign and highlight that they frequently outperform traditionally employed loss functions.
2. We design control variates for the gradients of GFlowNets' divergence-basee objectives. Therefore, it is possible to perform efficient evaluations of the optimization objectives using automatic differentiation frameworks [69], and the resulting experiments showcase the significant reduction in the variance of the corresponding estimators.
3. We developed a theoretical connection between GFlowNets and VI beyond the setup of finitely supported measures [56, 1 112, establishing results for arbitrary topological spaces.
2 Revisiting the relationship between GFlowNets and VI
Initially, we review Lahlou et al. [48]'s work on GFlowNets for distributions on topological spaces, a perspective applied consequentialyy to obtain the equivalence between GFlowNets training and VI divergence minimization in a more generic setting. Finally, we describe standard variance reduction techniques for solving stochastic optimization problems.
Notations. Let (S, T) be topological space with topology T and Σ be the corresponding Borel o-algebra. Also, let v: Σ + R4, be a measure over Σ and Ky. Ri: S x Σ - R4, be transition kernels over S. For each (B1, E E x E, we denote by y @ k(B1, B2) = SB. v(ds)k(s, B2). Likewise, we recursively define the product kernel as (s. = x(s, -) and, for 2 1, x x0 (s,
x (s. @ for a transition kernel x and s E S. Note, in particular, that A x is a function from S x go R4, with g00++ representing the product olalgebra of Σ [1. 96]. Moreover, if μ an absolutely continuous measure relatively to V/, denoted μ < v, we write du/az for the corresponding density RRadom-Nikodye
[1]. Furthermore, we denote by P(A) (S: S S A} the power-set of a set A C S and by [d] (1,..., d} the first d positive integers.
derivative)
GelowNets. A GelowNet is, in its most general form, built upon the concept of a measurable pointed directed acyclic graph (DAG) [48], which we define next. Intuitively, it extends the notion of a flow network to arbitrary measurable topological spaces, replacing the directed graph with transition kernel specifying how the underlying states are connected.
Definition 1 (Measurable pointed DAG [48]). Let T, be measurable topological space endowed with a reference measure / and forward ny and backward Ky kernels. Also, let s, E S and 8y E S be distinguished elements in S, respectively called initial and final states, and S S { We assume { is open. A measurable pointed DAG is then tuple (S,T, T,, TT,,. E, E, Kib, v) satisfying:
1. (Terminality) If (s, {sy}) > 0, then (s, {syh) = Vs E S. Also, (5 = Ezy-
S B) > 0, i.e., B is reachable from 8o. 2. (Reachability) For all B E Σ, 3n E S.L. Ki
3. (Consistency) For every (B1, E Σ E such that (Bi, B2) & (10,,0)) (81, & (B, B₂) Ky (22, B1). Moreover, Ko(So, B) = 0 for every B E E.
4. (Continuity) (s, B) is continuous for B Σ.
5. (Finite absorption) There is a N N such that K3 (s,-) da, for every a S. We designate the corresponding DAG as fimielly absorbing.
In this setting, the elements in the set = {s E S {8y): (s {ar}) > 0)) are called terminal states. Illustratively, when S is finite and is the counting measure, the preceding definition

---

corresponds to connected DAG with an edge from s E S to s' E S iff (r(s.{66) > 0, with condition 5) ensuring acyclicity and condition 2) implying connectivity. GFlowNeet then, is characterized by measurable pointed DAG, potentially unnormalized distribution over terminal states X' and learnable transition kernels on S (Definition 2). Realistically, its goal is find an IGP over S which, starting at 80. samples from X' proportionally to given positive function.
Definition 2 (GFlowNets [48]). A GFlowNer is tuple (G.Pr,PBB.)) composed of a measurable pointed DAG G, o-finite measure < v, and a- p-finite Markov kernels Pp & K and Pg & Kb+ respectively called forward and backward policies.
Training GFlowNets. In practice, we denote by pr, S x S - R the density of Pr relative to Ky- which parameterize using neural network with weigths O. Similarly, we denote by pn the density of PB wrt ky. Our objective is. for given target measure R < on X' with r = df/du, estimate the 0 for which the distribution over X' induced by Pp(so.)) matches R. i.c, for every B E E.
R(B)
Enso
(So, PF. S"
PF.
S1: s.) 8tn dsim) R(X)
Importantly, the above sum contains only finitely many non-zero terms due to the finite absorption
property of Ky. To ensure that pr, abides by this equation, Lahlou et al. [48] showed it suffices that one of the next balance conditions are concomitantly satisfied by Pp and Pn.
Definition 3 (Trajectory balance condition). For all n 2 0 and -ammostt surely Vsi.m E S", Min PFg (So, Sy) Zo PB (5n, $m:1, So), w/ Zo denoting the target distribution's 's partition function.
r(*n), 3em
Definition 4 (Detailed balance condition). For an auxiliary parametric function u: S + RR., and -lmost surely (s, E S², u(s)pr, (s, s') ((ss)pps(()))) (s',s) s) and u(x) = r() for x E X.
To enforce the trajectory balance (TB) or detailed balance (DB) conditions, we conventionally define a stochastic optimization problem to minimize the expected log-squared difference between the left- and right-hand sides of the corresponding condition under a probability measure & supported on an appropriate space [4, 15, 48, 49, 52, 55, 67]. For TB, e.g.. we let & be defined on M with support (u)p( Then, we estimate the GFlowNet's parameters by minimizing
Zo log Pr, (8i, (86, + log r(sn)
Σ OSiSN-1
(1)
PB(S:41,
g
with r == Sa:N and h = max {i: 81 F 85) being the last non-final state's index in r. As denoted by the subscript on Zo, using the TB loss incurs learning the target's normalizing constant. While tuning Pa during training is also possible, the common practice is to keep it fixed.
Henceforth, we will consider the measurable space of trajectories (Ps.Ep) with Ps (s.,,)..... Sn, s5) E S x {sy}: 0 S n S N 1} and Σp as the o-algebra generated by UN+1 g For notational convenience, we use the same letters for representing the measures and kernels of (S, E and their natural product counterparts in (Ps, Dp). which exist by Carathéodory extension's theorem [96]; for example, v(B) D (B) for B = (BI...., B.) y and Pr,(T] (T|s;; 0) is the density of p :) for (8o, 81,- ... Sn, 85) relatively to f n In this case, we will write 7 for a generic element of Ps and x for its terminal state (which is unique by Definition 1). For comprchensive overview of GFlowNets, please refer to [48, 55].
GFlowNets and VI. GFlowNets can be interpreted as hierarchical variational models by framing 3 demonstrated that, for discrete target distributions, the TB loss in (1) aligns with the KL divergence in terms of expected gradients. Extending this, our Proposition establishes that this relationship also holds for distributions over arbitrary topological spaces.
the forward policy Pr, (T|s,, 8) in (Ps, as a proposal to r(z) (T((z)) Malkin et al. [56]
Proposition (TB loss- and KL divergence gradients for topological spaces). Let CTB(T; = (So...., ......,... 8n 1, 88) Then,
(log Zor, (1or and pn(T) = ríz) (=(=n-11102)))
V Vell [TT((T; =2 (
(2)
where DKL(PF, pa] Pr [log
is the KL divergence between Py and PB-
3

---

This proposition shows that minimizing the on on-policy TB loss is theoretically comparable to mini- mizing the KL divergence between Pp and PB terms of convergence speed. Since the TB loss requires estimating the intractable R(2"), the KI, divergence, which avoids this estimation, can be more suitable objective. Our experiments in Section 5 support this, with proofs provided in Appendix C. Extending this result to general topological spaces broadens the scope of divergence minimization strategies, extending guarantees for discrete spaces to continuous and mixed spaces, This generalization aligns with advances in generalized Bayesian inference [45] and generalized VI in function spaces [95], via optimization of generic divergences. We make the method theoretically firm and potentially widely applicable by proving the equivalence in these broader contexts.
Variance reduction. A naive Monte Carlo estimator for the gradient in Equation 2 has high variance [19], impacting the efficiency of stochastic gradient descent [97]. To mitigate this, we use control variates random variables with zero expectation added to reduce the estimator's 's variance without bias [63, 71J. This method, detailed in Section 4, significantly reduces noise in gradient estimates and pragmatically improves training convergence, as shown in the experiments in Section
3 Divergence measures for learning GFlowNets
This Section presents four different divergence measures for training GFlowNets and the accompany- ing gradient estimators for stochastic optimization. Regardless of the learning objective, recall that our goal is to estimate by minimizing a discrepancy measure D between Pp and PB that is globally minimized if and only if PF PB. i.c.
0" === arg min D(Pr, Pn);
in which PB is typically fixed and Pp 's density p/p is parameterized by 8.
3.1 Renyi-a and Tsallis-a divergences
Renyi-a [74] and Tsallis-ox [91] are families of statistical divergences including, as limiting cases, the widespread KI. divergence (Section 3.2) [58]; see Definition 5. These divergences have been successfully applied to both variational inference [51] and policy search for model- based reinforce- ment learning [18]. Moreover, as we highlight in Section 5, their performance is competitive with, and sometimes better than, traditional learning objectives for GFlowNets based on minimizing log-squared differences between proposal and target distributions.
Definition (Renyi-a and Tsallis-a a divergences). Let a E R. Also, let pF, and Pæ be GFlowNet's forward and backward policies, respectively. Then, the Renyi-a divergence between Pp and Pp is
Ra (PrlPPP)) = - log I pr.
(xy(s))))(((s
ps
Similarly, the Tsallis-a divergence between PF and Pg is
-- 1 (PPlI(PBB - (L PF. (To (my( ( -1)
From Definition 5, we see that both Renyi-a and Tsallis-a divergences transition from mass- covering to a mode- sede-ns behavior as a ranges from -00 to co. Regarding GFlowNet-training this fiexibility suggests that R., and T, are appropriate choices both, e.g., for carrying out Bayesian inference [16] - where interest lies in obtaining an accurate approximation to posterior distribution and for combinatorial optimization [109] where the goal is to find few high-valued samples. Additionally, the choice of a provides a mechanism for controlling which trajectories are preferentially sampled during training, with larger values favoring the selection of trajectories leading to high- probability terminal states, resembling the effect of s-greedy [55], thompson-sampling [73], local- search [40], and forward- looking [65, 66) techniques for carrying out off- off-policy training of GelowNets [56].
To illustrate the effect of a on the learning dynamics of GFlowNets, we show in Figure 1 an early stage of training to sample from a homogeneous mixture of Gaussian distributions by minimizing Renyi-a divergence for different values of a; see Section 5.1 for details on this experiment. At this stage, we note that the GFlowNet covers the target distribution's 's modes but fails to separate them when is large and negative. In contrast, large positive causes the model to focus a single

---

boghh-robbbility region. Therefore, the use of an intermediate value for a 0.5 culminates in model that accurately approximates the target distribution. Also, our early experiments suggested the persistence of such dependence on for diverse learning tasks, with = 0.5 leading to the best results. Thus, we fix = 0.5 throughout our experimental campaign.
<
Importantly, we need only the gradients of R. and Ta for solving the optimization problem Equation 3 and, in particular, learning the target distribution's 's normalizing constant is unnecessary, as we underline in the lemma below. This property distinguishes such divergence measures from both TB and DB losses in Equation and, principle, simplifies the training of GFlowNets.
0.0
2.5
0.0
2.5
-2.0
Target
Lemma 1 (Gradients for R., and Ta). Let 0 be the param- eters of pr, in Definition 5 and, for T E Ps. g(T,0)
(pa(t/bbr(o///rr(/bbboo)) The gradient of R.a wrt 0 is
2.5
VaR. (PFollmn) Vog(T 0) + g(T, V log pF, (T (T|s::0) 0).
Figure 1: Mode-seeking (a = 2)
(a () 0)]
versus mass-covering (a -2)
behaviour in a-divergences.
the expectations are computed under Pp. Analogously, the gradient Ta wrt 0 is
Vo VoTa(pr,llee) C VoVoO(T, 0) + g(T, 0)Va log (r|sn;
(a-1)
in which denotes equality up to a multiplicative constant.
Lemma 1 uses the REINFORCE method [97] to compute the gradients of both R. and Ta and we implement Monte Carlo estimators to approximate the ensuing expectations based on a batch of trajectories (TI...., } sampled during training [56]. Also, note that the function g computed outside the log domain and, therefore, particular care required to avoid issues such as numerical underflow of the unnormalized distribution [3, 87]. In our implementation, we sample an initial batch of trajectories {T and compute the maximum of r among the sampled terminal states in log space, i.e., log max; logr(z.). Then, consider log)(x) log r(x) log ē the target's unnormalized log density. In Section 4, we will consider the design of variance reduction techniques to decrease the noise level of gradient estimates and possibly speed up the learning process.
3.2 Bielbbcck-Liibler divergence
The KL divergence [47] is a limiting member of the Renyi-a and Tsallis-o families of divergences, derived when a - [70], and the most widely deployed divergence measure in statistics and machine learning. To conduct variational inference, one regularly considers both the forward and reverse KL divergences, which we review in the definition below.
Definition 6 (Forward and reverse KL). The forward KL divergence between target PB and proposal Pr D [P P Pr] = Palee))) [log (o(())/pppp Also, the reverse KL divergence defined by PKL[PA) | = Py-
PF, (1ii///t)))))))ttt)))
Remarkably, we cannot use simple Monte Carlo estimator to approximate the forward KL due to the presumed intractability of PB (which depends directly on R). As first approximation, we could estimate D D [F]] via importance sampling w/ PF proposal distribution in [56]:
PT(T)
log (T(
Dxr[Pal[Pr) ET-PE
[PF. 80) Pr. (T|s.)
and subsequently implement a REINFORCE estimator to compute VoDKL[PE[IPe) Nevertheless, as we only need the divergence's 's derivatives to perform SGD, we apply the importance weights directly to the gradient estimator. We summarize this approach in the lemma below.
Lemma 2 (Gradients for the KL divergence). Let 0 be the parameters of Pp and s(T; = log Pr. (T(s::00 f). Then, the gradient of D (Pr[[P relatively to @ satisfies
PF. (T())) pn(T(2)r(x) Vos(T:
VoDKL [P]] Pa] Er-Pr(sor)
Ves(T; + log
5

---

Renglwe
Forword KI.
Rovene K1
Telliso
1.0
1.5
3.0
Winver
1.0
Varians
0.3
0.5
100
0.5
0.0
500
1000
300
1000
500
1000
Batch 5i0e
Figure 2: Variance of the estimated gradients as function of the trajectories" batch size. Our control variates greatly reduce the estimator's 's variance, even for relatively small batch sizes.
Correspondingly the gradient of D [PF] wri 0 is
VoDKI,PalIPP _ PByee((ar))
DF, (T(S)) Pb(T|x)r(z) Vos(T:0)
Crucially, choosing an appropriate learning objective is an empirical question that one should consider on a problem-by-pee po-oo---------mm basis similar to the problem of selecting among Markov chain simulation techniques [27]. In particular, a one-size-fits fitzsllftts-ll solution does not exist; see Section for thorough experimental investigation. Independently
the chosen method, however, the Monte Carlo estimators for the quantities outlined in Lemma 2 are of potentially high variance and may require a relatively large number trajectories to yield a reliable estimate the gradients [97]. The following sections demonstrate that variance reduction techniques alleviate this issue.
4 Control variates for low-variance gradient estimation
Control variates. We first review the concept of control variate. Let f: Ps + I real-valued measurable function and assume that our goal is to estimate E,sz [f[r]] according to probability measure π on Ep (see Section 2 to recall the definitions). The variance of a naive Monte Carlo estimator for this quantity is Var, (f(F)//n On the other hand, consider random variable (RV) g: Ps + IR positively correlated with ff and with known expectation [=[g(r)]. Then, the variance of naive Monte Carlo for EE, (f(r) a(g(r) [9())) for baseline IR is
(I Varr
which is potentially smaller than (f()) if the covariance between and is sufficiently large. Under these conditions, we choose the value of a that minimizes Equation 5 [94], namely, a = (ov=(((r).g(r))/vv) /ar, (o(r))) We then call the function g a control variate [63]. Also, although the quanti- ties defining the best baseline a are generally unavailable in closed form, one commonly uses a batch- based estimate of (f(()) g(T)) and (g(r)) the incurred bias is generally negligible rela- tively to the reduced variance [71. 79, 85]. For vector-valued RVs, we let a be diagonal matrix and exhibit, in the next proposition, the optimal baseline minimizing the covariance matrix's 's trace.
Proposition 2 (Control variate for gradients). Let f.g: Ps - Rd be vector-valued functions and be a probability measure on Ps. Consider a baseline E and assume EE, [g(T)] = 0. Then,
arg min Tr Cov=([(T) a g(T)) (= E2((()) E (x(g(T)T
aeR
Note that, when implementing the REINFORCE gradient estimator, the expectation we wish to estimate may be generally written as (r..)...)) (V (T) + f(7)V,
loylogrr Pr. (T)). For the second term, we use leave-one-ous estimator [85]; see below. For the first term, we use Vo log pr, as control variate, which satisfies Epg(...)) [V log PF, (Tis.; f)] = 0. Importantly, estimating the optimal baseline a" in Proposition 2 cannot be done efficiently due to the non non-linear dependence of the corresponding Monte Carlo estimator on the sample-level gradients [2]; i.e., it cannot represented as vector-Jacobian product, which is efficient to compute in -ooee automatic differentiation (autodiff)) frameworks [8, 69]. Consequently, we consider a linear approximation of both numerator and denominator defining a" in Proposition 2, which may be interpreted instantiation of the

---

delta method [83, Sec. 7.1.3]. Then, given a batch (TI,..., TN} of trajectories, we instead use
En m=1 Vo Vologpe,
(Tn), E Vof(Tn)
(6)
c+ E Vo Pog (Tw)
as the REINFORCE batch-based estimated baseline; (5)) represents the inner product between vectors. Intuitively, the numerator is roughly a linear approximation to the covariance between No Pr, and Vol under Pr. In contrast, the denominator approximately measures the variance of No Po, and > 0 is included to enhance numerical stability. As a consequence, for the reverse KL divergence, Vrf(r)
Ve PF, (T), ss and the term corresponding to the expectation of VT)(T) vanishes. We empirically find that this approach frequently reduces the variance of the estimated gradients by a large margin (see Figure above and Section 5 below).
Leave-one-out estimator. We Px=-P(e..)][(T))) (ogFF( As an alternative to the estimator of Proposition 2, Shi et al. [85] and Salimans and Knowles [82] proposed a sample-dependent baseline of the form a(T,) = EigneN (Ta) for i {I.....NN) NN. The resulting estimator,
focus obtaining low-variance estimate of
8 1 N
f(Tm) N
Σ f(3)) j=1,j%b
Vo log pe, Aoo
(Tn),
unbiased for (f( Vo log Pr, (7) due the independence between 7; and T, for * j. Strikingly, 5 can be swiftly computed with autodiff: if (f(T))) and p = (logpe, pF, (Tn)) then
8 Vo N sg N (1 T)f
- .P) .P)
with sg as the stop-ge graddrrdient operation (e.g., lax. lt.. coop-gradent in JAX [8] and torch. detach in PyTorch [69]). Importantly, these techniques incur minimal computational overhead to the stochastic optimization algorithms relative to the considerable reduction in variance they enact.
Relationship with previous works. Importantly, Malkin et al. [56] used à N Σ,, (TT) as baseline and an importance- weighted aggregation to adjust for the off-policy sampling of trajectories, introducing bias in the gradient estimates and relinquishing guarantees of the optimization procedure. A learnable baseline independently trained to match was also considered. This potentially entailed the inaccurate conclusion that the TB and DB are superior to standard divergence based objectives. Indeed, the following section underlines that such divergence measures are sound and practical learning objectives for GetowNets for a range of tasks.
Illustration of the control variates' effectiveness. We train the GFlowNets using increasingly larger batches of (2': i [[5, }0]]} trajectories with and without CVs. In this setting, Figure 2 showcases the drastic reduction in the variance, represented by the covariance matrix's 's trace, of the estimated learning objectives' gradients w.r.t. the model's parameters promoted by the CVs. Impressively, as we show Figure 6, this approach significantly increases the efficiency of the underlying stochastic optimization algorithm. See Section and Appendix D) for further details.
Training GFlowNets with divergence measures
Our experiments serve purposes. Firstly, show in Section 5.2 that minimizing divergence- based learning objectives leads to competitive and often better approximations than the alternatives based log-squared violations of the flow network's balance. This underlines the effectiveness of well-established divergence measures for training GFlowNets [56, 112]. Secondly, we highlight in Section 5.3 that the reduction of variance enacted by our control variates critically accelerates the convergence of GelowNets. We consider widely adopted benchmark tasks from GFlowNet literature, described in Section 5.1, contemplating both discrete and continuous target distributions. Please refer to Appendix B and Appendix E for additional information on the experimental setup.
5.1 Generative tasks
Below, we provide high-level characterization of the generative tasks used for synthetic data genera- tion and training. For more rigorous description in the light of Section 2, see Appendix B.
7

---

RIM
Seggansoon
Sax
1.5
2 1.0
0.6
2000
1003
2000
3000
0.0
3.0
8.0
Mixtures
Hyperpaii
0.50
0.
0.00
20:
600
Epochs
100000
1000 3000
Figure Divergence- learning objectives often lead to faster training than TB loss. Notably, contrasting with the experiments of [56], there is no single best loss function always conducting to the fastest convergence rate, and minimizing well-known divergence measures is often on par with or better than minimizing the TB loss in terms of convergence speed. Results were averaged across three different seeds. Also, we fix a = 0.5 for both Tsallis-a a and Renyi-a divergences.
Set generation [3, 34, 65, 66]. A state corresponds to set of size up to given S and the terminal states X' are sets of size S; a transition corresponds to adding an element from a deposit DD to 8. The IGP starts at an empty set, and the log-reward of a E X is Edea f(d) for a fixed f: D E.
Autoregressive sequence generation [32, 55]. Similarly, state is a seq. 8 of max size S and a termi- nal state is a seq. ended by an end-of-sequence token: transition appends d to s. The IGP starts with an empty sequence and, for r E X, log r(z) = Ei=l.. (f (x:) for functions f.g.
Bayesian phylogemetic inference (BPI) [111]. A state is a forest composed of binary trees with labeled leaves and unlabelled internal nodes, and a transition amounts to joining the roots of two trees newly added node. Then, is terminal when it is a single connected tree called phylogenetic tree. Finally, given a dataset of nucleotide sequences, the reward function is the unnormalized posterior over trees induced by J&C69's mutation model [37] and uniform prior.
Hypergrid navigation [3, 55, 56, 66]. A state E {O,.... H 1]dd is component of a H-sized and d-dimensional Euclidean grid. The IGP starts at O and, we let d, be the i-th line of the identity matrix and (ss) {ō;: e f1....,d d} A maxy (s + 64), < H), transition either adds a 6 E A(s) to or stops at 8. We use Malkin et al. [55, Section J's reward function with R. = 10-3
Bayesian structure learning [15, 16]. A state s is a DAG representing a Bayesian network; a transition either adds an edge s or stops the IGP. Similarly to Deleu et al. [15], we ensure the added edges preserve the state's 's acyclicity. The reward function is defined as the maximum likelihood of the linear Gaussian structural model induced by the current state based on fixed i.i.d. data set.
Mixture of Gaussians (GMs) [48, 110]. The IGP starts at 0 E R2 and proceeds by sequentially substituting each coordinate with a sample from a real-valued distribution. For a K K-component GM, the reward of x E Rd is defined Σx= (x>V(xlu Στ) with 04 2 0 and Σx ag 1.
Banana-shaper distribution [57, 76]. We use the same IGP implemented for bi-dimensional GM. For E 2R we set r(x) to normal likelihood defined on quadratic function of x, see Equation in the supplement. We use HMC samples as ground truth to gauge performance on this task.
5.2 Assessing convergence speed
Next, we provide evidence that minimizing divergence- based objectives frequently leads to faster convergence than minimizing the standard TB loss [55].
Experimental setup. We compare the convergence speed in terms of the rate of decrease of a measure of distributional eтrоr when using different learning objectives for a GFlowNet trained to sample from each of the distributions described in Section 5.1. For discrete distributions, we adopt the evaluation protocols of previous works [3, 54. 55, 66] and compute the L1 distance between the learned Pr Pr(x; ()) and target r(z). namely. Ezex (pr 0) r(=/2)) To approximate Pr, we use Monte Carlo estimate of pr(x; 0) = E=~Pp((,) (ro,::le::0/ppa(trr
For continuous distributions, we echo [48, 110] and compute Ansen-Shannon's 's divergence between (r 8) and R(x):
DJs Pr R = (Px + DKL [R]MM)
(r(()//)))())) + E [log (())/ Pr
E
z~
8

---

Sequeness
Hypergrid
0.0
Top AC
-0.5
0.0
Rev. KL
2000
250
500
3000
10,000
500
1000
10
Sequences
110
Hypengrid
x1FF
Traliio
TB
Number of modes
2.0
1.6
1.25
DB
12
1.5
VirGral
2000
250
500
5000
10000
500
1000
Epochs
Figure 4: Average reward for the K highest scoring samples (top-K) and Number of Modes found during training for the tasks of sequence design, set generation, hypergrid and DAG environments. With the only exception of the hypergrid task, the minimization of divergence-basee measures leads to similar and often faster discovery of high-valued states relatively their Malance-based counterparts.
Tsallis-o
Renyi-a
Reverse K1.
Forward
Target
0
0
-10
10
10
-10
100
10
-2.5
0.0
2.5
-25
0.0
2.5
Figure 5: Learned distributions for the banana-shaped target. Tsallis-o, Renyi-a and for. KL leads better model than TB and Rev. KL, which behave similarly predicted by Proposition
with M(B) = 1/2 (P, ((M)n(x) being the averaged measure of Pr and R and its corresponding density relatively to the reference measure μ.. Remarkably, for the GMs distri- bution,
directly sample from the target to estimate D and the autoregressive nature of the generative process ensures that pr(x) Pr, PF,(T|s, (T(S.) for the unique trajectory starting 8, and finishing Finally, let X, be the first f. terminal states encountered during training and {T(1):- be an descending ordering of according to Then, we select threshold т E E and an integer K to report NoM(X,) = Kr(x): r(x) 2 T A x E XxH,, called number of modes, and TopK(X,) = AVG (fr(z))): 1 i < K}), referred to as top-K average reward. Both NoM and TopK are metrics of substantial interest in the GFlowNet literature [3, 55, 56, 64, 65].
Hence, get an unbiased estimate of DKL [
F(x))
Results. Figure 3 shows that the procedure min- imizing divergence-bard measures accelerates the training convergence of GelowNets, whereas Figure 5 (for the banana-shaped distribution) and Table highlight that obtain curate model with fix compute budget. The dif- ference between learning objectives not statis- tically significant for the BPI task. Also, we may attribute the superior performance of reverse KL compared to the forward in the sequence genera- tion task the high variance the importance- sampling based gradient estimates. Indeed, the observed differences disappear when we increase the batch trajectories to reduce the estimator's variance (see Figure 8 in Appendix D). In conclusion, our empirical results based on experiments testing diverse generative settings and expanding prior art [48, 56, 112], shows that training methods based on minimizing/ cwanniiinggddvveeee -divergeecce objectives with adequate CVs implemented are practical and effective in many tasks. Correlatively, Figure 4 supports
Table Divergence minimization achieves better than similar accuracy compared to enforcing TB.
BPI
Sequenoes
Sets
GMs
TB
0.2220.01
0.28
0.07:00.00
U.31 20:00
Rev.
0.21::0.4
0.16:0.00
0.03
0.31
For. KL
0.22:0..4
0.23-1011
0.03 0.03-0.00
20.9:0.10
Renyi-a
0.22
0. 0.21::0.10
0.03 00000
0.19:0.15
Tsallis-o:
0.21::0.44
0.22.10.00
0.03-00.00
0.21

---

18.5
28.4
-3.2
Widhott Win
Dxc(PhllPy
Du PuriNNllPa
P.(PH|PP))
Ru(Py|]PP))
18.4
28.6
-1.4
18:2
22.8
-3.6
700
200
Fiodds
Ejochs
Epoclt
Epochs
Figure 6: Learning curves for different objective functions in the task of set generation. The reduced variance of the gradient estimates notably increases training stability and speed.
the fact that minimizing divergence- based objectives frequently implies better coverage of the target's high-probability regions; the only exception is the (extremely sparse) hypergrid task [55].
5.3 Reducing the variance of the estimated gradients
Figure 2 demonstrates that implementing CVs for the REINFORCE, estimator reduces the noise level of gradient estimates significantly. This reduction in variance also accelerates training convergence. To illustrate this, we use the same experimental setup from Section 5.2 and analyze the learning curves for each divergence measure with and without control variates.
Figure shows that the implemented gradient reduction techniques significantly enhance the learning stability of GFlowNets and drastically accelerate training convergence when minimizing the reverse KL divergence. Our results indicate that well-designed CVs for gradient estimation can greatly benefit GFlowNets training. Notably, similar improvements have been observed in the context of Langevin dynamics simulations [22, 31, 44] and policy gradient methods for RL [68, 100].
Results.
6 Conclusions, limitations and broader impact
Discussion. We showed in comprehensive range of experiments that divergence measures common in VI -fowward KI., reverse KI., Renyi-o, and Tsallis-o - are effective learning objectives for training GFlowNets, performing competitively with better than their balance- based counterparts. To achieve this, the introduction of efficacious control variates for low-variance gradient estimation of the divergence-basee objectives was crucial, which is key distinction between our work an prior art [55, 112]. Additionally, we developed the theoretical connection between GFlowNets and VI beyond the setting of finitely supported measures, establishing results for arbitrary topological spaces.
Limitations. Albeit comprehensive and on par with the wider literature, our empirical evaluation was performed on problems of relatively small size due to the intractability of probing GFlowNet's distributional accuracy on very large state spaces. That said, we acknowledge that an assessment on the domains of natural language processing [30] and drug discovery [3] based on context-speciffc metrics would strengthen our conclusions; we leave these tasks to future endeavors. Similarly, while we observed promising results for a 0.5, there might be different choices of a that, depending on the application, might strike a better explore-exple tradeoff and incur faster convergence. Thus, thoroughly exploring different a might be especially useful to practitioners.
Broader impact. Overall, our work highlights the potential of the dis-disssseed VI-inspired schemes for training GFNs, paving the way for further research towards improving the GFlowNets by drawing inspiration from the VI literature. For instance, one could develop redivergence-basee losses for GFNs [19], combine GFNs with MCMC using Ruiz and Titsias ['s]'s divergence, or employ an objective similar to that of importance- weeightee autoencoders [10]. Finally, although an c-greedy off-policy sampling scheme can be easily incorporated into a divergence-minimiziinn through an importance-sampling correction, it remains elusive whether this would be possible for more sophisticated sampling techniques such as replay buffer [15] and local search [40].
algorithm
Acknowledgements
This work was supported by the Fundação Carlos Chagas Filho de Amparo à Pesquisa do Es- tado do Rio Janeiro FAPERJ (STI-260003/000002222),,, the São Paulo Research Foundation FAPESP (2023/00815-6) the Conselho Nacional de Desenvolvimento Científico e Tecnológico CNPq (404336/2023-0), and the Silicon Valley Community Foundation through the University Blockchain Research Initiative (Grant #2022-199610),
10

---

References
S. Axler. Measure, Integration &; Real Analysis. Springer International Publishing, 2020.
ISBN 9783030331436. doi: 00.10079978-9-00--3344----
[2] A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M. Siskind. Automatic differentiation in machine learning: survey. Journal of Marchine Learning Research, 2018.
[3] E. Bengio, M. Jain, M. Korablyov, D. Precup, and Y. Bengio. Flow network based generative models for non-iterative diverse candidate generation. In NeurlPS (NeurIPS), 2021.
[4] Y. Bengio. S. Lahlou, Deleu, E. J. Hu, M. Tiwari, and E. Bengio. Gflownet foundations. Journal of Machine Learning Research (JMLR), 2023.
[5] M. Betancourt. A conceptual introduction to hamiltonian monte carlo. arXiv preprint
(ate::770.023333 2017.
[6] M. Bishop. Pattern Recognition and Machine Learning. Springer, 2007.
[7] M. Blei and et al. Variational inference: review for statisticians. Journal of the American Statisticul Association, 2017.
[8] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-MMin MMine,, and Q. Zhang. JAX: composable transforma- tions of Python+NumPy programs, 2018.
[9] L. Buesing, N. Heess, and T. Weber. Approximate inference in discrete distributions with monte carlo search and value functions. In AISTATS, pages 624-634. PMLR, 2020.
[10] Y. Burda, R. B. Grosse, and R. Salakhutdinov. Importance weighted autoencoders. In ICLR (Poster), 2016.
[11] P. Carbonetto, M. King, and F. Hamze. A stochastic approximation method for inference in probabilistic graphical models. NeurIPS, 2009.
[12] B. Carpenter, A. Gelman, D. Hoffman, D. Lee, B. Goodrich, M. Betancourt, M. Brubaker, J. Guo, P. Li, and A. Riddell. Stan: A probabilistic programming language. Journal of statistical software, 2017.
[13] T. da Silva, E. Silva, A. Ribeiro, A. Góis, D. Heider, S. Kaski, and D. Mesquita. Human- in-the-loop causal discovery under latent confounding using ancestral gflownets. arXiv 2330 12032, 2023.
[14] M. P. Deisenroth and C. E. Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In ICML, Proceedings Machine Learning Research, pages 465-472. PMLR, 2011.
[15] T. Deleu, A. Góis, C. C. Emezue, M. Rankawat, S. Lacoste-Julien, S. Bauer, and Y. Bengio. Bayesian structure learning with generative flow networks. In UAI, 2022.
[16] T. Deleu, M. Nishikawa- Toomey, J. Subramanian, N. Malkin, L. Charlin, and Y. Bengio. Joint Bayesian inference of graphical structure and parameters with single generative flow network. In Advances in Neural Processing Systems (NeurIPS). 2023.
[17] T. Deleu, P. Nouri, N. Malkin, D. Precup. and Y. Bengio. Discrete probabilistic inference as control in multi-path environments. CoRR, abs/2402.10309, 2024.
[18] S. Depeweg, J. M. Hernández-Lobato, F. Doshi-Velez, and S. Udluft. Learning and pol- icy search stochastic dynamical systems with bayesian neural networks. arXiv preprint urXiv:1605.071277 2016.
[19] A. B. Dieng. D. Tran, R. Ranganath, J. Paisley, and D. Blei. Variational inference via upper bound minimization. NewrIPS, 2017.
[20] J. Domke. Provable gradient variance guarantees for black-box variational inference. In NeurIPS, pages 328-337, 2019.

---

[21] J. Domke, Provable smoothness guarantees for black-box variational inference. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 2587-2596. PMLR, 2020.
[22] K. A. Dubey, S. J. Reddi, S. A. Williamson, B. Poczos, A. J. Smola, and E. P. Xing. Variance reduction in stochastic gradient langevin dynamics. NeurIPS, 2016.
[23] S. Dutta, M. Das, and U. Maulik. Towards causality-based explanation of aerial scene classifiers. IEEE Geoscience and Remole Sensing Letlers, 2023.
[24] J.-P. Falet, H. B. Lee, N. Malkin, C. Sun, D. Secrieru, D. Zhang, G. Lajoie, and Y. Bengio. Delta-ai: Local objectives for amortized inference in sparse graphical models, 2023.
[25] J. Felsenstein. Evolutionary trees from DNA sequences: maximum likelihood approach. Journal of Molecular Evolution, 1981.
[26] T. Garipov, S. D. Peuter, G. Yang. Garg, S. Kaski, and T. Jaakkola. Compositional sculpting of iterative generative processes. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
[27] C. J. Geyer. Markov chain monte carlo maximum likelihood. 1991.
[28] G. Hinton, N. Srivastava, and K. Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. 2012.
[29] E. J. Hu, N. Malkin, M. Jain, K. E. Everett, A. Graikos, and Y. Bengio. Gflownet-em for learning compositional latent variable models. In International Conference on Machine Learning (ICLR), 2023.
[30] E. J. Hu, M. Jain, E. Elmoznino, Kaddar, G. Lajoie, Y. Bengio, and N. Malkin, Amortizing intractable inference in large language models. In The Twelfth International Conference on Learning Representations, 2024.
[31] Z. Huang and S. Becker. Stochastic gradient langevin dynamics with variance reduction. CoRR, 2021.
[32] M. Jain, E. Bengio. A. Hernandez-Garcia. J. Rector-Brooks, B. F. P. Dossou, C. A. Ekbote, J. Fu, T. Zhang, M. Kilgour, D. Zhang. L. Simine, P. Das, and Bengio. Biological sequence design with GFlowNets. In International Conference Machine Learning (ICML), 2022.
[33] M. Jain, Deleu, Hartford, C.-H. Liu, A. Hernandez-Garchn and Y. Bengio. Gflownets for ai-driven scientific discovery. Digital Discovery, 2023.
[34] Jang, M. Kim, and Ahn. Learning energy decompositions for partial inference in GFlownets. In The Twelfth International Conference on Learning Representations, 2024.
[35] M. Järvenpää and J. Corander. On predictive inference for intractable models via approximate bayesian computation. Statistics and Computing. 33(2), Feb. 2023. ISSN 1573-1375
[36] M. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction variational methods for graphical models. Mach. Learn., 37(2):183-233, 1999.
[37] T. H. Jukes and C. R. Cantor. Evolution of protein molecules. In Mammalian Protein Metabolism. Elsevier, 1969.
[38] H. J. Kappen, V. Gómez, and M. Opper. Optimal control graphical model inference problem. Mach. Learn., 87(2):159-182 2012.
[39] K. Kim, Y. Ma. and J. Gardner. Linear convergence of black-box variational inference: Should we stick the landing? In AISTATS, volume 238 Proceedings of Machine Learning Research, pages 235-243 PMLR, 2024.
[40] M. Kim, T. Yun, E. Bengio, Zhang, Y. Bengio, S. Ahn, and Park. Local search gflownets. arXiv preprint 2r11v22310002710, 2023.
12

---

[41] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[42] D. P. Kingma, S. Mohamed, D. Jimenez Rezende, and M. Welling, Semi-supervised learning with deep generative models. NeurIPS, 2014.
[43] D. P. Kingma, T. Salimans, B. Poole, and J. Ho. Variational diffusion models, 2023.
[44] Y. Kinoshita and T. Suzuki. Improved convergence rate of stochastic gradient langevin dynamics with variance reduction and its application to optimization. In NewrlPS, 2022.
[45] J. Knoblauch, J. Jewson, and T. Damoulas. An optimization-centiie view bayes' rule: Reviewing and generalizing variational inference. J. Mach. Learn. Res., 13:112:11132:10: 2022.
[46] A. Kucukelbir, D. Tran, R. Ranganath, A. Gelman, and D. M. Blei. Automatic differentiation variational inference. J. Mach. Learn. Res., 18:14:1-14:45, 2017.
[47] S. Kullback and R. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 1951.
[48] S. Lahlou, Deleu, P. Lemos, D. Zhang, Volokhova, A. Hemández-Gareía. L. N. Ezzine, Y. Bengio, and N. Malkin. A theory of continuous generative flow networks. In JCML, volume 202 of Proceedings of Machine Learning Research, pages 18269-18300 PMLR, 2023.
[49] E. Lau, N. M. Vemgal, D. Precup, and E. Bengio. DGFN: Double generative flow networks. In NeurlPS 2023 Generative Al and Biology (GenBio) Workshop, 2023.
[50] S. Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. CoRR,
2018.
[51] Y. Li and R. E. Turner. Rényí divergence variational inference. NeurlPS, 29, 2016.
[52] D. Liu and et al. Gflowout: Dropout with generative flow networks. In International Conference on Machine Learning, ICML'23 23. JMLR.org, 2023.
[53] R. Liu, J. Regier, N. Tripuraneni, M. I. Jordan, and J. D. McAuliffe. Rao-blackwellized stochastic gradients for discrete distributions. In ICMI,, volume 97 of Proceedings of Machine Learning Research, pages 4023-4031. PMLR, 2019.
[54] Madan, J. Rector-Brooks, M. Korablyov, E. Bengio, M. Jain, A. C. Nica, T. Bosc. Y. Bengio, and N. Malkin. Learning gflownets from partial episodes for improved convergence and stability. International Conference on Machine Learning, 2022.
[55] N. Malkin, M. Jain, E. Bengio, C. Sun, and Y. Bengio. Trajectory balance: Improved credit assignment GFlownets. In NeurIPS (NeurIPS). 2022.
[56] N. Malkin, S. Lahlou, T. Deleu, X. Ji, F. Hu, K. Everett, D. Zhang, and Y. Bengio. GFlowNets and variational inference. International Conference on Learning Representations (ICLR), 2023.
[57] D. Mesquita, P. Blomstedt, and S. Kaski. Embarrassingly parallel MCMC using deep invertible transformations. In UAI, 2019.
[58] T. Minka. Divergence measures and message passing. Technical report, Research, Microsoft, 2005. URL https://wwwt seas. harvard. edu/ www..eddddaaaeeee ctt.//oeus2s/// minka-divergence pdf.
[59] S. Mohamed, M. Rosca, M. Figurnov, and A. Mnih. Monte carlo gradient estimation in machine learning. J. Mach. Learn. Res., 21:112::1112:66 2020.
[60] S. Mohammadpour, F. Bengio, E. Frejinger, and P.-.. Bacon. Maximum entropy gflownets with soft g-learning 2023.
13

---

[61] R. M. Neal et al. Mome using hamiltonian dynamics. Handbook of markov chain monte carlo, 2011.
[62] A. C. Nica, M. Jain, E. Bengio, C.-H. Liu, M. Korablyov, M. M. Bronstein, and Y. Bengio. Evaluating generalization in gflownets for molecule design. In ICLR2022 Machine Learning for Drug Discovery, 2022.
[63] A. B. Owen. Monte Carlo theory, methods and examples. 2013.
[64] L. Pan, N. Malkin, D. Zhang, and Y. Bengio. Better training of GFlowNets with local credit and incomplete trajectories. In International Conference Machine Learning (ICML), 2023.
[65] L. Pan, N. Malkin, D. Zhang, and Y. Bengio. Better training of gflownets with local credit and incomplete trajectories. arXiv preprint arXiv:2302.01687, 20233
[66] L. Pan, D. Zhang, A. Courville, L. Huang, and Y. Bengio. Generative augmented flow networks. In International Conference Learning Representations (ICLR), 2023.
[67] L. Pan, Zhang, M. Jain, L. Huang, and Y. Bengio. Stochastic generative flow networks, In UAI, volume 216 of Proceedings of Machine Learning Research, pages 1628-16660 1638. PMLR, 2023.
[68] M. Papini, D. Binaghi, G. Canonaco, M. Pirotta, and M. Restelli. Stochastic variance-reduced policy gradient. In International Conference on Machine Learning. 2018.
[69] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high perfforrffrmeeceeeeeee deep learning library, 2019.
[70] B. Poczos and J. Schneider. On the estimation of a-divergences. In AISTATS. PMLR, 2011.
[71] R. Ranganath, S. Gerrish, and D. M. Blei. Black box variational inference. In AISTATS, volume 33 of JMLR Workshop and Conference Proceedings, pages 814-822. JMLR.org. 2014.
[72] R. Ranganath, D. Tran, J. Altosaar, and D. M. Blei. Operator variational inference. In NeurIPS, pages 496-504, 2016.
[73] J. Rector-Brooks, K. Madan, M. Jain, M. Korablyov, C.-H. Liu, S. Chandar, N. Malkin, and Y. Bengio. Thompson sampling for improved exploration in gflownets, 2023.
[74] A. Rényi. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume I: Contributions to the Theory of Statistics. University of California Press, 1961.
[75] D. Rezende and Mohamed. Variational inference with normalizing flows. In International conference machine learning. PMLR, 2015.
[76] B. Rhodes and M. U. Gutmann. Variational coise-contrastive estimation. AISTATS, 2019.
[77] I.. Richter, A. Boustati, N. Nüsken, J. R. Ruiz, and Ö. D. Akyildiz. Vargrad: A low-variance gradient estimator for variational inference. 2020.
[78] C. P. Robert et al. The Bayesian choice: from decision-theoretic foundattion to computational implementation, volume 2. Springer, 2007.
[79] Roeder, Y. Wu, and D. Duvenaud. Sticking the landing: simple, lower-variance gradient estimators for variational inference. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 6928-6937, 6937, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 1781510860964.
[80] T. G. J. Rudner, V. Pong, R. McAllister, Gal, and Levine. Outcome-driven reinforcement learning via variational inference. In NeurlPS, pages 13045-13058, 2021.
14

---

[81] F. Ruiz and M. Titsias. A contrastive divergence for combining variational inference and meme. In International Conference on Machine Learning. 2019.
[82] T. Salimans and D. A. Knowles. On using control variates with stochastic approximation for variational bayes and its connection to stochastic linear regression, 2014.
[83] M. J. Schervish. Theory of statistics. Springer Science & Business Media, 2012.
[84] M. W. Shen, E. Bengio, E. Hajiramezanali, A. Loukas, K. Cho, and T. Biancalani. Towards understanding and improving gflownet training. In International Conference on Machine Learning, 2023.
[85] J. Shi, Y. Zhou, J. Hwang, M. Titsias, and L. Mackey. Gradient estimation with discrete stein operators. NeurIPS, 35, 2022.
[86] Y. Sun, L. Dong. S. Huang. S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei. Retentive network: A successor transformer for large language models, 2023.
[87] D. Tiapkin, N. Morozov, A. Naumov, and D. Vetrov. Generative flow networks as entropy- regularized rl, 2024.
[88] E. Todorov, General duality between optimal control and estimation. In CDC, pages 4286- 4292. IEEE, 2008.
[89] M. Toussaint and A. Storkey, Probabilistic inference for solving discrete and continuous state markov decision processes. In ICML, volume 148 of ACM International Conference Proceeding Series, pages 945-952, ACM, 2006.
[90] M.-N. Tran, M. K. Pitt, and R. Kohn. Adaptive cestroollis-hasting sampling using reversible dependent mixture proposals. Statistics and Computing. 2016.
[91] C. Tsallis. Possible generalization of boltzmann-gibbe statistics. Journal of statistical physics, 52:479-487, 1988.
[92] M. J. Wainwright and M. Jordan. Graphical models, exponential families, and variational inference. Found. Trends Mach. Learn., 1(1-2):1-305 2008
[93] X. Wang, T. Geffner, and J. Domke. Joint control variate for faster black-box variational inference. In AISTATS, volume 238 of Proceedings of Machine Learning Research, pages 1639-1647. PMLR, 2024.
[94] L. Weaver and N. Tao. The optimal reward baseline for gradient-based reinforcement learning. arXiv preprint arXiv: 22001.235,101 2013.
[95] V. D. Wild, R. Hu, and D. Sejdinovic. Generalized variational inference in function spaces: Gaussian measures meet bayesian deep learning. In NeurIPS, 2022.
[96] D. Williams. Probability with Martingales. Cambridge University Press, 1991.
[97] J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 1992.
[98] B. Xu, N. Wang, T. Chen, and M. Li. Empirical evaluation of rectified activations in convolu- tional network. arXiv preprint arXiv: /505. 00853, 2015.
[99] K. Xu, W. Hu, J. Leskovee, and S. Jegelka. How powerful are graph neural networks? International Conference on Learning Representations (ICLR), 2019.
[100] P. Xu, F. Gao, and Q. Gu. An improved convergence analysis of stochastic variance-reduced policy gradient. In UAI, 2020.
[101] Z. Yang. Molecular Evolution: A Statistical Approach, Oxford University PressOxford, May 2014. ISBN 9780191782251. doi: 10. 00..99//hanffss/7777000000000000
15

---

[102] J. S. Yedidia, W. T. Freeman, and Y. Weiss. Bethe free energy, kikuchi approximations, and belief propagation algorithms. Advances in neural information processing systems, 13(24). 2001.
[103] M. Yin and M. Zhou. Semi-implicit variational inference. In International conference machine learning. PMLR, 2018.
[104] L. Yu and C. Zhang. Semi-implicit variational inference via score matching. arXiv preprint arXiv:2308. 10014, 2023.
[105] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and J. Smola. Deep sets. In NeurIPS, 2017.
[106] C. Zhang, B. Shahbaba, and H. Zhao. Hamiltonian monte carlo acceleration using surrogate functions with random bases. Statistics and computing, 2017.
[107] D. Zhang, R. Chen, N. Malkin, and Y. Bengio. Unifying generative models with gflownets and beyond. ICML Beyond Bayes workshop, 2022.
[108] D. Zhang, N. Malkin, Z. Liu, A. Volokhova, A. Courville, and Y. Bengio. Generative flow networks for discrete probabilistic modeling. In International Conference on Machine Learning (ICML), 2022.
[109] Zhang, H. Dai, N. Malkin, A. Courville, Bengio, and L. Pan. Let the flows tell: Solving graph combinatorial optimization problems with gflownets. In NeurlPS (NeurIPS), 2023.
[110] D. Zhang, R. T. Q. Chen, C.-H. Liu, A. Courville, and Y. Bengio. Diffusion generative flow samplers: Improving learning signals through partial trajectory optimization. In The Twelfth International Conference on Learning Representations, 2024.
[111] M. Y. Zhou, Z. Yan, E. Layne, N. Malkin, D. Zhang, M. Jain, M. Blanchette, and Y. Bengio. PhyloGFN: Phylogenetic inference with generative flow networks. The Twelfth International Conference on Learning Representations, 2024.
[112] H. Zimmermann, F. Lindsten, J. van de Meent, and C. A. Naesseth. variational perspective on generative flow networks. Trans. Mach. Learn. Res., 2023, 2023.
16

---

A Related works
Generative Flow Networks. GFlowNets [3. 4] were initially proposed as reinforcement learning algorithm for finding diverse high-valued states in discrete environment by sampling from a distribution induced by reward function. Shortly after, they were extended to sample from complex distributions in arbitrary topological spaces [48]. Remarkably, this family of models was successfully applied to problems as diverse as structure learning and causal discovery [13, 15, 16], discrete probabilistic modeling and graphical models [24, 29, 107, 108), combinatorial optimization and stochastic control [109, 110], drug discovery [3, 33, 62], design of biological sequences [32], natural language processing [30], and aerial scene classification [23]. Concomitantly to these advances, there is growing literature concerned with the development of more efficient training algorithms for GFlowNets [4, 40, 55, 84] primarily drawing inspiration from existing techniques in the reinforcement learning literature [60, 65, 66, 87]. In the same spirit, Tiapkin et al. [87] showed is reinforcement learning. In study closely related to ours, Malkin et al. [56] proved the equivalence between GFlowNets and hicrarchical variational inference (HVT) for discrete distributions; however, when training GFlowNet using divergence- based methods from the VI literature, the authors found no improvement relatively to the traditional flow-matching objectives. Thus, extending beyond discrete distributions, this work provides definitive analysis of training GFlowNets by directly optimizing a set of divergences typically employed in variational inference training, given clear context and conditions for effective use of divergence objectives for efficient learning procedures applied on GFlowNets models.
possible to frame GFlowNets as aatropy-regularie
Reinforcement Learning as Inference. Reinforcement Learning (RL) has been studied form of probabilistic inference extensively, generating relevant insights in the literature, and alternatively referred to as control as inference. Todorov [88] demonstrates duality between estimation and optimal control, establishing conditions where estimation algorithms could applied for control problems. Kappen et al. [38] demonstrated that optimal control problems could be framed as inference problems in graphical models, providing a unified perspective for solving control tasks. Levine [50] presents complete and modern RL formulation, linking with VI in particular. Rudner et al. [80] integrates even further RL with VI methods, demonstrating the conceptual and algorithmic gains of leveraging outcome-driven RL with variational inference to optimize policy distributions. Developing further, Toussaint and Storkey [89] applies approximate probabilistic inference methods to solve Markov Decision Processes (MDPs) with discrete and continuous states. The approach also aligns with model-based RL techniques, such as PILCO, which utilizes probabilistic models to enhance data efficiency in policy search [14]. Recent work by Deleu et al. [17] positions discrete control problem in multi-path environments, highlighting the synergy
probabilistic inference between control theory and probabilistic modeling in the context of GFlowNets. This body of works relates to the approach presented in this paper, comparing optimization of trajectory balance and fow-matching losses related to sequential decisions modeled by the GFlowNet with f-divergence measures minimization procedures related to approximated variational inference and generalized posterior inference [36, 45, 51, 92, 95].
Divergence measures and gradient reduction for VI. Approximate inference via variational inference (VI) methods [6, 7, 36, 92] initially relied on message passing and coordinate ascent methods to minimize the KL divergence of an unnormalized distribution and proposal in parameterized tractable family of distributions. Despite the initial generality of the optimization perspective, the concrete implementation of algorithms often requires specialized update equations and learning objectives for specific classes of models. On the other hand, the development of algorithms and software for automatic differentiation [2] and stochastic gradient estimators [59] unlocked the potential application of generic gradient-based optimization algorithms in inference and learning tasks for a comprchensive class of models. Seminal works such as Black-Box VI (BBVI) [71], using the REIsFORCE/score function estimator, and Automatic Diffcrentiation VI (ADVI) [46]. using reparameterization and thaage-of-variables demonstrated practical algorithms for Bayesian inference in generic models, including models combining classical statistical modeling with neural networks Overall, Mohamed al. [59] explain the development of the main gradient estimators: the score function [11, 71, 97, 103], and the pathwise gradient estimator, also known as the parametrization trick [42, 43, 75]. The vanilla REINFORCIE/ccre function estimator has notoriously high variance [11, 71, 77, 97], which prompted body of work exploring variance reduction techniques. In the original BBVI proposal, Ranganath et al. [71] explored noo--Blackellizzttion combining iterated conditional expectations and control variates, using the score function estimator (given its zero
17

---

expectation) as control variate. Subsequent works have continued to refine these techniques; Liu al. [53] uses Blackwellized stochastic gradients for discrete distributions, while Kim et al. [39] and Wang et al. [93] explored joint control variates and provable linear convergence in BBVI. Additionally, Domke [21] and Domke [20] provided smoothness and gradient variance guarantees, further enhancing the robustness of score function estimator for VI methods. Our work demonstrates that effective variance reduction techniques applied to a f-diieereeece minimization training can significantly enhance the convergence speed and stability of the procedure. In theory and practice, we observed high compatibility between our results of variance-reduced. feree-rrduueddddiieeeee f-divergence GFlowNets training and the body of work of cariance-reduced ecore-function estimators for VI. Furthermore, by showing that these techniques apply to a broad class of models and optimization objectives, including continuous and mixed structured supports, move GFlowNets' f-divergence minimization training closer to recent notions of generalized Bayesian inference and generalized VI[45] and variational inference in function spaces [95] with the common thread of casting posterior inference as an optimization problem guided by some divergence measure. This generalization can enable applications of GFlowNets to diverse range of machine learning tasks, enhancing their versatility and practical relevance.
18

---

B Detailed description of the generative tasks
Set generation 33, 65, 66]. S contains the sets of size up to N with elements extracted from a fixed deposit D size D 2 N and s, = 0. For E S with [s| < N. (s.s..) is counting measure supported at (the o-algebra induced by) {s J {d) : d E D s}; for |s] = N, (e -) = ōsy- Then,
X = (a E S: |a| N). Similarly, (s,,))) support is {s ) [d] d E D) for 8 + 80- We define the unnormalized target's density by log r(z) = Edez [(d) for a fixed function f: D - IR. We
parameterize pr -) as a deep set [105] and fix PB((s -) as uniform density for E S.
Autoregressive sequence generation [32, 55]. A sequence in D for any K > n. bijectively m):
associated to an element of ' x [K] by s + {(sm. 1 I m S n) U m)) K 2 m > n); I is a token denoting the sequence's is end. In this context, we let S C P(D x [N + +1) be the set of sequences of size up to N, i.e. if s E S and (L, n + 1) E 8, then (d. m) E s iff d =
n; the initial state is So = 0. For conciseness, we write d $ 8, meaning that (d. & 8 for every i. Next, (s).). is the counting measure supported at {s U { + 1)): d D (1)) if
for n
S N + and there is (DU {1))" such that (Um, m) E for
< N and
if =====0 N; and at {8f} otherwise. Thus, X = {8 E S: &). Also, -x) is supported at {s \ {(d, [s]]]: d E D), which has single element corresponding to the removal of the element of 8 of the largest index. On the other hand, the target's 's density is
I. E 8; at
log logr(z)= = E(ds)Ea 1 (()) for functions f.g: D + R. We employ an MLP to parameterize ps(.,))
Bayesian phylogenetic inference (BPI) [111]. A (rooted) phylogeny is a complete binary tree with labeled leaves and weighted edges; each leaf corresponds biological species, and the edges' weights are a measurement of evolutionary time. Formally, we let B be the set of ob- served biological species and V8 be the set of [ž] + unobserved species. Next, we repre- sent a phylogeny over B as a weighted directed tree GB = (B U VB, EB, WB) with edges EB featured with a weight assignment wB: we denote its root by r(GB). Under these conditions,
define S = {U= Gr. Fa =BA 9r, is tree) as the set of forests built upon phy- logenetic trees over disjoint subsets of B; LJ represents a disjoint union of non-empty sets and so = -neE Gioz is the forest composed of singleton trees Gtb)- Also, we define the amalga- mation of phylogenies Gr, and Gy,, A(Gr,, Goy), as the tree obtained by joining their roots (Gr,,) and r(G,;) to new node r(Gr, U UG,,), ). with corresponding extension of the edges' weights. In contrast, the dissolution of a tree GF, R(GF), returns the union of the two sub- trees obtained by removing r(Gr) from Gy. Then, .s) is the counting measure supported at { GF. UAGA GF (i,5) E [K]2, + j} with s = U GF. and K 2 2; if 8 == 98, (8, = E.g. Hence, X' is the set of phylogenies over B. Likewise, Ki (668 y's support is {U=_IMe GF. UR(GA)): i E [K] (r())) $ B) for s = U GF. and K S [B]. Finally, the unnormalized target is the posterior distribution defined by JC69's mutation model [37] given data set of genome sequences of the species in B. More specifically, we let the prior be a uniform distribu- tion and compute the model-induced likelihood function by the efficient Felsenstein's 's algorithm [25]. We assume the edges' weights are constant. See [101] for further details. We parameterize pr($,) with GIN [99] and fix pB(s, -) as an uniform distribution.
Mixture of Gaussians [48, 110). The training of GFlowNets in continuous spaces is challenging, and the problem of designing highly expressive models in this setting is still unaddressed [16, 48]. However, as we show in Section 5.2, divergence-based measures seem to be very effective learning objectives for autoregressive sampling of a sparse mixture of Gaussians. For d-dimensional Gaussian distribution, S = (1, 0), (I,, i): 1 S i S n). n S d, x E R C ((0, JU (d) and s, = (0. 0); note S is isomorphic to Rd. Also, for s {(z,, [(z,,))]=_ (s, is Lebesgue's measure at {sU (z, n + 1): x E R if n < d and (s. = da, otherwise. In particular, X = {s E S: (a,)e i = d}. Moreover, (o(s.-) -) is a measure on {s (z, ls)): x E R). which is isomorphic to IR, which is a singleton. We define the target's density with a homogeneous mixture of Gaussian distributions, 1 Σx= E N(Hs, W(1,,021) T) with μ E IRd, We similarly define Pp(#,) :) as a mixture of one- mic-dimensional Gaussians with mean and variance learned via an MLP [48].
19

---

Banana distribution. [57, 76] We consider sampling from the banana distribution, defined by
NV 8/2 -
0
1
0.9
)
(8)
0
0.9
Given its geometry and shape, this distribution is a common baseline in the approximate Bayesian inference literature [90, 104, 106]. This task is identical to sampling from mixture of Gaussian distributions, except for the different target density specified by the model in Equation (8). Also, we
rely on the implementation Hamiltonian Monte Carlo (HMC) [5, 61] provided by Stan [12] obtain accurate samples from (8).
A similar description may be utilized for the hypergrid and structure learning domains.
20

---

C Proofs
We will consider the measurable space of trajectories (Ps, Er), with Ps { 81,.... Sn, Sn, 85)
S { 0 I n s N } and Σp as the o-algebra generated by Un=1 I For notational convenience, we use the same letters for representing the measures and kernels of (S. E) and their natural product counterparts in (Ps. Σp), which exist by Carathéodory extension's 's theorem [96]; for example, v(B) y (B) for B (Bi,... Bn) E X and PF, (T]80; 0) is the density of PF pon+ (So. for (Sa, $1..... Sn 8)) relatively to n In this case, we will write T for generic element of Ps and z: for its terminal state (which is unique by Definition 1).
C.1 Proof of Proposition!
We will show that the gradient of the expected on-policy TB loss matches the gradient of the KL divergence between the forward and backward policies. Firstly, note that
Voll PF log pF(T|s0:0] PB(T)
Kz [F
V log pF(T|soi 0) dP dPp(so.
(T(
( 0)
V
log
PF (s,
(B(T)
/ V log
PF(T|soi 0)
Pr(so. dr)
PF(T)
+ f.1o log pr[T|so: 0)
Vops (T|s:: ()
PB(T)
by Leibniz's rule for integrals and the product rule for derivatives. Then, since Vef(0) V() log f(0) for any differentiable function f: 0 f(0).
VODKL [Prl
Vo (oggpr(T)s)) + log Pr(T|s0) (NoP)
E
Vo (F((s))
Ps(T)
(9)
log PF(T so)
Ve logpr(r(s.)
(Po))
(T((T
we omitted the dependency of Pp (and of pF thereof) on the parameters 0 for conciseness, On the other hand,
VoLTB(T;0)- = log pr(T(sa;0)
Ve logpF(T)
(10)
PB(T)
by the chain rule for derivatives. Thus,
(r VoLTD(T; 2V Pa),
(11)
ensuring that the equivalence between LTB and DRL in terms of expected gradients holds in context broader than that of finitely supported distributions [56].
C.2 Proof of Lemma
Henceforth, we will recurrently refer to the score estimator for gradients of expectations [97]. namely,
E PP..)
[fo[r]]
+ 11
(12)
(r
which can be derived using the arguments of the preceding section. In this context, the Renyi-a's divergence satisfies
VoLr-re(s.. (g(T, 0)]
VoR. (a 1)Br Pr( (9( 0)
with O(T:0) (pu((1))))()))p(((((1100).) and * 1; similarly, the Tsallis-a's divergence abides by
(oTa(Pr|PP)== = (a 1) V [g(
(13)
21

---

may be The statement then follows by substituting Vo (oB.))o(s..)l(1 0)] with the corresponding score estimator given by Equation (12).
C.3 Proof of Lemma
Forward KI, divergence. The gradient of PRL[Palle is straightforwardly obtained through the application of Leibniz's 's rule for integrals,
VoDKı -B
V (ogpp(r)s.;;)) 0)]
since the averaging distribution Pn do not depend on the varying parameters However, as we compute Monte Carlo averages over samples of Pr. apply importance reweighting scheme [63, Chapter 9] to the previous expectation infer that, up to a positive multiplicative constant,
pB ()))(()) V (oggpp(()ss::))
VoDRi[Pa|IPp] E -Ero)((s.r)
(T((
with S denoting equality up to positive multiplicative constant. We emphasize that most modern stochastic gradient methods for optimization, such as Adam [41] and RMSProp [28], remain un- changed when multiply the estimated gradients by fixed quantity; thus, we may harmlessly compute gradients up to multiplicative constants.
Reverse KL divergence. We verified in Equation (9) that
VoDkr(Pr|1Pp = PA (Hor) log pr(T(80) Voso(T)) PB(T)
Since (r) = ()(())))()))/ and Pp Voso(T) the quantity VoD VoDxz rewritten as
log PF(T(so) VosS(T)) (T(T)
VoDKLIPFIPPp = EyNPr(
+ EETPP
[log 2)sesp(T)
log Pr(T(sa) (T(T)=)r(z)
Vose(T)
in which x is the terminal state corresponding to the trajectory т. Thus proving the statement in Lemma 2.
C.4 Proof of Proposition'
We will derive expression for the optimal baseline of a vector-valued control variate. For this, let be the averaged function and g: T ++ g(T) be the control variate. Assume, without loss generality, that [g] 0 for the averaging distribution over the space of trajectories. In this case, the optimal baseline for the control variate a* found by
arg min Tr (Cov~~xIf() -((())))))())) aE3
(14)
Thus,
a" arg min Tr (-2a 2a Cov=[f(T), + +oo==<
AER
which is convex optimization problem solved by
at Tr [Cov)
Tr ([
Tr E. E. (T))(()))
Tr E, (9())
(15)
Ex [Tr (f - E. (f)) g())
E,, (T (
[ - (x))) ((({)
[x[o(T)T
which we used the circular property of the trace. This equation exactly matches the result in Propo- sition 2. In practice, we use g(T) = Va log (F(() for both the reverse KL- and a-divergences, render- ing a baseline a² that depends non-linearly on the sample gradients and hence difficult to compute in GPU-powered autodiff framework efficiently. We thus use Equation (6) to estimate a".
22

---

Segaanezee
Crniin Mistrees
Penare dieritation
70
2000
200
0000
00
100
200
2003
1000
Epochs
Figure 7: Learning curves for GFlowNet trained by minimizing the TB loss. The curves' smoothness highlights the low variance of the optimization steps incurred by the stochastic gradients of CTB. which do not use score function estimator.
2=5.
Target
Figure 9: Additional illustration of the effect of a when learning GFlowNets by minimizing the Renyi-a divergence in the hypergrid environment. For such a sparse target distribution, a large and negative value of (left) is beneficial to ensure that all modes are appropriately covered by the learned distribution. contrast, the mode-seeking behavior induced by large value of a entails the collapse of the model in single mode (right).
D Additional experiments
Gradient variance for flow-network-baser objectives. Figure 7 shows the learning curve for the TB loss in each of the generative tasks. Notoriously, it underlines the low variance of the optimization steps - which, contrarily to their divergence-basee counterparts, do not rely on a score function estimator and suggests that the design of control variates for estimating the gradients of these objectives would not be significantly helpful. Also, the gradient of Lr depends non- non-linearly on the score function log pF and, consequently, it is unclear how implement computationally efficient variance reduction techniques in this case.
Forward KI. for sequence generation. Figure 3 shows that alternative approaches in terms convergence speed outperformed GFlow Fet trained to minimize the forward KI., One possible cause of this underperformance is the high variance induced by the underlying importance sampling estimator. To verify this, re-run the corresponding experiments, increasing the size of the batch trajectories for the forward KL estimator 1024. Figure 8 presents the experiment's results, with an increased batch size corresponding to estimator of smaller variance that accelerates the GFlowNet's training convergence. More broadly, this suggests that the design GFlowfet-speciffc variance reduction techniques, which leave future endeavors, may further improve this family of models.
Sequences
2
Rev, KL
KL
Renyl-o
5
Tsailis-a
TB
0
2000
Epochs
Figure 8: Results for sequence generation with larger batches.
Influence of the learning of GelowNets. As mentioned earlier, divergence-based measures fall short compared to their balance-based counterparts for the hypergrid navigation task. For this extremely sparse problem, the benefits from off-policy exploration during training seem to supersede the statistical efficiency enacted by the minimization divergences, which fail to properly cover the high-probability regions of the target distribution. In this scenario, Figure 9 suggests that this issue can be mildly circumvented by choosing sufficiently negative a for the Renyi-a divergences, thereby imposing move-cov behavior to the learned model. However, these results should be substan- tiated by further experimental analysis to be conclusive. Currently, we would suggest a practitioner to stick to the balance-basse objectives when dealing with very sparse target distributions.
23

---

E Experimental details
The following paragraph provides further implementation details. Regarding open access to the code, we will make the code public upon acceptance.
Shared configurations. For every generative task, we used the Adam optimizer [41] carry out the stochastic optimization, employing learning rate of 10-1 for log Zo when minimizing Lr B and 10-3 for the remaining parameters, following previous works [48, 55, 56, 66]. We polynomially annealed the learning rate towards O along training, similarly to [86]. Also, we use LeakyReLLU [98] as the non-linear activation function of all implemented neural networks.
Set generation. We implement an MLP of 2 64-dimensional layers to parameterize the policy's 's logits log (s, -). We train the model for 512 epochs with a batch of 128 trajectories for estimating the gradients. Also, we let D = 32 and N = 16 be the source's 's and set's sizes, respectively.
Autoregressive sequence generation. We parameterize the logits of the forward policy with MLP of 2 64-dimensio layers; we pad the sequences to account for their variable sizes. We respectively consider D = 8 and N = 6 for the source's 's and sequence's sizes. To approximate the gradients, rely on a batch of 128 sequences.
Bayesian phylogenetic inference. We parameterize the logits of the forward policy with 2-layer GIN [99] with 64-dimensional latent embedding, which linearly projected to log pr. Moreover, simulated the JC69 model [37] to obtain 25-sized sequences of nucleotides for each of the 7 observed species, setting 0.3 for the instantaneous mutation rate; see [101] for an introduction to computational phylogenetics and molecular evolution. To estimate the gradients, we relied on batches of 64 trajectories.
Hypergrid navigation. We consider H = 12 for Figure and Figure 4 and H = 9 for Figure 9. In both cases, d and R., 10 10-3, 33 see [55, Section 5.1]. To parameterize the policy, we used a 2-layer 256-dimensional MLP with ReLU activations. We trained the models for 10000 epochs.
Bayesian structure learning. We simulated 100-sized 5-variable data X from randomly parameterized linear Gaussian structural model based on a fixed Bayesian network. We implemented a 2-layer 256-dimensional MLP with ReLU activations for the policy network, which received the flattened adjacency matrix of the current state as input. Training lasted for 4000 epochs.
Mixture of Gaussian distributions. We consider mixture of 9 2-dimensional Gaussian distributions centered at jiy = (f.j) for I í.j < 2, each of which having an isotropic variance 10 see Figure 1. We use MLP of 2 64 64-dimensional layers to parameterize the forward policy.
Banana-shaped distribution. The model is specified by Equation (8). We also consider an MLP of 2 64-dimensional layers to parameterize the forward policy.
24

---

NeurIPS Paper Checklist
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatiio right after your answer (even for NA).
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: the proofs for the main claims in Proposition 1, Lemma 1, Proposition Lemma are presented in Appendix C, and practical training of GFN empirical validation in Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should elearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The middle paragraph in Section 6 discusses limitations, including the potential impact of different a values and the need for further exploration this area.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiscless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
25

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The statements of the propositions and lemma include the assumptions, as well in the notations and definitions presented in Section 4, and proofs provided in Appendix C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Details provided in Section 5, Appendix B and Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
26

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Provided in zip file during the review period, with the plan for public release once the paper is made public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc) gww.dde//oeeeuunnnineeee iwtpbbmiissiooooooccyy.o. onP olicy) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
ttips.cc/pbblice /owpe/udddee/e/eeediiiieieeeeeee
Cwt///oooeSSeeeeeeeeeeeee www.isssiciiiiiyiioccoomcm for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
27

---

Justification:
Section 5 in the main text presents information about the experiments with full details provided in Appendix B and Appendix and supplemented zip-file with the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Plots count on error bars and tables count on standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: First paragraph of Appendix E.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
28

---

The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethies
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: htt///nerrppppp htc/ppullic/thhccscieeeeeeeeeeee
Answer: [Yes]
Justification:
Our submission follow the NeurIPS ethical guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We provide perspective on broader impacts in Section 6, but do not foresee any direct negative societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
29

---

Justification:
We do not foresee any direct risk stemming from our work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: All code was made by the authors
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.coo/c/cooocoo datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
30

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
31


Reviews:
[{"review_id": "T16Kjp6jOa", "reviewer": "Reviewer_NozV", "summary": "This paper investigates divergence measures as learning objectives for Generative Flow Networks, which are amortized inference models designed for sampling from unnormalized distributions over composable objects. The authors review four divergence measures - Renyi-\\alpha, Tsallis-\\alpha, reverse and forward Kullback-Leibler divergences, and design estimators for their stochastic gradients in the context of training enerative Flow Networks. The authors verify that minimizing these divergences yields correct and empirically effective training schemes on several toy environment, and show that it often lead to faster convergence than previously proposed optimization methods. The authors also design control variates based on REINFORCE and score-matching estimators to reduce gradient variance.", "questions": "1. How do the proposed divergence-based objectives perform on more complex and high-dimensional datasets commonly used in fields such as drug discovery and natural language processing?\n\n2. Can the control variate techniques be extended to other GFlowNet training objectives beyond the divergence measures considered in this paper?\n\n3. How does the choice of the \\alpha parameter affect the learned GFlowNet's ability to capture multi-modal target distributions or discover diverse high-quality samples in combinatorial optimization tasks?\n\n4. Are there any theoretical guarantees or bounds on the sample complexity or convergence rates when using the proposed divergence-based objectives for training GFlowNets?", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper provides evaluation of different divergence measures as learning objectives for GFlowNets, showcasing their effectiveness in improving training convergence in several toy environments.\n\n- The authors develop control variates for reducing the variance of gradient estimates.", "weaknesses": "1. While the paper provides empirical evidence for the effectiveness of divergence-based objectives, a more extensive comparison with traditional GFlowNet training methods across a wider range of datasets and applications would strengthen the claims.\n\n2. The choice of the \\alpha parameter in Renyi-\\alpha and Tsallis-\\alpha divergences is not extensively explored. More insights into the impact of \\alpha on the learning dynamics and guidance on selecting an appropriate value would be beneficial.\n\n3. The computational overhead introduced by the control variates and their impact on training time is not explicitly discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We hope our answers address your concerns and elevate your appraisal of our work. Otherwise, we would be happy to engage further.\n\n> … a more extensive comparison across a wider range of datasets and applications would strengthen the claims.\n\nWe have included SubTB [1], DB [2], and VarGrad [3,4] losses as baselines and included the hypergrid [5] and causal DAG environments [6]. With this, our experimental campaign is broader than most works in the GFlowNet literature, including six different environments with continuous and discrete target distributions. In contrast, most works in the GFlowNet literature use three to four environments [1, 2, 4, 5, 6, 7, 8], typically using discrete targets and comparing against a small set of learning objectives. \n\nGiven our comprehensive experimental suite, we can confidently conclude that divergence-based objectives are beneficial for training GFlowNets as long as they are equipped with appropriate control variates, contrary to the belief established by the results in [18]. \n\n> on choosing $\\alpha$ and its effect on GFlowNet's ability to capture modes\n\nThank you for the opportunity to improve our discussion on the role of $\\alpha$. To the best of our knowledge, delineating specific guidelines for choosing optimal $\\alpha$ for VI is still an open problem [11, 12]. Our submission (Lines 164-186 and Figure 1) discusses how $\\alpha$ modulates mode-seeking vs. mass-covering behavior. This is a necessary starting point practitioners considering the choice of $\\alpha$. \n\nTo further illustrate the effect of varying $\\alpha$, we executed a series of experiments for the hypergrid task – please refer to Fig 3 of the rebuttal PDF. Overall, our analyses indicate that, for sparse distributions, large negative values of $\\alpha$ perform better – on par with the mass-covering effect of the corresponding divergence measure. In addition, we also ran experiments for the set generation, sequence design, and BPI tasks with varying values of $\\alpha$, observing similar results.\n\nWe will include all additional results in the revised manuscript.\n\n> computational cost of control variates\n\nIn practice, the computational overhead of our variance reduction techniques is negligible. We report in Table 1 the runtime (in minutes) of the training process averaged over all environments considered in the experiments, a total of 120 runs, for each learning objective. The avg runtime confirms the small overhead of our variance reduction method. Furthermore, the control variate significantly speeds up the convergence, as shown in Fig. 5 of our manuscript (Section 5.3).\n\nTable 1: Runtime in minutes, avg over runs and environments. \n| criterion        |     avg |\n|:-----------------|---------:|\n| KL               | 9.5| \n| Renyi-$\\alpha$   | 9.4| \n| Rev. KL          | 10.4| \n| Tsallis-$\\alpha$ | 10.5| \n| TB               | 9.2|  \n  \n> Performance in complex and high-dimensional tasks\n\nThanks for the question. We would like to clarify that our adopted environments represent both realistic and prototypical for GFlowNets. For example, the phylogenetic tree inference [7] and sequence generation [8], presented in Figure 3, are real-world tasks with papers assessing the effectiveness of GFlowNets in solving them [7, 8]. Also, we estimate  ~$10^{7}$  possible final objects in the set generation and sequence design tasks, highlighting their realistic scales.\n\nWe have now included hypergrid [1] and Causal DAG [2] as novel tasks; and DB, SubTB, and VarGrad as baselines. Strikingly, Renyi, Tsallis, and KL perform on par with or better than balance-based losses in most scenarios (Figs 1 and 2, rebuttal PDF).\n\n> Can the control variate techniques be extended to [balance-based] GFlowNet training objectives [...]?\n\nIn principle, yes. However, it is unclear how to devise efficient CVs for balance-based objectives. \n\nFirstly, conventional objectives rely on off-policy sampling, and it is mostly unclear how to define a control variate that (i) has zero expectation under a chosen policy and (ii) is correlated to the objective's gradient. Secondly, as shown in Fig 6 (supplement), the smoothness of the learning curve for TB suggests that gradient variance is not an issue for balance-based objectives. Notably, it is well-known that the high variance for divergence-based objective stems from the score-function estimator, $\\mathbb{E}\\_{\\tau \\sim p_{F}}[\\nabla \\log p_{F}(\\tau)]$, which is not present in the gradient of balance-based objectives such as TB.\n\n> Theoretical guarantees on the training convergence\n\nDeveloping convergence rate analysis for GFlowNets remains an open problem. Many works focus on designing sample-efficient learning objectives for GFlowNets, but none provide convergence rate analyses.\n\nAlso, establishing convergence rates for VI for generic distribution classes is an open problem. Initial works [17] assumed strong convexity in a lower-bound problem relevant to GANs, obtaining geometric convergence rates. More recently, [16] obtained guarantees for BBVI and Gaussian approximations. Adapting these results to GFlowNet is a fruitful line of work, meriting an investigation of its own.\n\n[1] Learning GFlowNets from partial episodes for improved convergence and stability. ICML 2023\n\n[2] GFlowNet Foundations. JMLR 2023\n\n[3] VarGrad: A Low-Variance Gradient Estimator for VI. NeurIPS 2020\n\n[4] Robust Scheduling with GFlowNets. ICLR 2023\n\n[5] TB: Improved credit assignment in GFlowNets. NeurIPS 2022 \n\n[6] Bayesian structure learning with GFlowNets. UAI 2022\n\n[7] PhyloGFN: Phylogenetic inference with GFlowNets. ICLR 2024\n\n[5] Biological sequence design with GFlowNets. ICML 2022\n\n[12] Rényi Divergence VI. In NeurIPS 2016\n\n[13] Meta-learning divergences for VI. In AISTATS 2021\n\n[16] Provable convergence guarantees for BBVI. NeurIPS 2024\n\n[17] g-GAN: Training generative neural samplers using variational divergence minimization. NeurIPS 2016\n\n[18] GFlowNets and VI.  ICLR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CF32LVjvFh", "reviewer": "Reviewer_HRNm", "summary": "This paper investigates the potential of using a variety of divergence measures directly as training losses for GFlowNets, relying on many of the connections made between GFlowNets and variational inference. Training GFlowNets essentially consists in enforcing balance/flow-matching conditions between a proposal and a target distribution given some data. Given the links between GFlowNets and variational inference, training the GFlowNet to directly minimize the KL divergence was observed to result in biased and high-variance estimators in general. This paper aims to verify the latter claim, for different divergence measures (KL, reverse KL, Tsallis-\\$\\alpha\\$, Renyi-\\$\\alpha\\$). The paper alleviates the afore-mentioned limits by proposing low-variance gradient estimates to that of the latter quantities through control variates. Finally, the paper verifies the proposed methods experimentally through an extensive set of experiments.", "questions": "- How is the diversity of samples impacted? \n- There should be experiments that show that the different behaviors of each method with respected to the number of discovered modes throughout training (that should be compared to TB too!). For instance, similarly to Figure 4 in (Malkin et al., https://arxiv.org/pdf/2201.13259).", "limitations": "Limitations adequately addressed throughout the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper widens the scope of the theoretical links between GFlowNets and variational inference beyond the assumption of finitely supported measures. \n- The authors alleviate the high variance of divergence measures' gradients in practice, relying on control variates. \n- The paper is very well written and presented and is easy to follow. \n- The experimental setup is exhaustive, and shows the effect of each of the proposed design choices.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for valuable suggestion and review. We did our best to address each of your questions, and extended our experiments following your suggestion. Please let us know if you have other questions or require further clarification.\n\n> How is the diversity of samples impacted?\n\nConsidering that the *diversity of the samples* could be approached from different perspectives, we made our best effort to address your question in a broad sense. \n\n**Support coverage: divergence measures versus TB.** Overall, we observe that our divergence-based learning procedures yield better support coverage than the TB, consequently capturing shapes more accurately. Figure 4 in the manuscript illustrates this phenomenon for a banana-shaped target distribution. \n\n**Number of visited modes.** Compared to balance-based objectives, we observe that our divergence-based training leads to faster increase in NoM and average reward of the top-K samples during the training process in the majority of environments — Figure 2 (rebuttal PDF). This metrics can be interpreted as indicative of diversity of the high-reward samples. \n\n**Impact of $\\alpha$ on sample diversity.** In general, $\\alpha$ allows us to modulate between mode-seeking and mass-covering behaviors, directly impacting the sample diversity. Low $\\alpha$ leads to capturing more modes, while exceedingly high $\\alpha$ may cause mode collapse — see, e.g., Figure 1 (manuscript) and Figure 3 (rebuttal PDF).\n\n> There should be experiments that show that the different behaviors of each method with respected to the number of discovered modes throughout training (that should be compared to TB too!). For instance, similarly to Figure 4 in (Malkin et al., https://arxiv.org/pdf/2201.13259).\n\nThank you for the question and suggestion. We have included an analysis of the number of modes (NoM) discovered as a function of time in Figure 2 (rebuttal PDF). Overall, except for the Hypergrid environment, the divergence measures (Rev. KL, KL, Renyi-α and Tsallis-α, w/ α=0.5) have a higher NoM visited earlier in the training process than the balance-based losses (TB, DB, SubTB, VarGrad). Furthermore, their NoM learning curves display a faster rate of increase in the Sets and Sequences environment. For completeness, we include the average reward for the K highest scoring samples (top-K), showing similar results as the NoM. \n\nThe distinct behavior for Renyi-α and Tsallis-α in the hypergrid environment could be explained by the results in Figure 3 (rebuttal PDF), since α=0.5 leads to mode collapse.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WEDq8eXD79", "reviewer": "Reviewer_3sBX", "summary": "GFlowNets are a probabilistic framework for training amortized samplers for high-dimensional compositional spaces. The samplers are typically trained using local consistency objectives which are squared log losses. This paper examines alternatives to these obejctives in the form of general statistical f-divergence measures. The authors consider forward and reverse KL, Renyi-$\\alpha$ and Tsallis-$\\alpha$ divergences. The authors derive gradients for the divergences in the case of a fixed $p_B$ and on-policy samples. These gradient computations rely on REINFORCE-style estimators and consequently can suffer from high-variance gradient estimates which affect the optimization procedure. For variance reduction, the authors derive control variates for their gradient estimators. Through a series of experiments on a variety of tasks, the authors demonstrate the effectiveness of these divergences for training GFlowNets.", "questions": "In addition to the points in Weaknesses:\n\n* L135 says TB requires estimating $Z$ but KL doesn't but I am not sure that is correct? Since even in the KL you need the normalizing constant in the $P_B$ term.\n\n* L146: [1] would be a more appropriate reference here I think?\n\n* L157: Missing reference to [2]\n\n* What is the computational performance (in terms of runtime) of the proposed objectives relative to TB? \n\n\n[1] Rubinstein, R. Y. (1981). Simulation and the Monte Carlo Method. In Wiley Series in Probability and Statistics. Wiley. https://doi.org/10.1002/9780470316511\n\n[2] f-Divergence Variational Inference. Neng Wan, Dapeng Li, Naira Hovakimyan. https://arxiv.org/abs/2009.13093.", "limitations": "The authors do not explicitly address the limitations of their approach (discussed in the weaknesses) section, though the assumptions are mentioned briefly in Section 2 and 3. The authors only mention the choice of $\\alpha$ as a limitation.\n\nThere is also no discussion of broader impacts (the reference in the checklist is broken too).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper is quite well written and clear. The authors are thorough and clear in introducing the central ideas and provide sufficient details making it easy to follow.\n* The paper studies the important problem of finding the \"right\" learning objective in the context of training GFlowNets. Following a long line of work in VI, the authors leverage statistical divergences and propose optimizing them directly to learn the GFN sampler.\n* Additionally, the authors improve upon prior work drawing a connection between GFNs and VI by proposing principled control variates to reduce the variance in the gradient estimates for the divergences from the REINFORCE estimators. The CVs seem to have a significant effect on the performance.\n* The empirical analysis covers a variety of problems including continuous and discrete spaces as wells as general DAGs and tree spaces in the case of discrete spaces. \n* The authors also include code with the submission aiding reproducibility.", "weaknesses": "* The paper considers the on-policy setting for training GFlowNets and the proposed learning objectives based on the divergences assume on-policy smaples. However, existing flow-based objectives are all off-policy, and the advantage of GFlowNets on a lot of tasks (specifically challenging tasks with multi-modal target distributions) comes from the ability to train on off-policy trajectories (e.g. replay buffer[1] to local search [2]). So while the proposed learning objectives empirically perform better than TB on tasks where on-policy sampling is enough, it lacks the flexibility of accomodating off-policy training.\n* The authors also assume that the backward policy $P_B$ is fixed (L153). While this is true in some cases, learning $P_B$ results in significant improvements to the learned sampler[3,4]. As far as I can tell, modifying the proposed objectives to accomodate learning the P_B is non-trivial.\n* I appreciate the diversity of problems studied by the authors in their experiments, but there are some gaps in the empirical analysis. Specifically, the authors only include a TB baseline and not other objectives such as SubTB, DB which can perform better than TB (and have better training stability) in some cases. Additionally, the paper also does not include the VarGrad-style objective [5] which does away the need for estimating $Z$ in TB.\n* Moreover, the tasks consider relatively small tasks so it is not clear how scalable the proposed objectives are. \n\n\n[1] Towards Understanding and Improving GFlowNet Training. Max W. Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, Tommaso Biancalani. ICML 2023. \n\n[2] Local Search GFlowNets. Minsu Kim, Taeyoung Yun, Emmanuel Bengio, Dinghuai Zhang, Yoshua Bengio, Sungsoo Ahn, Jinkyoo Park. ICLR 2024. \n\n[3] Trajectory balance: Improved credit assignment in GFlowNets. Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, Yoshua Bengio. NeurIPS 2022. \n\n[4] A theory of continuous generative flow networks. Salem Lahlou, Tristan Deleu, Pablo Lemos, Dinghuai Zhang, Alexandra Volokhova, Alex Hernández-García, Léna Néhale Ezzine, Yoshua Bengio, Nikolay Malkin. ICML 2023.\n\n[5] Robust Scheduling with GFlowNets. David W. Zhang, Corrado Rainone, Markus Peschl, Roberto Bondesan. ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the suggestions and for appreciating our work. We did our best to address each of your concerns below. Please let us know if you have other questions or require further clarification.\n\n> The paper considers the on-policy setting for training GFlowNets and the proposed learning objectives based on the divergences assume on-policy smaples. However, existing flow-based objectives are all off-policy, and the advantage of GFlowNets on a lot of tasks [...] comes from the ability to train on off-policy trajectories (e.g. replay buffer [1] to local search [2]). So while the proposed learning objectives empirically perform better than TB on tasks where on-policy sampling is enough, it lacks the flexibility of accomodating off-policy training.\n\nIndeed, many heuristics for off-policy learning of GFlowNets cannot be adapted to the context of divergence-based training.\n\nNonetheless, please note that KL-, Renyi-, and Tsallis- divergences are also amenable to a degree of off-policy learning by implementing an importance sampling estimator — as long as the sampling policy can be directly evaluated at each individual trajectory. This is the case for, e.g., the widely-used $\\epsilon$-greedy policy.\n\nFundamentally, we see this as an instantiation of Wolpert’s No Free Lunch theorem: while, as we show, the minimization of statistical divergences is more sample-efficient and frequently leads to faster training convergence for GFlowNets, implementing these objectives constrains the user to the adoption of tractable off-policy sampling schemes.\n\nWe will include this discussion in the revised manuscript.\n\n> The authors also assume that the backward policy is fixed (L153). While this is true in some cases, learning PB results in significant improvements to the learned sampler [3,4]. As far as I can tell, modifying the proposed objectives to accomodate learning the P_B is non-trivial.\n\nThank you for the thought-provoking remark. In principle, learning $\\log p_{B}$ is quite straightforward; the objective\n\n$$\n\\min \\mathcal{D}(p_{B}, p_{F})\n$$\n\ncan be jointly minimized in both $p_{F}$ and $p_{B}$ for any divergence measure $\\mathcal{D}$ in a VAE-style learning. However, implementing a learnable $p_{B}$ incurs a non-trivial computational overhead due to the additional number of backward passes for reverse-mode autodifferentiation. To illustrate this, we will run additional experiments for divergence-based training of GFlowNets with a learnable $p_{B}$ and report the results during the discussion period.\n\n> I appreciate the diversity of problems studied by the authors in their experiments, but there are some gaps in the empirical analysis. Specifically, the authors only include a TB baseline and not other objectives such as SubTB, DB which can perform better than TB (and have better training stability) in some cases. Additionally, the paper also does not include the VarGrad-style objective [5] which does away the need for estimating Z in TB.\n\nThank you for your compliment. We have included DB, SubTB, and VarGrad as additional baselines for our experiments in Figures 1 and 2 of the attached PDF, in addition to two novel generative tasks.\n\nRemarkably, with the exception of the extremely sparse hypergrid environment, divergence-minimization algorithms lead to the fastest convergence rate among the tested learning objectives. We also note that, for the very sparse and hard-to-explore hypergrid, off-policy training is necessary and (as we discussed earlier) purely on-policy-based methods should be avoided. Nonetheless, for these problems, our empirical analysis shows that the mode-covering behavior of $\\alpha$-divergences with large and negative $\\alpha$ is very beneficial for speeding up training convergence (please refer to Figure 3 of the attached PDF). We will include these experiments in the revised manuscript.\n\n>  Moreover, the tasks consider relatively small tasks so it is not clear how scalable the proposed objectives are.\n\nThanks for bringing up the discussion. We emphasize that the computation overhead incurred by the proposed gradient estimation techniques is negligible. In this sense, the divergence-based learning objectives are as scalable as their balance-based counterparts; please refer to Table 1 below. Results represent averages of 24 runs per criterion.\n\nTable 1: Runtime in minutes, avg over runs and environments.\n| criterion  | avg |\n|:-----------------|---------:|\n| KL         |9.5| \n| Renyi-$\\alpha$   | 9.4| \n| Rev. KL          | 10.4| \n| Tsallis-$\\alpha$ | 10.5| \n| TB               | 9.2|  \n  \n\n> L135 says TB requires estimating Z\n\nIndeed, the mathematical definition of TB and KL depends on the constant Z. Nevertheless, it is correct that Z is not needed in the context of gradient-based optimization of the KL. To see this, please note that we may write the reverse KL-divergence as\n\n$$ \n\\mathcal{D}\\_{KL}[p\\_{F} || p\\_{B}] = \\mathbb{E}\\_{\\tau \\sim p\\_{F}}[\\log p\\_{F}(\\tau) - \\log p\\_{B}(\\tau | x) R(x) / Z] = \\mathbb{E}_{\\tau \\sim p\\_{F}}[\\log p\\_{F}(\\tau) - \\log p\\_{B}(\\tau | x) R(x)] + \\log Z.  \n$$ \n\nConsequently, when taking gradients with respect to the parameters of  $p_{F}$, the terms corresponding to $\\log Z$ in the right-hand side of the equation above vanish, as it does not depend on $p_{F}$. In particular, $\\log Z$ does not interfere with the problem of minimizing $\\mathcal{D}\\_{KL}[p_{F} || p\\_{B}]$ and can be ignored. A similar argument holds for the forward KL-, Renyi-, and Tsallis divergences. We will update the revised manuscript to clarify this point. \n\n> L146: [1] would be a more appropriate reference here I think?; L157: Missing reference to [2]\n\nWe agree! These important references will be incorporated into the revised manuscript. \n\n> What is the computational performance (in terms of runtime) of the proposed objectives relative to TB?\n\nPlease refer to Table 1 above. As we remarked earlier, our approach adds a negligible computational overhead to the training of GFlowNets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OaFUM0uRt4", "reviewer": "Reviewer_Gn1e", "summary": "This paper investigates alternative training methods for Generative Flow Networks (GFlowNets) by evaluating various divergence measures, including Renyi-α, Tsallis-α, reverse, and forward Kullback-Leibler (KL) divergences. Traditional methods focusing on minimizing log-squared differences are shown to lead to biased and high-variance estimators. The authors propose efficient estimators for the stochastic gradients of these divergences and introduce control variates to reduce gradient variance. Empirical results across diverse tasks demonstrate that minimizing these divergence measures significantly accelerates training convergence and enhances stability compared to traditional methods. The paper establishes theoretical connections between GFlowNets and variational inference, extending these insights to arbitrary topological spaces, and highlights the practical effectiveness of control variates in improving training efficiency.", "questions": "See **Weaknesses**.", "limitations": "See **Weaknesses**.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- **Comprehensive Evaluation**: The paper thoroughly evaluates multiple divergence measures and their impact on GFlowNet training.\n- **Theoretical Insights**: Establishing theoretical connections between GFlowNets and VI broadens the understanding of these models. \n- **Practical Contributions**: The design of control variates to reduce gradient variance is a significant practical contribution that can be applied in various optimization scenarios.\n- **Variance Reduction Techniques**: The introduction of variance reduction techniques, including control variates and leave-one-out estimators, effectively addresses the high variance issue in gradient estimates, enhancing the learning stability and efficiency.", "weaknesses": "- **Theory** The theoretical contribution builds upon previous works [4,5]. However, I do not think the theory itself has enough contribution. Because the Measurable pointed DAG is from previous work [5] and the main theoretical claim, Proposition 1, has limited novelty compared with Proposition 1 in [4]. Therefore, unlike previous works [4,5], they have their novel theoretical contributions and synthetic experiments would suffice. Thus, for this paper, stronger experiments are required, as will be discussed in the next.  \n- **Experiments - Part 1** This paper does not contain enough real-world tasks, such as fragment-based molecule generation [1], graph combinatorial optimization [2], and RNA sequence generation [3] to illustrate its main contribution. These are standard tasks in evaluating GFN performances and are necessary to include. For the only real-world task in the paper, i.e., the BPI task, this paper admits that \"not statistically significant\" in line 334. Therefore, it is unclear whether the proposed methods to use other divergence measures will be meaningful in real-world scenarios. \n- **Experiments - Part 2** For the synthetic tasks, the plots in Figure 3 are also missing some lines. For example, why are there only two lines in the **Sets** plots? Also, there is no single best loss function that is uniformly better than the KL baseline. Therefore, additional divergence might seem unnecessary. There should be stronger evidence to support the use of other measures. Or, the authors can provide a guideline on how to choose the best measures given prior information about the tasks. \n\n- I would raise my scores if additional experiments are included and the proposed methods are indeed beneficial in more complex tasks. \n\n\n[1] Jain, Moksh, et al. \"Biological sequence design with gflownets.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Zhang, Dinghuai, et al. \"Let the flows tell: Solving graph combinatorial problems with GFlowNets.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[3] Kim, Minsu, et al. \"Local search gflownets.\" arXiv preprint arXiv:2310.02710 (2023).\n\n[4] Malkin, Nikolay, et al. \"GFlowNets and variational inference.\" arXiv preprint arXiv:2210.00580 (2022).\n\n[5] Lahlou, Salem, et al. \"A theory of continuous generative flow networks.\" International Conference on Machine Learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback and suggestions. Below, we address the specific weaknesses and questions.  We hope our clarifications and additional experiments address your concerns and elevate your appraisal of our work.\n. \n> Because the Measurable pointed DAG is from previous work [5] and the main theoretical claim, Proposition 1, has limited novelty compared with Proposition 1 in [4]. \n\nWe would like to emphasize Proposition 1 is not the main contribution of our work. In fact, note this proposition is part of our background section. While perhaps a simple extension of previous results, Proposition 1 is a necessary formalism to our developments in the following sections, as we consider discrete as well as continuous state spaces in our work. \n\n> This paper does not contain enough real-world tasks, such as fragment-based molecule generation [1], graph combinatorial optimization [2], and RNA sequence generation [3] to illustrate its main contribution. These are standard tasks in evaluating GFN performances and are necessary to include. For the only real-world task in the paper, i.e., the BPI task, this paper admits that \"not statistically significant\" in line 334. Therefore, it is unclear whether the proposed methods to use other divergence measures will be meaningful in real-world scenarios. \n\nWe would like to emphasize that, before our work, divergence measures were perceived as inferior choices to traditional GFlowNet criteria (e.g., TB) as suggested by Malkin et al. [4]. The main reason for our success is the design of appropriate control variates (CVs) for variance reduction, which preserve the unbiasedness of gradient estimates. It is important to note that Malkin et al. [4]'s gradient estimators differ from ours — and are not bias-free, as we discuss in lines 272-278 of our manuscript.\n\nTo strengthen our empirical claims, we have included additional baselines (DB, VarInf and SubTB) [8, 9, 10, 11] and environments (hypergrid and causal structure learning) [4, 6, 7] — see Figure 2 in the rebuttal PDF. It is worth mentioning that causal structure learning is one of the prime applications of GFlowNets [6, 7] and can be seen as an instance of combinatorial optimization. Our results reinforce that (given appropriate control variables) divergence-based objectives perform similarly to or better than balance-based objectives.\n\nPlease let us know if this addresses your issue. If you believe considering additional environments is necessary to verify our claim, we would gladly run more experiments in the discussion period.\n\n> For the synthetic tasks, the plots in Figure 3 are also missing some lines. For example, why are there only two lines in the Sets plots? Also, there is no single best loss function that is uniformly better than the KL baseline. Therefore, additional divergence might seem unnecessary. There should be stronger evidence to support the use of other measures. Or, the authors can provide a guideline on how to choose the best measures given prior information about the tasks. \n\nWe apologize for the oversight in Figure 3. The missing lines in the plot for the set generation task were due to the overlap of learning curves for the divergence-based objectives. We will include dashed lines to improve the visualization.\n\nWe highlight that we are considering four divergence measures: Forward-KL, Reverse-KL, $\\alpha$-Renyi, and $\\alpha$-Tsallis, obtaining for each a variance-reduced gradient estimators for GFlowNet training. These estimators are distinct from the proposal in Malkin et al. [4]. \n\nWe considered the TB loss as a baseline. We have also run comparisons against the DB, SubTB, and VarGrad losses for the rebuttal (Figs 1 and 2, rebuttal PDF). Compared to the baseline losses, we can always find a divergence loss with better performance. On the other hand, no single divergence has uniform dominance over the others (Reverse-KL and Forward-KL are distinct measures). \nOverall, the experimental evidence favors the class of divergence measures. One practical advantage of Renyi and Tsallis is to represent intermediate measures between the extremes of mode-seeking and mass-covering of Forward-KL and Reverse-KL. We have added experiments exploring the effect of $\\alpha$ for the distribution of Hypergrid (Fig 3, rebuttal PDF). \n\nRegarding how to choose the most appropriate divergence, we agree that clear guidelines would be useful. However, developing these guidelines an open problem in the VI literature [12, 13].\n\n[1] Biological sequence design with gflownets. ICML 2022\n\n[2] Let the flows tell: Solving graph combinatorial problems with GFlowNets. NeurIPS 2024\n\n[3] Local search gflownets. arXiv 2023\n\n[4] GFlowNets and variational inference.  ICLR 2023\n\n[5] A theory of continuous generative flow networks. ICML 2023\n\n[6] Bayesian structure learning with generative flow networks. In UAI, 2022\n\n[7] Joint Bayesian Inference of Graphical Structure and Parameters with a GFlowNet. NeurIPS 2023\n\n[8] Learning GFlowNets from partial episodes for improved convergence and stability. ICML 2023\n\n[9] GFlowNet Foundations. JMLR 2023\n\n[10] VarGrad: A Low-Variance Gradient Estimator for VI. NeurIPS 2020\n\n[11] Robust Scheduling with GFlowNets. ICLR 2023\n\n[12]  Divergence measures and message passing. 2005\n\n[13] Rényi Divergence Variational Inference. NeurIPS 2016", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers and AC, 

We appreciate that reviewers evaluation of our work as both theoretically principled [Gn1e, HRNm]  and empirically well-grounded [Gn1e, 3sBX, NozV], expanding the link between VI and GFlowNets [Gn1e, 3sBX, HRNm], with practical contribution of effective control variates for variance reduction [Gn1e, 3sBX, HRNm, NozV], presented with clarity [3sBX, HRNm], and with reproducible results [3sBX] supported by a comprehensive range of experiments [Gn1e, 3sBX, HRNm]. 

We present a summary of the central points in the discussion and how they were addressed.

1. Reviewers NozV, 3sBx, Gn1e, and HRNm suggested including additional benchmark tasks, baseline objectives, and evaluation metrics. 

    a. Firstly, we included the hypergrid and DAG environments in our experimental suite. 

    b. Secondly, we added SubTB, VarGrad, and DB among our baselines. 

    c. Thirdly, we measured the number of modes and the average reward of the highest $K$ scoring samples during training (top-$K$). 

    d. Our results confirm that, for the majority of problems, divergence-based objectives lead to faster training convergence than balance-based ones. To the best of our knowledge, this is the **broadest experimental evaluation of GFlowNet objectives in the literature**. 

2. Reviewers NozV and 3sBx requested a runtime analysis to quantify the computational overhead introduced by the control variates. We provide a table in their respective answers showing that their overhead is negligible for both the KL- and $\alpha$-divergences.

3. Reviewers NozV and Gn1e asked for an assessment of the effect of $\alpha$ on training convergence and guidelines for choosing it. Our novel experiments ilustrate how varying $\alpha$ modulates the balance between mode-seeking and mass-covering behaviors and that large negative $\alpha$ values are more suited to sparse target distributions (Fig. 1, main text; Fig. 3, rebuttal PDF).
 
4. Reviewers NozV and Gn1e questioned the comparative benefit in real-world and high-dimensional tasks; reviewer 3sBX raised concerns about scalability. Beyond the extensive experiments and realistic tasks such as BPI, we included the hypergrid and Causal DAG as additional environments. Compared to baselines (TB, DB, SubTB, and VarGrad), divergence-minimization algorithms lead to faster training convergence in most problems (Figs 1 and 2, rebuttal PDF). 

We would also like to thank the reviewers again for their invaluable feedback and their help in strengthening our work.

Meta-review: This paper f-divergence based objectives for GFlowNets (GFNs). These provide an alternative to the standard trajectory-balance objective for GFN and define generalizations of variational objectives for GFNs based on KL divergences. The authors consider Renyi-$\alpha$, Tsalli-$\alpha$, as well as the forward and reverse KL. The authors derive gradients for these objectives in the case of a fixed backward policy and on-policy samples, and develop control variates that reduce the variance of gradient estimates. Experiments demonstrate that using these objectives can accelerate convergence and enhance stability of training. 

Reviewers did not fully arrive at consensus regarding this submission. Overall reviewers were sympathetic to the exploration of alternative objectives for GFNs presented in the paper, as well the contributions in terms of developing gradient estimators and control variables. However, but reviewers also noted limitations of the approach in terms of the setting with fixed backward policy and on-policy samples, as well the experimental evaluation, which primarily focuses on relatively simple toy problems. Overall two reviewers (HRNm and 3sBX) support acceptance, with 3sBX noting during the AC discussion phase that in spite of shortcomings, they consider the paper to make enough contributions to warrant acceptance. Out of the more critical reviewers, Gn1e raised their score after the author response. Reviewer NozV, who was the most critical of the submission in their review, did not engage with the author response or the AC. As far as the AC can determine, at least some of their concerns were addressed directly by the author response, and as such the AC will somewhat discount the score from this reviewer. 

The AC took a quick look at the submission, which appears to be well-presented piece of work that makes a reasonable set of contributions to this space. This seems above the bar for acceptance. Do please incorporate the revisions that were discussed in the author response.
Decision: Accept (poster)