Beyond Squared Error: Exploring Loss Design for Enhanced Training of Generative Flow Networks
Authors: Rui Hu
Yifan Zhang
Zhuoran Li
Longbo Huang
Venue: ICLR 2025 Spotlight (2025)
Keywords: GFlowNet
Generative Models
f-Divergence
Loss Function
Primary area: generative models

Abstract: Generative Flow Networks (GFlowNets) are a novel class of generative models designed to sample from unnormalized distributions and have found applications in various important tasks, attracting great research interest in their training algorithms. In general, GFlowNets are trained by fitting the forward flow to the backward flow on sampled training objects. Prior work focused on the choice of training objects, parameterizations, sampling and resampling strategies, and backward policies, aiming to enhance credit assignment, exploration, or exploitation of the training process. However, the choice of regression loss, which can highly influence the exploration and exploitation behavior of the under-training policy, has been overlooked. Due to the lack of theoretical understanding for choosing an appropriate regression loss, most existing algorithms train the flow network by minimizing the squared error of the forward and backward flows in log-space, i.e., using the quadratic regression loss. In this work, we rigorously prove that distinct regression losses correspond to specific divergence measures, enabling us to design and analyze regression losses according to the desired properties of the corresponding divergence measures. Specifically, we examine two key properties: zero-forcing and zero-avoiding, where the former promotes exploitation and higher rewards, and the latter encourages exploration and enhances diversity. Based on our theoretical framework, we propose three novel regression losses, namely, Shifted-Cosh, Linex(1/2), and Linex(1). We evaluate them across three benchmarks: hyper-grid, bit-sequence generation, and molecule generation. Our proposed losses are compatible with most existing training algorithms, and significantly improve the performances of the algorithms concerning convergence speed, sample diversity, and robustness.

Published as a conference paper at ICLR 2025
BEYOND SQUARED ERROR: EXPLORING Loss DESIGN FOR ENHANCED TRAINING OF GENERATIVE FLOW NET- WORKS
Rui Hu;; Yifan Zhang; Zhuoran Li, Longbo Huang Institute for Interdisciplinary Information Sciences, Tsinghua University thu-x24, zhangyif21, aar200mmaalleee : Esinghua. edu. en longbohuang@ta nigua edu.en on
ABSTRACT
Generative Flow Networks (GFlowNets) are novel class of generative models designed to sample from unnormalized distributions and have found applications in various important tasks, attracting great research interest in their training algorithms. In general, GFlowNets are trained by fitting the forward flow to the backward flow on sampled training objects. Prior work focused on the choice of training objects, parameterizations, sampling and resampling strategies, and backward policies, aiming to enhance credit assignment, exploration, or exploitation the training process. However, the choice of regression loss, which can highly influence the exploration and exploitation behavior of the under-training policy, has been overlooked. Due to the lack of theoretical understanding for choosing an appropriate regression loss, most existing algorithms train the flow network by minimizing the squared error of the forward and backward flows in log-space, i.e. using the quadratic regression loss. In this work, we rigorously prove that distinct regression losses correspond to specific divergence measures, enabling us to design and analyze regression losses according to the desired properties of the corresponding divergence measures. Specifically, we examine two key properties: zero-forcing and zero-avoiding, where the former promotes exploitation and higher rewards, and the latter encourages exploration and enhances diversity. Based our theoretical framework, we propose three novel regression losses, namely, Shifted-Cosh, Linex(1/2), and Linex(l). We evaluate them across three benchmarks: hyper-grid bit-sequence generation, and molecule generation. Our proposed losses are compatible with most existing training algorithms, and significantly improve the performances of the algorithms concerning convergence speed, sample diversity, and robustness.
INTRODUCTION
Generative Flow Networks (GFlowNets), introduced by Bengio et al. (2021; 2023), represent a novel class of generative models. They have been successfully employed in wide range of important applications including molecule discovery (Bengio et al., 2021), biological sequence design (Jain al., 2022), combinatorial optimization (Zhang et al., 2023), and text generation (Hu al., 2024), attracting increasing interests for their ability to generate a diverse set of high-quality samples.
GFlowNets are learning-based methods for sampling from an unnormalized distribution. Compared to the learning-free Monte-Carlo Markov Chain (MCMC) methods, GFlowNets provide an alternative exchange the complexity of iterative sampling through long chains for the complexity of training a sampler (Bengio et al., 2023). GFlowNets achieves this by decomposing the generating process into multiple steps and modeling all possible trajectories as a directed acyelic graph (DAG). The training goal is to determine a forward policy on this DAG, ensuring that the resulting probability distribution over terminal states aligns with the unnormalized target distribution. However, achieving this alignment is challenging due to the necessity of marginalizing the forward policy across vast
Equal contribution Corresponding author

---

Published as a conference paper at ICLR 2025
Objective Function
Training Object o
Training Object o Pla
Balence Loss
in (a) Σ, و()) log
Sackeaard Polley
Tranning Citjerts
Farameter 22torr
Pr (o)
mapping
Pa
Equal gradient
Resampling Weighis
Samplling 6
Reg ression
Loss
Zero-forcing Zero-avoiding
Minimal Cut C Dj (AFOIIFFO)
Dive gonoo
Measure
Minimal Cut C
Figure 1: An illustration of our main theoretical results: the unified framework for GelowNet training algorithms and the correspondence between regression losses over forward and backward flows on training objects and f-divergences between the two flows over minimal cuts.
trajectory space. To address this, GelowNets utilize backward flow to distribute the unnormalized target distribution over trajectories, thereby aligning the forward and backward flows.
Building on this foundation, various training algorithms for GelowNets have been proposed, aiming to enhance the training of GFlowNets from different aspects such as credit assignment, exploration, and exploitation. Depending on the main focus of the methods, these algorithms can be divided into four categories, including training objects (Malkin et al., 2022; Madan et al., 2023), parameterization methods (Pan et al., 2023a; Deleu et al., 2022), sampling and resampling strategies (Rector-Broo et al., 2023; Kim et al., 2024d; Lau et al., 2024) and the selection of backward policies (Shen et al., 2023; Jang et al., 2024a).
Most existing algorithms train the flow network by minimizing the squared error of the forward and backward flows in log-space, iee. using the quadratic regression loss. However, there may exist more potential choices for loss functions beyond square error. Intuitively, any convex function that is minimized at zero point also provides a valid objective, in the sense that the forward and backward policies are aligned if and only if the loss is minimized. Further, the gradients of different regression losses lead to different optimization trajectories of the forward policy, thus highly influencing the exploration and exploitation behaviors. Yet, due to the lack of theoretical understanding for choosing an appropriate regression loss, it is unclear whether the above intuition is practical. In particular, the following central question remains open:
Can a theoretical foundation be established for designing and analyzing regression loss functions?
To answer this question, we conduct a systematic investigation of the largely overlooked regres- sion loss aspect in GFlowNet training. Specifically, building on the work of Malkin et al. (2023), which established that training GFlowNets analogous optimizing a KL divergence, we rig- orously prove that the gradient of the objective function using different regression losses, when combined with appropriate proposal distributions and resampling weights, equal to that of dis- tinct divergence measures between the target distribution and the flow network-indun tion. As different divergence measures endow the training objectives with different properties, and hence show different characteristics in the training process, our results provide unified framework to generalize existing training methods and provide a prin- cipled way of designing efficient regression losses for GFlowNets training. Fig. provides an overview of our technical results.
distribu-
zero-forcing
non-zero-foreing
Shifted-Cosh
Linex(I) ((t) =ct-111111
In particular, we study two important properties of the training objectives, i.e., zero-foreing and zero-avoiding, and systematically investigate their effects. In general, zero-forcing losses encourage exploitation, while zero- avoiding losses encourage exploration. Equipped with our new framework, we design three novel regression losses, namely Linex(1), Linex(1/2), and Shifted-Cost Cosh, filling the four quadrants made up of the zero- forcing and zero- avoiding properties. We evaluate the new losses on three popular benchmarks: hyper-grid, bit-sequence generation, and molecule generation. Our results show that the newly
9(t)
Quadratic 1
Lincx(1/2)
s(0) 7
-
4et
2t
Figure 2: Our proposed regression losses and their properties.
2

---

Published as a conference paper at ICLR 2025
proposed losses exhibit significant advantages over existing losses in terms of diversity, quality, and robustness, demonstrating the effectiveness of our design framework.
Our contributions can be summarized as follows:
We develop novel framework of the objective functions for training GFlowNett. This new framework identifies five key components of the objective function and unifies existing GelowNet training algorithms including Flow-Matching GetowNets (Bengio et al., 2021), Detailed-Balance GFlowNets (Bengio et al., 2023), Trajectory-Balance GelowNets (Malkin et al., 2022), Sub- TrajectryyBallncee GelowNets (Madan et al.. 2023) and their variants like Norward-Looking GFlowNets (Pan et al., 2023b) and DAG GFlowNets (Deleu et al., 2022), etc (see Section 4.
We establish the correspondence between the various objective functions for GelowNets and different divergence measures. This insight facilitates a deeper understanding of how to design and analyze effective training objectives for GFlowNets (See Section 4.2).
Based on our framework, we conduct an in-depth investigation on two key properties of regression losses: zero-forcing and vero-avoiding. We then design three new loss functions possessing different exploration/exploottionn features, namely, Linex(1), Linex(1/2), and Shifted- (see Section 4.
We conduct extensive experiments on three popular benchmarks: hyper-grid (Bengio et al., 2021). bit-sequence generation (Malkin et al., 2022), and molecule generation (Bengio et al., 2021) I). Our results demonstrate that the new losses significantly outperform the common squared loss in metrics including convergence speed, diversity, quality, and robustness (see Section 5).
2 RELATED WORK
Generative Flow Networks (GFlowNets). GFlowNets were initially proposed by Bengio et al. (2021) for scientific discovery (Jain et al., 2023a) framework for generative models capable of learning to sample from unnormalized distributions. The foundational theoretical framework was further developed by Bengio et al. (2023). Since then, numerous studies have focused on enhancing GFlowNet training from various perspectives, such as introducing novel balance conditions and loss functions (Malkin et al., 2022; Madan et al., 2023), refining sampling and resampling strategies (Shen et al., 2023; Sector-Brooks et al., 2023; Kim et al., 2024d; Lau et al., 2024), improving credit assignment (Pan al., 2023a; Jang al., 2024b) and exploring different options for backward policies (Shen et al., 2023; Mohammadpour et al., 2024; Jang et al., 2024a). Notably, our proposed method is compatible with all the aforementioned works as we have identified a novel key component of the objective functions for training GFlowNets.
From a theoretical perspective, GFlowNets are closely related to variational inference (VI, Malkin et al. 2023; Zimmermann et al. 2023) and entropy-regularized reinforcement learning (RL) on deterministic MDPs (Tiapkin et al., 2024; Mohammadpour al., 2024). All of them can be viewed as solving distribution matching problems, and the gradients of their training objectives are equivalent to that of the reverse KL divergence. Our main theorem (Theorem 4. and Theorem B. generalizes the theoretical results of Malkin et al. (2023) by (i) extending reverse KL to the whole family of f f-divergence and (ii) including more parameterization methods.
People also try to extend the formulation of GelowNets to more complex scenarios, allowing continu- ous space (Lahlou et al., 2023), intermediate rewards (Pan et al., 2023b), stochastic rewards (Zhang et al., 2024b), implicit reward given by priority (Chen & Mauch, 2024), conditioned rewards (Kim et al., 2024c), stochastic transitions (Pan et al., 2023c), non-acyclic transitions (Brunswic et al., 2024), etc. Equipped with these techniques, Gels are applied to an increasingly wide range of fields including molecular discovery (Jain et al., 2023b; Zhu et al., 2024; Pandey et al., 2024), biological sequence design (Jain al., 2022: Ghari et al., 2023), causal inference (Zhang et al., 2022; Atanackovic et al.. 2024; Deleu et al., 2024), combinatorial optimization (Zhang et al., 2023; Kim et al., 2024b), diffusion models (Zhang et al., 2024a; Venkatraman et al., 2024) and large language models (Hu et al., 2024; Song et al., 202Fani4).
Divergence measures in training generative models. The properties of divergence measures and their effects training objectives have been studied by Minka et al. (2005). As the original training objectives of many generative models are equivalent to one of the divergence measures (typically the reverse KI. divergence), it is natural to introduce more general class of divergence measures to
3

---

Published as a conference paper at ICLR 2025
replace it. This idea has successfully improved the performances of a variety of algorithms for training generative models, including GAN (Nowozin et al., 2016; Arjovsky et al., 2017). VAE (Zhang et al., 2019). VI (Li & Turner, 2016; Dieng et al., 2017). Distributional Policy Gradient (DPG for RL, Go et al. 2023), and Direct Preference Optimization (DPO for RLHF, Wang et al. 2024). A very recent study (Silva al., 2024) also explores this idea in the context of GFlowNets by investigating the use of four different divergence measures; forward KL, reverse KL, Renyi-a and Tsallis-a.
In contrast the studies mentioned above, this work establishes two-way connection between f- divergences and the regression loss function in the training objectives of GFlowNets. By examining the aero-foriing and zero-avoiding properties of these divergence measures, we can opt for the desired regression loss to enhance exploration and/or exploitation for training GFlowNets.
3 PRELIMINARIES OF GFLOWNETS AND f-DIVERGENCE
In this section, we first present preliminaries of GFlowNets and the f-divergence, which will be the foundation of our subsequent exposition.
3.1 GELOWNETS
GFlowNet defined on a directed acyclic graph G EV with a source node s, and sink node 8,, such that every other vertex is reachable starting from 8.. and 8y is reachable starting from any other vertex. Let T be the collection of all complete trajectories, and Σ be the corresponding o-algebra, then flow is measure F on (T,E). Σ).
Further, we define state-flow, edge-flow and total flow by
F(s) == F([:: T)), F(s + 8') F((T: (')
F==F([]))
flow then induces a forward probability Pr(''(s) and a backward probability Pn((s defined as:
Pp(s'(s) = P(s s'|s) F(s + s') Ps(s)s') = P(s s'|s') F(s +')
F(s)
F(s')
Markovian flow is a special family flows such that at each step, the future behavior a particle in the flow stream only depends on its current state. Formally speaking, let be any trajectories from 8o to 8, then ((s > (c) = P(s - x'1s) = (s'')s) We focus on Markovian flows in the following.
A set of (not necessarily complete) trajectories C' is cut if and only if for any complete trajectory there exists & E C' such that i is a part of r. Here we view vertices and edges as trajectories of length 1 or 2 and further extend the definition of F to all trajectories
F(1) F(fT is a part of T}).
minimal cut a cut such that the sum of flows in the cut minimized. According to the max-flow min-cut theorem, this amount is equal to Z the total flow. Let C be the collection of all minimal cuts, then for each minimal cut E C, let (c)) F(t) for all C, then p (-) can be viewed as an unnormalized distribution over C.
Let the terminating set 5 be the collection of nodes that directly link to sy. Note that C = {(s + s') s' } is a minimal cut, so pC (-) induces a distribution over Sy. We denote it as pf and its induced probability distribution as Pr (called the terminating probability):
PF((s)
F(s +5))
vs E S5, (F)) = f(s + sz), (s()) = Evest PF((')
Z
The ultimate goal of training GFlowNet is to match PT with R, so that the forward policy draws samples from Pr = PR. where Pi denotes the normalized probability distribution defined by R.
3.2 F-DIVERGENCE
The f diivergencce is general class of divergence measures (Liese & Vajda, 2006; Polyanskiy, 2019):
Dy(pllg) = Σ {(x) (p(x)
+ ('(o)) E q(x) = -0))
((x))
#EX
4

---

Published as a conference paper at ICLR 2025
where p and q are two probability distributions measurable space (X f R++ - R
twice differentiable convex function with f(1) f1( = 0, and (0())) = lime-s+ f(1) Hence, the Kullback-Leible (KL) divergence (Zhu & Rohwer, 1995) DRL and DKL (glpp) correspond to D)(pllg) with f(t) = log + 1 and f(t) = 1 - logt t 1, respectively, When f(1) a(1-a) + I-a a the f-divergence corresponds to the a-divergence Da (pllgg introduced in (Zhu & Rohwer, 1995; Amari, 2012).
The f f-divergence preserves the following nice properties of KL divergence, ensuring that they can also serve as good optimization objectives.
Fact 3.1 (Liese & Vajda (2006)). Dy(pllq) if and only i = q. Fact 3.2 (Liese & Vajda (2006)). Dy(pll4) is convex with respect to either or q.
The definition of Dy(pllg) can be further extended to all twice differentiable functions f withh f(1) (1(1) = 0, termed pseudo f-divergence.
TRAINING GENERATIVE FLOW NETWORKS
In this section, present perspective analyzing GFlowNet training algorithms in detail. In Section 4.1, we provide a general framework with five customizable components to unify existing training algorithms. In Section 4.2, dive deep into the regression loss component, which has been overlooked in existing research, and establish a rigorous connection between and divergence measures. In Section 4.3, we further show how utilize this connection for designing and analyzing objective functions.
4.1 A UNIFIED FRAMEWORK FOR GFLOWNET TRAINING ALGORITHMS
Consider the following general objective function for forward policy:
Σ p(o)g log Pn(0; 0) OEO
Lo.
(I)
This formulation is defined by five key components. (i) The set of training objects O, which can include states, transitions, partial trajectories, or complete trajectories. (ii) The parameterization mapping Po, which defines how the parameters of the flow network represent the forward flow by and the backward flow PB. (iii) The sampling and resampling weights μ4, which influence how training objects are sampled and weighted. (iv) The choice of backward policy PB. which can be either fixed or learned. (v) The regression loss function ensuring that the forward and backward policies align when minimized.
Table 1: Summary of existing GFlowNet training algorithms and techniques.
Design Component
A Algorithms
FM-GFN (Bengio et al., 2021). DB-GFI (Bengio al., 2023), TB-GFN (Malkin ct al., 2022), STB-GFN (Madan et al., 2023),
Training Objects O and Parameterization Mapping foo
FL-GFN (Pan et al., 2023a), DAG-GFN (Deleu al., 2022; Hu
al., 2024)
PRT (Shen et al., 2023), TS-GFN (Rector-Brooks et al., 2023),
Sampling/Resampling Weights
LS-GFN (Kim et al., 2024d), QGFN (Lau al., 2024), Genetie-
GFN (Kim et al., 2024a)
GTB (Shen al., 2023), ME-GEN (Mohammadpour et al., 2024),
Backward Policy Ps
PBP-GFN (Jang et al., 2024a)
Regression Loss
Ours
While most GFlowNets training objectives are not explicitly written in this form, Equation (1) unifies all existing training objectives. Table presents a categorization of existing algorithms according to the components they specify.
In previous literature, g() 1,22 has been the only choice for regression loss, and the term
@n(a:() is usually referred to as the balance loss. It is specified by log (n(o:d) pr(0:0)) = 1/2 (log pr(or)]
5

---

Published as a conference paper at ICLR 2025
the training objects and parameterization mapping. Popular balance losses are flow-matching (FM) loss, detailed-balance (DB) loss, trajectory- balance (TB) loss sub-trajectory trajectory-ballnce balance (STB) loss, and their modified versions. For example, the objective function of on-policy TB loss with fixed uniform PB can be written as
PF(T: PF:(:: 0) (log (T 8) pr( 0)
C=Σ
C=
TET
where for T (s0 = So. 81, 82,
-T--1 =s)). we have
=(()))))2(0)P)((:::)) = 2(0) II Pr((((+(11;0),
1=1
- R(sT-1) II
Ps(se-i(s:) = ((s1-1) II indegree (st)
1=1
Please refer to Appendix A for the detailed correspondence of other algorithms under this unified framework.
4.2 THE INFORMATION- THEORETIC INTERPRETATION APORMATION-THEORETIC OF TRAINING OBJECTIVES
Based on our proposed framework, We establish a novel connection between the functions and the f-divergences. The result is summarized in Theorem 4.1 below.
Theorem 4.1. Let OF be the parumeters for forward policies. For each minimal cut C C, the restrictions of both forward and backward flow functions C. can be viewed unnormalized distributions over it, denoted as py and PB. respectively.
If there exists
C- .R+ such that p(o) (p(()) Ecec.oEc w(C) for any then
wCC) where f(1) ((1)=1] _/""" /-' 9 g((ogg))
= Vor Σ CEC
$2 ds.
(2)
The theorem states that the expected gradient of objective function equals the gradient of a weighted sum of /-divergence over minimal cuts if the sampling and resampling weights μ on each training object o equals the forward flow times the accumulated weights on minimal cuts consisting of o. For example, w(C) I[C T] corresponds to TB GFlowNets using on-policy sampling. The detailed proof of Theorem 4.1 is provided in Appendix B. We also provide a thorough discussion about the interpretations of FM, DB, and subTB loss under this framework. Please see Appendix C for details.
Note that when gtt 2 i.e., the popular squared loss, obtain f(t) the reverse KI. divergence, recovering the results in Malkin et al. (2023). Compared to existing work, Theorem 4. offers general connection and applies to wide range of algorithms shown in Table 1. Below, we conclude this section with the following two remarks on the connections between function g and f.
logt 1. Thus. D,
Remark 4.2. Note that f(1) 0, (1) g0(), and f"(t) ="(los t) If g is twice differentiable, then D is an f-divergence and only if g is convex and is minimized at zero point.
Remark 4.3. Solving for from f(t f gf (log s) ds and g()) 0 gives g(t) f(et) -1"" f(s) ds.
"
4.3 DESIGNING NEW REGRESSION LOSSES
Equipped with the connection established Theorem 4.1, we now show how one can build upon it and design regression losses with two important properties: zero-forcing and zero-avoiding. zero-forcing objective leads to a conservative result, while a zero-avoiding objective offers a diverse approximated distribution. As pointed out by previous studies (Minka al., 2005; Go et al., 2023), pero-forcing property encourages exploitation, while zero-avoiding property encourages exploration. Therefore, zero-avoidee avoiding loss may converge faster to more diverse distribution, while a zero-forcing one may converge to distribution with a higher average reward.
To this end, first study the effect of using different divergence measures optimization objectives.

---

Published as a conference paper at ICLR 2025
Proposition 4.4 (Liese & Vajda (2006)). Denote f(0) lim, f(t), f'( lim +00
50)
1. Suppose f(0) = oc, then (p(pllg) = cc if p(z) = 0 and q(z) > 0 for some x. if p (x) and q(x) 0 for some
2. Suppose fo( oc, then D (y(pll9)
In particular, (r.(plg) for S 0, including reverse KL divergence, satisfies the first condition, while (pllg) for a 2 including forward KL divergence, satisfy the second condition. Proposition 4.4 leads to the following results of approximating a distribution by using /-divergence.
Proposition 4.5 (Liese & Vajda (2006)). Let S a subset of the collection of distributions over X.
Let Ps E arg min Dy(pllg).
Zero-forcing: Suppose [(0) - oo, then -0
2. Zero-avoiding: Suppose ('(co) = co, then p2( > 0 p()) 0.
Proposition 4.5 suggests that when S does not cover the target distribution p, the best approximation may vary according to the divergence chosen as the objective.
Since the objective functions for GFlowNets with varying regression losses are closely related to different divergence measures, we similarly define their zero- zero-forcing and vero-avoiding properties.
Definition 4.6. An objective function L for training GFlowNets is
Zero-foreing; for any parameter space O and 8 arg min L(0).
Vs E S R(s) = = (s; 03) = 0, Zero-avoiding: for any parameter space O and 0 arg min L()).
vs E S R(s) > 0 = Pr(s;))) > O.
In such cases, we also say that the regression function g itself is zero-forcing or zero-avoiding.
We then have the following theorem regarding the zero-forcing and zero-avoiding objective functions and regression losses of GelowNets.
Theorem 4.7. Let L. be objective function for training GFlowNets, whose regression loss corresponds to D according to Theorem 4.1. If D is zero-forcing, then L and are both zero- forcing. Dy zero-avoiding, then L and g are both zero-avoiding.
Table 2: Four representative g functions and their corresponding f-divergences. Quadratic loss corresponds reverse KL-divergence or the a-divergence with - O. Linex(1) corresponds to forward KL-divergence or the a-divergence with a - 1. (12)x(1/2) corresponds to Hellinger distance (Hellinger, 1909) the a-divergence with 0.5. Shifted-Cosh corresponds divergence that is both zero-forcing and zero-avoiding.
Loss
gt)
f(t)
f(0)
f'()))
Zero-forcing
Zero-avoiding
Quadratic Linex(1)
f logt flogt -
0
--1
1
Linex(1/2)
4ef - 21 - 4
21 4v1+2
2
2
Shifted-Cosh
+ -t
llog $
80 00
According to Theorem 4.7, the quadratic regression loss g(() 1 is zero-forcing regression loss and focuses on exploitation. Combined with Remark 4.3 that enables us to determine a from an arbitrary Dy. we can easily find regression losses with both, either or neither of the zero-foreing and zero-avoiding properties. Since these two properties are finally rooted in f0) and f'(oc) our framework allows us directly design desired g loss from desired Dy. This provides a systematic and principled way of designing regression losses. For example, to obtain zero-avoiding loss that focuses on exploration, we can solve for g from (t) = f log t - - 1 the forward KL divergence, which gives g(( = et t the Linex(1) function(Garg al., 2023). We also design Linex(1/2) and Shifted-Cosh The former is neither zero-fore cero-forcing nor zero-avoiding, while the latter is both zero- forcing and zero-avoiding (see Table 2). In addition, have also derived another five novel losses, corresponding to the forward and backward x² distance, total variation, symmetric KL divergence, and Jensen- -aanennnShnnnnn divergence, respectively Please Appendix FF for details.
7

---

Published as a conference paper at ICLR 2025
Note that convex function 9 with g(0) g0(0) ensures the whole objective function is valid in the sense that the target distribution is perfectly matched if and only the objective function minimized, regardless of the choices of backward policy, training objects, parameterization, and exploration strategies. We will also see that they preserve the zero-forcing and zero-avoiding properties well from the empirical results in the following section.
EXPERIMENTS
In this section, we consider four representative g-functions (Table 2) and evaluate their performances over Flow-matching GFlowNets. Trajectory-balamn GFlowNets, Detailed-baland GFlowNets, and GFlowNets, across different choices of backward policies and sampling strategies. We consider the following three popular benchmarks, hyper-grid, bit-sequence generation, and molecule generation. Although the sampling and resampling weights μ may not fully meet the conditions of Theorem 1.1, the effects of zero-forcing and zero-avoiding properties are significant, demonstrating great compatibility with existing algorithms.
Lub-trajectory-balatt
5.1 HYPER-GRID
We first consider the didactic environment hyper-grid introduced by Bengio et al. (2021). In this setting, the non-terminal states the cells of a D-dimensional hypercubic grid with side length H. Each non-terminal state has a terminal copy. The initial state is at the coordinate x = (0, (0,0, ...... .0). For non-terminal state, the allowed actions are to increase one of the coordinates by without exiting the grid and to move to the corresponding terminal state.
The reward of coordinate (Fi, IDD) is given according
R(x) = Ro + R1
2 0.5
0.25
+ R2
II
0.3
0.5
0.4
1=1 H
i=1
where 0 < R1 < Ro < R2. Therefore, there are 2D reward modes near the corners of the hypercube.
We conducted experiments in 4-dimensional and 5-dimensional grids with H 20, Ro 10-1, Ri 9.9 x 10-5, R2 = 10-6 The backward policy learned using the same objectives as the forward policy. We use the forward policy to sample training objects. We plot the empirical L errors between Pr and PR in Figure 3. Additional details can be found in Appendix E.
40-2hta
42.gre
40
4P-grds
5D-ghts
SP.grth T3
SPugrde
Quadratic (baselinc)
Linex(1) (1)
Linex (1/2)
Shifted Cosh
Figure 3: Hyper-grid results: the empirical L1 distance between Pr and Pn-
As shown in Figure 3, the quadratic loss (baseline) converges the slowest among the four losses in 4D grids, and completely fails in 5D grids, while the other three losses remain robust in most of the cases. This is because the quadratic loss has the poorest exploration ability. Despite the differences in convergence speed, the L, errors between Py and PR are almost the same at convergence when using different regression functions.
8

---

Published as a conference paper at ICLR 2025
TB
DB
STB
Taining steps
training steps
training stops
Quadratic (baseline)
Linex(I)
Linex(1:2)
Shifted Cosh
Figure The number of modes found by the algorithm during training.
5.2 BIT-SEQUENCE CENERATION
In our second experimental setting, we study the bit-sequence generation task proposed by Malkin al. (2022) and Tiapkin al. (2024). The goal is to generate binary strings of length given fixed word length n. In this setup, an n-bit string is represented as a sequence of n / k-bit words. The generation process starts with sequence of n/k special empty words. At each step, valid action replaces an empty word with any k-bit word. Terminal states are sequences with no empty words. The reward is defined based on the minimal Hamming distance to any target mode in the given set ming'M EM d(x,
M c Z3. Specifically, R(x)= exp {
In our experiments, we follow the setup in Tiapkin et al. (2024) where 120, k 8, M 60. P'B is fixed to be uniform during training. We use the e-noisy forward policy with a random action probability of 0.001 sample training objects and the newward-looking style parameterizations for DB and STB experiments. We evaluate the number of modes found during training (the number of bit sequences M such that a candidate within a distance as the Spearman Correlation between Pr and PR over a test set, which has also been adopted by Malkin et al. (2022), Madan et al. (2023) and Tiapkin et al. (2024). Additional details can found in Appendix E.2.
30 has been generated) as well
Table 3: The number of runs that find all modes within 250k steps, and the median of the steps before they find all modes.
Quadratic (baseline)
Linex(1)
Linex(1/2)
Shifted-Cosh
TB
1/5,
5/5, 98.0k
5/5. 111.2k
4/5, 92.2k
DB
5/5, 13.4k
5/5, 10.8k
5/5, 11.7k
0/5,
STB
4/5, 50.6k
5/5. 20.3k
5/5, 55.9k
5/5, 90.0k
Table 4: The Spearman correlation between Py and Pn over test set (the higher the better). The failed runs that modal collapse happened are eliminated.
Quadratic (haseline)
Linex (1)
Linex Linex(1/2)
Shifted-Cosh
zero-foreing
x
X
TB
0.8081 [1.000 0.0199)
0.7421(10.0211)
0.7454(10.0022
0.8122
DB STB
0.7907
000175
0.7464(+0.0107)
0.8213 0..080
0.8088 +0.0169
0.7517(+0.0246)
0.7.11(+0.010))
0.8132(#0.0149)
As shown in Figure 4, the quadratic loss seems to find new modes faster than the other three, but it always slows down and then overtaken before finding all modes. As shown Table 3, quadratic loss fails to find all modes in one of the five STB runs, and four out of the five TB runs. On the contrary, Linex(l) and Linex(1/2) succeed finding all modes all 15 runs with three different settings, and Linex(1) 1) is always faster. The performance of shifted- Cosh varies from different algorithms. As analyzed in Section 4.3, a zero- avoiding loss benefits exploration, while a zero-forcing loss does the opposite. These results are consistent with our analysis in general.
In this environment, the state space so large that the training objects can not fully cover the whole space. Consequently, although the algorithms appear to converge, the distribution Py only approximates Pn rather than perfectly matching it. In such cases, zero-forcing losses have advantages
9

---

Published as a conference paper at ICLR 2025
on the qualities of samples compared to non-zero- non-zero-forcing forcing ones. As shown in Table 4, sero-foreing losses (Quadratic and Shifted- Shifted-Cosh) result in a higher correlation between Pr and PR. meaning that they fit the target distribution better within its support. Besides, also observe that for TB GelowNets with quadratic loss, the forward policy sometimes collapses to fitring only a small proportion of the modes in the target distribution, resulting in extremely low correlation. We eliminate these runs when presenting Table 4.
5.3 MOLECULE GENERATION
STB
18
TB
STB
roward
reward
roward
roward
Quadratic (baseline)
Linex(1)
Linex(1/2)
Shifted Cosh
Top 50
Top 200
Top 1000
Figure 5: Molecule generation results. Top: Average reward and pair- mair-wise similarities of all 200k
generated molecules during each training episode. The similarities are calculated among a randomly chosen subset of 1000 molecules. Bottom: Average reward and pair-wird pair-wise similarities of the top generated molecules during each training episode.
The goal of this task is to generate binders of the sEH (soluble epoxide hydrolase) protein by sequentially joining 'blocks' from a fixed library to the partial molecular graph (Jin et al. (2018)). The reward function is given by pretrained proxy model given by Bengio et al. (2021), 1), and then adjusted by reward exponent hyperparameter 3, i.e., R(xr) R(x)) where It(x:) is the output of the proxy model. For DB, TB, and STB experiments, the backward policies are fixed to uniform. The training objects sampled from the e-noisy forward policy with random action probability of 0.05. Additional details can be found Appendix E.3.
It can be seen in Figure that zero-forcing objectives (Quadractic and shifted-Cosh) have higher overall average reward, while zero-avoiding objectives (Linex(1) and Linex(1/2)) have lower overall similarities, meaning that the samples more diverse. However, things become different when comes to the top & molecules, but Linex(1/2), which is neither zero-forcing nor zero-avoiding, demonstrates the best robustness among them.
CONCLUSION
In this work, we develop a principled and systematic approach for designing regression losses for effi- cient GFlowNets training. Specifically, we rigorously prove that distinct regression losses correspond to specific divergence measures, enabling us to design and analyze regression losses according to the desired properties of the corresponding divergence measures. Based on our theoretical framework, we designed three novel regression losses: Shifted-Cost Linex(1/2), and Linex(l). Through extensive evaluation across three benchmarks: hyper-gric bit-sequence generation, and molecule generation, show that newly proposed losses compatible with most existing training algorithms and significantly improve the performance of the algorithms in terms of convergence speed, sample diversity, and robustness.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China Grants 52450016 and 52494974. The authors thank Professor Yang Yuan for discussions.
REFERENCES
Shun-ichi Amari. Differential-geomenii methods in statistics, volume 28. Springer Science & Business Media, 2012.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017.
Lazar Atanackovic, Alexander Tong, Bo Wang. Leo Lee, Yoshua Bengio, and Jason S Hartford. Dyngfn: Towards bayesian inference of gene regulatory networks with gflownets. Advances in Neural Information Processing Systems, 36, 2024.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381-27394, 2021.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward Hu, Mo Tiwari, and Emmanuel Bengio.
Gflownet foundations. The Journal of Machine Learning Research, 24(1):10006-1000 2023.
Leo Brunswic, Yinchuan Li, Yushun Xu, Yijun Feng. Shangling Jui, and Lizhuang Ma. A theory of non-acyclic generative flow networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 11124-11131, 2024.
Yihang Chen and Lukas Mauch. Order-preserving gftownets. In The Twelfth International Conference on Learning Representattinss, ICLR 2024. Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
URL https: www.revin.eeeeeeeeeeeeeeeee.e.ww net forum? iidiid-iaan VBdd=VXXPPXUII wWW.VVXuXEX..... ou
Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks. In Uncertainty in Artificial Intelligence, pp. 518-528. PMLR, 2022.
Tristan Deleu, Mizu Nishikawa- Nishikawa-Tommey Jithendaraa Subramanian, Nikolay Malkin, Laurent Charlin, and Yoshua Bengio. Joint bayesian inference of graphical structure and parameters with a single generative flow network. Advances in Neural Information Processing Systems, 36, 2024.
Adji Bousso Dieng, Dustin Tran, Rajesh Ranganath, John Paisley, and David Blei. Variational inference via X upper bound minimization. Advances in Neural Information Processing Systems, 30, 2017.
Divyansh Garg. Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent RL without entropy. The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: oppnreeiie...................... net feormm d==30OLde3tRI.
Pouya Ghari, Alex Tseng, Gökcen Eraslan, Romain Lopez, Tommaso Biancalani, Gabriele Scalia, and Ehsan Hajiramezanali. Generative flow networks assisted biological sequence editing, In NeurIPS 2023 Generative Al and Biology (GenBio) Workshop. 2023.
Dongyoung Go, Tomasz Korbak, Germàn Kruszewski, Jos Rozen, Nahyeon Ryu, and Mare Dymet- man. Aligning language models with preferences through f f-divergence minimization. In Interna- tional Conference on Machine Learning, pp. 1546 11583. PMLR, 2023.
Emstt Hellinger. Neue begründung der theorie quadratischer formen unendlichvielen veränderlichen. Journal für die reine und angewandte Mathematik, 1909(136):11--211, 1909.
Edward J. Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference in large language models. In The Tweifth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: http///wwpwweeee///etp/...ee..ep net. forum? d=ouj6p4ca66..

---

Published as a conference paper at ICLR 2025
Moksh Jain, Emmanuel Bengio, Alex Semnandez-Gariiae Jarrid Rector- Rector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang. Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference Machine Learning, pp. 9786-9801. PMLR, 2022.
Moksh Jain, Tristan Deleu, Jason Hartford, Cheng-Hao Liu, Alex Hernandez-Garcia, and Yoshua Bengio. Gflownets for ai-driven scientific discovery. Digital Discovery, 2(3):557-577, 2023a.
Moksh Jain, Sharath Chandra Raparthy, Alex Hernández-GGarcaa Jarrid Rector-Brooks, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio, Multi-objective gflownets. In International conference on machine learning, pp. 14631-14653, PMLR, 2023b.
Hyosoon Jang, Yunhui Jang, Minsu Kim, Jinkyoo Park, and Sungsoo Ahn. Pessimistic backward policy for GFlownets. In The eiiht Annual Conference on Neural Information Processing Systems, 2024a. URL https: wppp/peeeeeeieeeeeeeeee net nnt/ forum werrrummid--@Q2211 jmd.
Hyosoon Jang. Minsu Kim, and Sungsoo Ahn. Learning energy decompositions for partial inference in gflownets. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. URL wwwpreveeee.eeeeeeeeeeeee..o.eee net/ forum? 2 ddt.:1151P11...1111
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning, pp. 2332.. PMLR, 2018.
Hyconah Kim, Minsu Kim, Sanghyeok Choi, and Jinkyoo Park. Genetic-guided GFlownets for sample efficient molecular optimization. In The eiihty- Amnual Conference on Neural Information Processing Systems, 2024a. URL https::ttt///ttctt hatpp::/opeereviieee n forum? 1
Minsu Kim, Sanghyeok Choi, Jiwoo Son, Hyconah Kim, Jinkyoo Park, and Yoshua Bengio. Ant colony sampling with gflownets for combinatorial optimization. arXiv preprint arXiv:2403.07044 2024b.
Minsu Kim, Joohwan Ko, Taeyoung Yun, Dinghuai Zhang, Ling Pan, Woochang Kim, Jinkyoo Park, Emmanuel Bengio, and Yoshua Bengio. Learning to scale logits for temperature-condinn gflownets. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Aus- tria, July 21-27, 2024. OpenReview.net, 2024c. URL https: id=GUEsK 9xJJy.
/ http:///ppeeeeeeeeeeee.pe...e..p net forum?
Minsu Kim, Taeyoung Yun, Emmanuel Bengio, Dinghuai Zhang, Yoshua Bengio, Sungsoo Ahn, and Jinkyoo Park. Local search gflownets. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024d. URL htttps://openreeee
net wwt.neeummiiiiooooo foormmid=6cFoooo. Rxw..
Salem Lahlou, Tristan Deleu, Pablo Lemos, Dinghuai Zhang, Alexandra Volokhova, Alex Hernández- Garcia, Léna Néhale Ezzine, Yoshua Bengio, and Nikolay Malkin. A theory of continuous generative flow networks. In International Conference on Machine Learning, pp. 18269-18300, PMLR, 2023.
Elaine Lau, Stephen Zhewen La, Ling Pan, Doina Precup, and Emmanuel Bengio. QGFN: Con- trollable greediness with action values. In ICML 2024 Workshop on Structured Probabilistic Inference & Generative Modeling, 2024. URL https: httpps:://ppeeeee opwnrevieeeeeeeeeeeeeeeeee net / forum?ie hрBKv8kkOP.
Yingzhen Li and Richard E Turner. Rényi divergence variational inference. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 1081-1089, 2016.
Friedrich Liese and Igor Vajda. On divergences and informations in statistics and information theory. IEEE Transactions on Information Theorry 52(10):4394 4412, 2006.
Kanika Madan, Jarrid Rector- Rector-Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, An- drei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability. In International Conference on Machine Learning, pp. 23467-23483. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2025
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. Advances in Neural Information Processing Systems, 35: 5955-5967, 2022.
Nikolay Malkin, Salem Lahlou, Tristan Deleu, Xu Ji, Edward J. Hu, Katie Everett, Dinghuai Zhang, and Yoshua Bengio. Gflownets and variational inference, 2023. URL htt.:tts///ppeeeiiieee net/ / Eorum? ii=
Tom Minka et Divergence measures and message passing. Technical report, Technical report, Microsoft Research, 2005.
Sobhan Mohammadpour, Emmanuel Bengio, Emma Frejinger, and Pierre-Luc Bacon. Maximum entropy gflownets with soft q-learning. In International Conference on Artificial Intelligence and Statistics, pp. 2593-2601. PMLR, 2024.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
Ling Pan. Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of gflownets with local credit and incomplete trajectories. In International Conference Machine Learning, pp. 26878-266 26890. PMLR, 2023a.
Ling Pan, Dinghuai Zhang, Aaron C. Courville, Longbo Huang, and Yoshua Bengio. Genera- tive augmented flow networks. In The Eleventh International Conference on Learning Rep- resentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b. URL https:/ /ttp:::///peeeevieeee net/ Forrmmid=uuF_CcKS5
Ling Pan, Dinghuai Zhang, Moksh Jain, Longbo Huang, and Yoshua Bengio. Stochastic generative PMLR, 2023c.
flow networks. In Uncertainty in Artificial Intelligence, pp.
Mohit Pandey, Gopeshh Subbaraj, and Emmanuel Bengio. GFlownet pretraining with inexpensive rewards. In Neurips 2024 Workshop Foundation Models for Science: Progress, Opportunities, and Challenges, 2024. URL httpt:://////. http:p/ss///pe.eiie....e... http//ppnnrevieeeeeeeeeeeee net wettt//oouniiiincAAA.a
4mA.
Yury Polyanskiy, 2019, URL: pestts::/ppepp 1 lids. mit.mit.m.... htt.s.mtt..bbb/y////mmaggeeeeeee
LN iN_fdiv. pdf. Last visited on 2024/09/23.
Jarrid Rector-Brooks. Kanika Madan, Moksh Jain, Maksym Korablyov, Cheng- -ao Liu, Sarath Chandar, Nikolay Malkin, and Yoshua Bengio. Thompson sampling for improved exploration in GFlownets. In ICML 2023 Workshop Structured Probabilistic Inference Generative Modeling, 2023. URL wwwps:ttpt... opwnoeveeewi..ee net/ wwt.:/wwwwreeeeeeeeeeeee f
Max W Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving gflownet training. In International Conference Machine Learning, pp. 30956-30975. PMLR, 2023.
Tiago Silva, Eliezer de Souza da Silva, and Diego Mesquita. On divergence measures for training GFlownets. In The eiihth- Annual Conference on Neurul Information Processing Systems, 2024. URL htt.tt:///// httLhttss///peerree//
opw.revie....ee.eeeeee. net forum? 1:=::44:554550000 2P zvn.
Zitao Song, Chao Yang, Chaojie Wang, Bo An, and Shuang Li. Latent logic tree extraction for event sequence explanation from llms. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 202Fani4. URL https:/ www.:n/oooumidddeeeooooooooo
Daniil Tiapkin, Nikita Morozov, Alexey Naumov, and Dmitry P Vetrov. Generative flow networks as catropy-regularize International Conference on Artificial Intelligence and Statistics, pp. 4213 4221. PMLR, 2024.
13

---

Published as a conference paper at ICLR 2025
Siddarth Venkatraman, Moksh Jain, Luca Scimeca, Minsu Kim, Marcin Sendera, Mohsin Hasan, Luke Rowe, Sarthak Mittal, Pablo Lemos, Emmanuel Bengio, Alexandre Adam, Jarrid Rector- Brooks, Yoshua Bengio, Glen Berseth, and Nikolay Malkin. Amortizing intractable inference in diffusion models for vision, language, and control. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: /top/pppevviveeeee ew net. forum? gd=gVTKMaaaGI.
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse KL: generaliz- ing direct preference optimization with diverse divergence constraints. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024. URL https: /wwp//eeee.eeeeeee//////e. net / forum? -ct.iiddd2ccc2kk999
Dinghuai Zhang, Nikolay Malkin, Zhen Liu, Alexandra Volokhova, Aaron Courville, and Yoshua Bengio. Generative flow networks for discrete probabilistic modeling. In International Conference on Machine Learning, pp. 26412-26428. PMLR, 2022.
Dinghuai Zhang, Hanjun Dai, Nikolay Malkin, Aaron C Courville, Yoshua Bengio, and Ling Pan. Let the flows tell: Solving graph combinatorial problems with gflownets. Advances neural information processing systems, 36: 1952-1 11969, 2023.
Dinghuai Zhang, Ricky T. Q Chen, Cheng-Hao Liu, Aaron C. Courville, and Yoshua Bengio. Diffu- sion generative flow samplers: Improving learning signals through partial trajectory optimization. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024a. URL https::/p///o /oppnrvveeeeeeeeeeeeeee ew net. / fo forum? id= OIsahqlUYC. UYC
Dinghuai Zhang, Ling Pan, Ricky T. Q. Chen, Aaron C. Courville, and Yoshua Bengio. Dis- tributional gflownets with quantile flows. Trans. Mach. Learn. Res., 2024, 2024b. URL
net/ htt::/oormmidwweeeeeee SsRYd=p
Mingtian Zhang, Thomas Bird, Raza Habib, Tianlin Xu, and David Barber. Variational f-divergence minimization. arXiv preprint 1907. 1891, 2019.
Huaiyu Zhu and Richard Rohwer Information geometric measurements of general- isation. 1995. URL https://research aston. ac uk ww//e//pblicattion.. of informat iion wwweeteii- -etpreentttto--geeraeeeeeeee generalisation
Yiheng Zhu, Jialu Wu, Chaowen Hu, Jiahuan Yan, Tingjun Hou, Jian Wu, et al. Sample-efficient multi-objective molecular optimization with gflownets. Advances in Neural Information Processing Systems, 36, 2024.
Heiko Zimmermann, Fredrik Lindsten, Jan- Williilemm van de Meent, and Christian Naesseth. A variational perspective generative flow networks. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL
14

---

Published as a conference paper at ICLR 2025
Appendix
Unifying Training Algorithms of GelowNets
16
A.I Training Objects and Parameterization Mapping
16
A.2 Sampling and Resampling Weights
18
A.3 Backward Policy
18
Theorem 4.1 and its Proof
19
Interpretation of Theorem 4.1 for Different Kinds of Losses
20
C.I Flow Matching Loss
20
Detailed Balance Loss
................................
20
C.3 Sub-Trajectory Balance Loss
21
Proof of Theorem 4.7
21
Experimental Details
21
E. Hyper-grid.....
21
E.2 Bit-sequence Generation
22
E,3 Molecule Generation
22
More Divergence-based Losses
22
15

---

Published as a conference paper at ICLR 2025
A UNIFYING TRAINING ALGORITHMS OF GFLOWNETS
An objective function for training GFlowNets is specified by five key components, the training objects O, the parameterization mapping Po. the sampling and resampling weights μ. the backward policy P3 and the regression loss Most existing algorithms specify only one to two of the former four components.
A.I TRAINING OBJECTS AND PARAMETERIZATION MAPPING
The choice of these two components are usually coupled since the parameters are mapped the flow functions defined on training objects. The choice of training objects include states, edges, partial trajectories and complete trajectories, corresponding to Flow-Matching GFlownets (FM-GFN, Bengio et al. 2021), SecaleddBallnnee GelowNets (DB- DB-GFN, Bengio et al. 2023), Sub- Teao--rrretttyyBBulie GelowNets (STB-GFN, Madan et al. 2023) and Trajectory-Ballen Balance GFlowNets (TB-GFN, Malkin al. 2022), respectively. Detailed-BBi Balleecce-Baeeeeee GelowNets and Sub- Tuj-rrajcctory Eanjectory-BBllncee -Bllance GFlowNets can be parameterized in different ways, the variants which are Forward- SrwarddLLokiin GelowNets (FL-GFN, Pan et al. 2023a) and DAG GFlowNets IDAG-GEN also called modified-DB modified-STB. Deleu et al. 2022; Hu et al. 2024). These algorithms can be summarized in Table 5.
Table 5: The training objects and parameterization mappings of different GFlowNet training algo- rithms. Among the parameters, Pa can be either fixed learned.
Algorithm
Training Objects
Parameters
Parameterization mapping
FM
states
F(s (()>)))
Equation (3), (4)
DB
transitions
F(s), Pp(s'(s) (, (s(()))) F(s),
Equation (5), (6)
FL-DB
transitions
Equation (11), (12)
modified-DB
transitions
P=(((ss) (, PB({)s')
Equation (15), (16)
TB
complete trajectories
Pr (1
Equation (7), (8)
STB
partial trajectories
F(s), Pr(s'(s) (sP((s))))) Pr(s'ss (.Ps(s)s'))
Equation (9), (10)
FL-STB modified-STB
partial trajectories
Equation (13), (14)
partial trajectories
Pr(3'(s) (p(sss))
Equation (17), (18)
Flow-Matching GFlowNets (FM-GFN). An FM-GFN is parameterized by an edge-flow function
F E' + R.. It uniquely determines valid flow network if and only if the flow- dow-matching conditions hold:
Vs E V 85m
Σ
F(d 8) =))s))
R(s)+ Σ
F(s + 5")
(8'+))EB
(s-W")E 8"
The flow-matching loss for state is defined
log Ps(s) 2
LFa (s) 2 Pr(s)
where (r(s)= = Σ E(ss + 8)
(3)
(5-+))E
pa(s) =R(s) +
Σ
F(s P(s
(4)
(s (8-25) JE
s" $"
Detailed-Balance GFlowNets (DB-GEN). A DB-GFN is parameterized by state-flow function F V {sy} + R4, a forward probability function Pp V {sy} A(V) and a backward probability function Pn V 85 A(V). They uniquely determine valid flow network if
16

---

Published as a conference paper at ICLR 2025
and only if the detailed balance conditions hold:
e S ((5)) R(s)
v(s - x) E E, # $8, =
The detailed-balance loss for transition + s' is detined as
LDB(S - > 1
PB(s + + log
pr(s + s') where PF(s - 5') (s
2
(5)
(F(s')
Ss + 6 =
dn(s - 5') R(s))
(6)
Prajectoryy-allnce GFlowNets (TB-GEN). A TB-GFN is parameterized by total flow function
Z, forward probability function Pp : ( A(V {so}) and a backward probability function
PB : V {sa} - (V \ {sy}). They uniquely determine a GFlowNet if and only the trajectory
balance conditions hold:
T=1
T-1
V (5g So $1, ST-1,8T sy), 2 Pr(sst(ss)) R((T-1)
II
II
Po (1:-))
The trajectory-balant loss for complete trajectory (so defined as
So, $1 ... ST-1, 81) is
LTB(T) = log (T(T) (r(T)
(
2
T
where PF(T) =2 II
8t)
t=
T-1
II t=1
(8)
Sub- Habb-Trajector-BBlaane GFlowNets (STB-GEN). An STB-GFN uses the same parameters as DB-GFN with an alternative loss, the sub-trajectory-balane loss. It is defined for partial trajectory 6 (5o, 51 . 8T 118))
LSTB(() = log pn(()}
2
pF(()
T=1
where PF(() =F(s)) II (5(4115)))
(9)
t=0
((ST) II P R(ST-1) I[= P ST 81
,T # 91
PD(()
(10)
forward-looking GFlowNets (FL-GFN). FL-GENs require the assumption that the reward func- tion can be extended to the whole state space, instead of restricted to only terminal states. The parameters of FL-GFN are quite similar to that of the original DB GFlowNets and STB GFlowNets, including a forward-looking state-flow function F V \ { + R., a forward probability function Pp V { + A(V) and a backward probability function PB V V\ {so. sy} } + A(V). The only difference is that the original state-flow function F is replaced by the forward- -lokidg version F, following (s) R(s) (ss). The forward-looking eetailed-balance loss and forward-looking
17

---

Published as a conference paper at ICLR 2025
sub-trajectoryy-bllnce loss can be obtained by substituting them with the original ones:
LAL-DA 5 1 where PF(s + 5') = F(s))
fa(s +5))
log
pr(s pr(8~>8))
(11)
DB(s + 8') ((s)))(s)PPP(sss
8 s sf
(12)
R(s),
8 = 8f
1
log Pn((1) PF(()
LEL-STB(() = 2
T=1
where pF(() = =((sp) (so)
II
Pp (st+1ls:)
(13)
(B(1) ((ss))F(ss) IT PB(So-1(ss), ST + 81 R(ST- Pp(st-1(s))
(14)
ST 85
DAG GelowNets (DAG-GFN). DAG-GENs require that each state is terminated and has a non-zero R(a) for all 8. Therefore, the reward. Then according to the Aatailed-balance condition, F(s) 5r(*y|a]
flow network can be parameterized by only the forward probability function PF V \ {sy} A(V)
and the backward probability function PB V \ {so. sy} + A (V). The modified detailed-balance loss and modified sub-trajectory-balane loss can be obtained by substituting them into the original ones:
1
PB(s +
modiificdDB( +)
log
2
PF(s - 8²)
where pF(s + 5 R(s PP((s))
(15)
Pr(s)(s)
8 7
6B(s + 5')
Pflaris))
(16)
R(s))
s =
Temodiffeed STD(())))((()))))))))))) 1 log (n(()
(r(()
R(so)
T=1
where (r(()
II (8(8841)8))
(17)
Pr(s (so)
R(sr)
II Ps (s
(n(()
(eyle)) R(ST-1)
,TT 31
(18)
TT-1 PB(
ST = 8s
A.2 SAMPLING AND RESAMPLING WEIGHTS
There exist various strategies sample training objects enhance exploration and hence accelerate convergence. The usual practice is to use the forward policy, the backward policy, tempered or e- noisy version of them, an offline dataset, or mixture of these strategies. Other choices include using a reward prioritized replay buffer (Shen et al., 2023), applying Thompson sampling (Rector- -Boooks al., 2023), local search (Kim al., 2024d) or genetic search (Kim al., 2024a) the sampled trajectories for extra samples, increasing greediness according to state-action value Q (Lau et al., 2024), etc. The sampled objects may also be reweighed. For example, STB-GFN weights each partial trajectory by factor proportional to X', where / is its length and À is hyper-parameter.
A.3 BACKWARD POLICY
The most common choice of PA is to either fix it to be uniform or simultaneously train it using the same objective as the forward policy. Other criteria include matching (possibly non-Markovian)
18

---

Published as a conference paper at ICLR 2025
prior (Shen et al., 2023), maximizing the entropy of the corresponding forward policy (Mohammad- pour et al., 2024) and learning a pessimistic one that focuses on observed trajectories (Jang et al., 2024a).
B THEOREM 4.1 AND ITS PROOF
Theorem B.1 (An extension of Theorem 4.1). Let 05 and 0g be the parameters for forward and buckward policies, respectively. For each minimal cul C E C, the restrictions of both forward and buckward flow functions on C can be viewed as unnormalized distributions over it, denoted Pr and PB. respectively.
If there exists : C-R,, R4 such that p(o) = -r((o) Ecec,oEc w(C) for any o E O. then
+t/ 1,¹ g'ooo s)
VepLo.so.u.o. = Ver
Σ
w(C) ( where fi(t) ((t)==//
s2 ds
CEC
Ve Po = V
Σ
(1C)) where fa(t) = glogt)
Cee
If there exists w : + R24 such that plo) = pB(0) Ecec,occ wC)) for any o E O, then
Ver po =
Ver
wCC) pf), where fa(t) = tlogt))
OEC
( where fa(t) = J. g' (log s)ds
Vo po.u = Von
CEC
Proof. We prove the theorem by deriving the correspondence. Specifically, assume μu(o) po( Ecec,oec w(C).. Then,
PE(O) (F(0)
]
Ver Σ w(C)D,, (P
Σ
(C)
Σ
Var
DF(O))
CEC
CEC
OEC
(%
(0)
(9((0) (5(()
Σ (c())
Σ
/1
(0) S
VorPF(o)
(F(()
CEC
OEC
log ())
Σ n(()) Σ -gg
V VFrPF(1)
(C)
CEC
DEC
log P5((
)
Σ w(C)
Σ
(F())))'
()( VoLDf(o)
(F((
CEC
oEC
log Pi((0) pF(()
Σ V(o)
TEO
Vo po.pp.
(0(
Ven
Σ w(CDD) Σ (C))
Ven
CC (
CEC
CEC
DEC
log 1 (0)
Σ w(C) Σ
pr (0) Veng
pf (0)
CEC
(og
PZ((0) (C(O)
Von)
0EO
=Vo, Lo Lo
In the second case, suppose u40)= po (0) ECec,ecc WCC). Then,
19

---

Published as a conference paper at ICLR 2025
Pg(()
Ver
Σ
w(C)
Σ
Ver
fo)
(C(()
CEC
CEC
OEC
(2)(0)
(P)(0) (F(()
Σ
(c()) Σ
f3
f3
VorPF(o)
(7(0))
CEC
OEC
log (0
Σ wcC)
Σ
VorDf(o)
(0)
pf(0))
CEC
DEC
log PE((0)
Σ w()
Σ
()))
VonDf(o)
()
()(()
CEC
oEC
log (1) (F(o)
Σ V( 0
OEO
8 Lo Co.po..PP.
PE(O
Vea
Σ
(0
Σ w(C)
VVD
(F(()
CEC
CEC
OEC
Σ w(C) Σ
fí
pF(()
CEC
OEC
()(
Σ (c) Σ
Do(l))''
log
V (g
pF(0) ())
CEC
OEC
log (1(
p(o) Vong 0
D3 (0)
AEO
0n Lo St.
C INTERPRETATION OF THEOREM 4.1 1 FOR DIFFERENT KINDS OF LOSSES
C.I FLOW MATCHING Loss
For any E Velet ((s) be the length of the longest trajectory from s, to For any (s + s') E, if I(s) + 1 < t(s"). then we insert l(s') - -(s) - virtual states on this edge, denoted as $(s->5)),1 for
1(5) < < (s'). and define
= f (s
= F(s + s')
then these virtual states have no contribution to the total loss, thus we can assign to them arbitrary weights.
Let V be the collections of states in layer i, and let wVi)) = 1. then we have
m(s) pk (s) (s)
C.2 DETAILED BALANCE LOss
For any E V, let ((s) be the length of the longest trajectory from 5, to s. For any (s (5)+)) E E, if I(s) +1 1 < I(s'), then we insert 1(5') - [(s)) - virtual states on this edge, denoted as $(s-25)), for
1(8) < I < 1(s'), and define
p +s) =pr(s s
il + 8) = (pr(s + s²) ((s))
(5(s +88 8¹) 11s))s)
20

---

Published as a conference paper at ICLR 2025
then these virtual transitions have contribution to the total loss, thus assign to them arbitrary weights.
Let E" be the collections of edges from layer i to layer 1, and let w(E") then we have
pls s') = pF (s -+ )'s
p
pr ss)
C.3 SUB-TRAJECTORY BALANCE LOSS
Assume that G is graded DAG with L + layers. Suppose T = (su So. $1,".. , =)) complete trajectory, we use Taj = (8i, 8i+1, ,88) to denote a partial trajectory. Let T be the collections of trajectories from layer layer j. then
A
u(()
Σ PF(T)
Eosiesel X-1
Tu=Ti
X
Eosieist N-iPr
Hence w(+))) = Eossese and otherwise.
D PROOF OF THEOREM 4.7
Theorem D.I. Let L be objective function for training GFlowNets, whose regression loss corresponds to Dj according to Theorem 4.1. If Dy is zero-forcing, then L and are both zero- forcing. Dy is zero-avoiding, then L and 9 are both zero-avoiding.
Proof. Assume that Dy is zero-forcing, and Pr(s; 0*) > 0 for some terminating state Then there exists trajectory (s..-.. ;,, 85) such that PF (T; 0) > thus
(o) po(
W(C) > Since Dy is zero-foreing, PB(O) ( for any o E т, meaning that PB(T) > O and R(s) > 0. Thus, R(s) = 0 implies P1 (s; 0) = 0, so L zero-forcing, and then g is fero-forcing well.
for any o T,0 C,
Similarly, assume that D is zero-avoiding, and R(s) > 0 for some terminating state 8. Then there exists trajectory r (s.. ... 8,) such that P (T) > 0, thus
po((0 (o) > 0
for any T,0 E C, w(C) > 0. Since D, is zero-avoiding, PF(() -p((0) > 0 for any E meaning that PF(T; 0) 0, Pr(s;)) 0) 0. Thus, R(s) 0 implies that Pr 0) 0. Cis zero-avoiding, and then is zero-avoiding as well.
EXPERIMENTAL DETAILS
E. HYPER-GRID
Our implementation the baselines is based on Tiapkin al. (2024). All models are parameterized by an MI.P with 2 hidden layers of 256 neurons. We train the model with Adam optimizer using batch size of 16 and learning rate of 0.001. For the TB case, we use larger learning rate of 0. for learnable total flow  For STB parameter A, we use the value of 0.9 following Tiapkin et al. (2024) and Madan et al. (2023). We repeat each experiment 3 times using different random seeds. In each run, we train the models until 800k trajectories have been collected, and the empirical sample distribution is computed over the last 80k seen trajectories.
21

---

Published as a conference paper at ICLR 2025
E.2 BIT-SEQUENCE GENERATION
In this experiment, our implementation of the baselines is based on Tiapkin et al. (2024) and Pan et al. (2023a). The model is 3-layer Transformer with 64 hidden units and 8 attention heads per layer. We train the model with Adam optimizer using batch size of 16 and a learning rate of 0.001. For the TB case, we use a larger learning rate of 0.002 for learnable total flow Z. For STB parameter X, we use the value of 1.5. Following Tiapkin al. (2024), we use reward exponent of 2. To calculate the Spearman Correlation, we use the same Monte-Carie estimation for Pr as Zhang et al. (2022) and Tiapkin et al. (2024), namely
N
PF(T")
(2
PB(TT|2)
i=1
with N 10. We repeat each experiment times using different random seeds.
E.3 MOLECULE GENERATION
In the molecule generation experiment, our implementation of the baselines is based on Tiapkin et al. (2024). We use Message Passing Neural Networks (MPNN) as the model architecture. We train the model with Adam optimizer using a batch size of 4 and a learning rate of 0.0005. We use a reward exponent of 4, and the STB parameter À is set to 0.99. We repeat each experiment times using different random seeds. In each run, We train the models for 50000 steps, generating 200k molecules.
F
MORE DIVERGENCE-BASER LOSSES
Apart from the four representative divergence-based losses in Section 4.3, we also derive another five novel losses from some well-known divergence measures, including the forward and backward x² distance, total variation distance, symmetric KI. divergence and Table 6).
divergence (See
Table 6: Five well-known f-divergences and their corresponding regression losses.
Divergence
f(t)
g(t)
Zero-forcing
Zero-avoiding
Forward x²
}1 - 1)2
$ {c2t 21 1) e e'+
Reverse x²
+ -2)
Total Variation
1|
|
Symmetric KL
1 1) log
1 (e' 112 -
JS Divergence
(tlogt (t + 1) log (41)
2 1. log (++ee) doc
2 distance
The x² distance between p and is defined
(p(x) q(x))
(pllg) Σ
(19)
((x)
ZE
is a special case of a-divergence with a = 2, or ddiiergeence with f(t) = 1/(tt ( 112. According to Theorem 4.1, we obtain the corresponding regression loss Linex(2): g(t) = (c 2t 1).
By exchanging p and q in (19), we obtain the reverse x2 distance, which a special case of a- divergence with 1, or f-divergence with f(t) (t - 22). The corresponding regression loss Linex(-1): g(t) e -
The forward x² distance and Linex(2) are zero-avoiding, while the reverse x² and Linex(-1) distance zero-foreing.
Total Variation.
The total variation between p and q is defined as
TV(pl14)
Σ pp(z) q(x)
2
REX
22

---

Published as a conference paper at ICLR 2025
It corresponds to the f-divergence with f(t) 2/1t 1 1/,, and the regression loss g(t) |t| Since ʃ(0) (0) 2: this loss function is neither zero-forcing zero-avoiding.
Symmetric KL Divergence. The symmetric KL divergence between and is defined as
(skl(pllo) (Dxc(pllg) + DKL(all))
Σ p(x) log p(z) TEX g(x))
+ g(x)) log g(z) p(x)
-
corresponds to the f-divergence with f(I) [l 1) log and the regression loss g(t)
(ce + 1 - 1). Since f(0) (oo) oc, this loss function is both zero-foreing and zero-avoiding.
Jensen-Shannon Divergence. The defined as
divergence (JS divergence) between p and is
Dus(plla) - (Kl -2) (al P
2 9))
2p(x)
2g(x)
Σ ((x) p(x) + q(x) FEX
+ +g(z) log p(x) + q(z)
It corresponds to the f-divergence with f(() (olog (1 1) log ( )) and the regression
1 zero-forcing 2 l log (I+x²) dr. Since f(0) = f'(oo) = 1, this loss function is neither
loss gt) =
vero-avoiding.
We evaluate their performance on bit-sequence generation task using the same metrics (Please refer to Section 5.2 and Appendix F.2 for details). It turns that losses with the same zero-forcing zero-avoiding properties lead to similar behaviors.
Table 7: The number of runs that find all modes within 250k steps, and the median of the steps before they find all modes.
TB
DB
STB
Reverse KL (baseline)
1/5.
5/5.
13.4k 0/5,
4/5, 50.6k
Reverse x²
0/5.
0/5,
Forward KL
5/5. 98.0k
5/5. 10.8k
5/5, 20.3k
Forward X²
5/5, 80.3k
5/5, 8.1k
5/5. 10.2k
Hellinger
5/5, 111.2k
5/5. 11.7k
55.9k
Total Variation
1/5.
5/5, 47.1k
2/5,
Jensen-Shannon
4/5, 162.2k
5/5, 12.8k
3/5, 165.2k
Shifted-Cosh
4/5, 92.2k
0/5,
5/5, 90.0k
Symmetric KL.
4/5, 122.2k
5/5. 13.7k
5/5, 27.5k
23

---

Published as a conference paper at ICLR 2025
Table The Spearman correlation between Pr and PR. over test set (the higher the better). The failed runs where modal collapse happened are eliminated.
TB
DB
STB
Reverse KI. (baseline)
0.8081(+0.0159) 0.8074(+0.0129)
0.7907(+0.0175)
1.8088(+00016))
Reverse
0. 1.7899(+00001)) +000016))))
Forward KL
0.7421(#0.0216)
0.7464(+000007)
0.7517(#0.0246)
Forward X2
0.7507(+0.0174)
0.7266(+0.0178)
0.7439(+0.0126)
Hellinger
0.7454(10.0021))
0.7580(10.0132)
0.7711(10.0190)
Total Variation
0.7893 10.00144)
0.7266( 10.0178)
Jensen-Shannon
0.7852 0.7852 +0.0256)
0.7542(+0.0046)
0.7640(+0.0213)
Shifted-Cosh
0.8122(10
0.8213(10.0094
0.8132 (00.019)
Symmetric KL.
0..005(+
00.007))))
7.7886 #0.0227
24


Reviews:
[{"review_id": "EWq2ajIj7m", "reviewer": "Reviewer_Xe2i", "summary": "This paper presents a novel framework for GFlowNet objective functions, unifying existing training algorithms and clarifying key components. By establishing a connection between objective functions and divergence measures, it offers valuable insights into designing effective training objectives. The authors investigate key regression properties—zero-forcing and zero-avoiding—and propose three new loss functions (Linex(1), Linex(1/2), and Shifted-Cosh) to balance exploration and exploitation. Extensive experiments on benchmarks, including hyper-grid, bit-sequence, and molecule generation, show that these losses outperform the common squared loss in convergence speed, diversity, quality, and robustness.", "questions": "Could the authors clarify if they’ve noticed stability shifts in higher-dimensional or complex tasks and if adjustments might bolster robustness? Additionally, what drives the choice of a limited set of losses—are there theoretical or practical reasons for omitting other f-divergences, like Hellinger? Lastly, insights into each loss function’s hyperparameter sensitivity and effects on convergence guarantees would further clarify their resilience and adaptability.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper introduces a systematic framework for designing regression losses in GFlowNet training, linking each loss to specific divergence measures for targeted properties. Resulting in three new losses—Shifted-Cosh, Linex(1/2), and Linex(1)—that enhance exploration and exploitation balance.", "weaknesses": "Broader exploration of other potential divergence-based losses would offer a more comprehensive understanding of the effects of different divergence properties on GFlowNet training. Although the novelty lies in extending GFlowNet loss functions, there are similar attempts in reinforcement learning and generative models. Although the paper derives theoretical properties of zero-forcing and zero-avoiding, it lacks direct theoretical comparison with existing GFlowNet training algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3mIWZPymAN", "reviewer": "Reviewer_EhxP", "summary": "This paper presents a novel theoretical finding for GFlowNets regarding their objective function. Using f-divergence theories, they connect existing objectives of GFlowNets and show that they are special cases of the squared loss. They design a new loss structure that combines both properties together: (1) zero forcing (as considered in existing losses) and (2) zero avoiding, which compensates for exploration. Their new loss function seems to have empirical benefits.", "questions": "Areas for improvement and suggestions:\n\n1. Making a literature connection with f-GAN, which also uses f-divergence in GANs, might be insightful to readers.\n\n\n2. Include a discussion connecting with off-policy exploration methods. Are your loss and off-policy search orthogonal? Which means, is your loss function combined with an off-policy method (e.g., local search) better than the TB loss combined with an off-policy method?\n\n\n3. It's good to see the categorization of prior GFlowNet works. Can you include this recent work [1] that uses genetic search as an off-policy method for training GFlowNets and provide some discussion?\n\n\n\n[1] Hyeonah Kim et al., \"Genetic-guided GFlowNets for Sample Efficient Molecular Optimization,\" NeurIPS 2024.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n\n\n2. The theories are insightful.", "weaknesses": "1. The empirical results seem to be weak; they are only varied in synthetic tasks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PFDyqg6opa", "reviewer": "Reviewer_8npT", "summary": "The authors propose to modify the loss function of Gflownet (which has been completely overlooked by prior work). They show that dinstinct losses lead to different divergences. They propose three new loss functions, evaluate them extensively on diverse benchmarks.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Contributions:\n- generalizing the objective function of gflownet\n- derive impact of loss function on the gradient\n- define zero-forcing (encourage epxloitation) and zero-avoiding (encourage exploration) as two key properties induced by certain loss functions\n- They create 3 new losses (alonside the existing quadratic loss) to tackle all 4 possible combinations (with/without zero-avoiding, with/without zero-forcing)\n- Linex(1) corresponds to the KL divergence\n- experiments on 3 datasets\n\t- Non-zero-forcing losses (Linex(1) and Linex(0.5)) converge faster on hyper-grid\n\t- Linex(1) obtains all the modes almost always the fastest, but spearman corr between train and test is highest for shifted-cos on bit-sequence\n\t- Linex(1) tends increase diversity while quadratic and shifted-cos give higher quality (high average rewards) on molecule generation\n\nPaper is well written.", "weaknesses": "- it would be nice to have a few more choices of losses, taking inspiration let say from f-divergences (Reverse-KL, JSD, etc.)\n- there may be more than just zero-forcing and zero-avoiding to the key properties of loss functions hence why studying more losses would be helpful\n- it would be nice to let say consider hybrid methods with some kind of annealing. For example, why not use Linex(1) for its fast convergence to a large number of nodes, before then transitioning to shifted-cos for higher rewards around those now-discovered modes.\n\nSo to me, the paper is great, but its kind of stopping too quickly, it feels like its only just tapping the surface. These kinds of ideas would be easy to test out and add to the papers. \n\nIf the authors add bit more meat to the paper, i.e. extra loss functions and hybrid-annealing (as discussed above), I would likely increase my score. Like I said, its just missing a little bit of filling to make it a great paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)