Investigating Generalization Behaviours of Generative Flow Networks
Authors: Lazar Atanackovic
Emmanuel Bengio
Venue: Accepted by TMLR ()

Abstract: Generative Flow Networks (GFlowNets, GFNs) are a generative framework for learning unnormalized probability mass functions over discrete spaces. Since their inception, GFlowNets have proven to be useful for learning generative models in applications where the majority of the discrete space is unvisited during training. This has inspired some to hypothesize that GFlowNets, when paired with deep neural networks (DNNs), have favorable generalization properties. In this work, we empirically verify some of the hypothesized mechanisms of generalization of GFlowNets. We accomplish this by introducing a novel graph-based benchmark environment where reward difficulty can be easily varied, $p(x)$ can be computed exactly, and an unseen test set can be constructed to quantify generalization performance. Using this graph-based environment, we are able to systematically test the hypothesized mechanisms of generalization of GFlowNets and put forth a set of empirical observations that summarize our findings. In particular, we find (and confirm) that the functions that GFlowNets learn to approximate have an implicit underlying structure which facilitate generalization. Surprisingly ---and somewhat contradictory to existing knowledge--- we also find that GFlowNets are sensitive to being trained offline and off-policy. However, the reward implicitly learned by GFlowNets is robust to changes in the training distribution.

Published in Transactions on Machine Learning Research (04/2025)
Investigating Generalization Behaviours of Generative Flow Networks
Lazar Atanackovic University of Toronto Vector Institute
Latomackowic@maill.toooo wtw.::/@mmmmui.ooooooooooo
Emmanuel Bengio Valence Labs
Reviewed on OpenReview: https: h/t:://opppee openreview. net/ formn? 9L0B5N5NUX
Abstract
Generative Flow Networks (GFlowNets, GFNs) are a generative framework for learning un- normalized probability mass functions over discrete spaces. Since their inception, GFlowNets have proven to be useful for learning generative models in applications where the majority of the discrete space is unvisited during training. This has inspired some to hypothesize that GFlowNets, when paired with deep neural networks (DNNs), have favorable generaliza- lion properties. In this work, we empirically verify some of the hypothesized mechanisms of generalization of GFlowNets. We accomplish this by introducing novel graph- graph-based benchmark environment where reward difficulty can be casily varied, p(r) can be computed exactly, and an unseen test set can be constructed to quantify generalization performance. Using this graph-based environment, we are able to systematically test the hypothesized mechanisms of generalization of GFlowNets and put forth a set of empirical observations that summarize our findings. In particular, GFlowNels learn lo approximale have implicit underlying structure which facilitate generalization. Surprisingly and somewhat contradictory to existing knowledge we also find that GFlowNets are sensitive to being trained offline and off-policy. However, the reward implicitly learned by GFlowNets robust to changes in the training distribution.
find (and confirm) that the functions that
1 Introduction
Generative Flow Networks (GFlowNets, or GFNs) have emerged as a generative modeling framework for learning unnormalized probability mass functions of discrete objects such graphs, sequences, sets (Bengio et al. 2021 2023). They have show particular promise for wide range of problems and applications where optimization is conducted over very large combinatorial spaces. To name few, discrete probabilistic
modeling (Zhang et al. 2022), molecular discovery (Bengio el. al. 2021 Jain el al.] 2023), biological sequence design (Jain et al. 2022), and causal discovery (Deleu et al. 2022 2023 Atanackovic et al.) 2023) are areas where the discrete spaces may be intractably large. In these applications, the model will realistically only visit small fraction of the overall state space. Therefore, understanding how well GFlowNets model and assign probability mass to the unvisited areas of the state space is critical step. This is fundamentally question of generalization and is yet to be practically investigated within GFlowNets.
It has been hypothesized that GFlowNets work well because they synergistically leverage the generalization palential of DNNs to assign probability mass in unvisited areas of the state space (Bengio et al. 2021 Indeed, the ability of GFlowNets to learn environment structure plays a critical role in their ability to generalize a
familiar result found in reinforcement learning (RL) Cobbe et al. 2019 Schrittwieser al. 2020 Although
Part of this work was accomplished during an internship Valence Labs.
Our code available at: hhttps://github. wwwp:/ttuu...oooaaaaeeeee com/lszarataa/gflomm http://ogooottttggaaaalaaa ization

---

Published in Transactions on Machine Learning Research (04/2025)
have some insight into how GFlowNets work, we know of many failure cases of generalization which exist
in RL. (Zhang et al. 2018 Packer et al. 2018 Bengio et al. 2020). With the close relationship between GFlowNets and RL Tiapkin et al., 2024 Deleu et al. 2024 Mobammadpour et al., 2024), we can expect to observe similar problems in GFlowNets when it comes to generalization. Hence, constructing a systematic investigation into generalization within GFlow.Nets is useful for future algorithmic development, advancement.
in scientific applications, and our overall understanding.
While the works of Nica al. (2022) and Shhnn et al. (2023) probe at the question of generalization within GFlowNets, they only do superficially. Questions regarding the mechanisms of generalization in GFlowNets have yet to be systematically investigated and wholly understood. Thus, unravelling some intuitive notions on why and how GFlowNets generalize motivates this work. We center our investigation around three primary hypotheses for generalization in GFlowNets:
1. GFlowNets generalize well only under a narrow set of distributions, which includes, but is not limited to, sampling from Pr(s'|s;
2. GFlowNets generalize well because the objects they are learning have structure; Pe(s'(s) and F(s) are not "arbitrary" functions.
3. The difficulty for GFlowNets to generalize is modulated more by the complexity of the reward (functionally, the generalization error of supervised DNN) than the properties of the distribution induced by the reward; e.g. skewness, temperature.
To investigate these hypotheses, we devise an empirical investigation under experimental settings. (1) Distilling (regressing to) the true flows of the environment. This lets evaluate generalization an unseen test set of states and lest Hypotheses 2 & 3 (2) Measuring memorization gaps when learning the true flows while controlling for data and environment structure. Here we can probe the effect of environment structure on generalization to verify Hypothesis 인 Finally, (3) training GFlowNets offline and off-policy under different training distributions. This lets us explore the effects of fifinne-fff-oliccy training and changes in the training distribution on generalization; testing Hypothesis 1 These experimental settings help us isolate specific factors of variation which may play a role in why GFlowNets generalize.
To conduct our empirical investigation, propose a set of new graph-based generation tasks to benchmark the performance of GFlowNets for learning unormalized probability mass functions over discrete spaces. Tn most existing works, it is common to use the hypergrid and sequence environments to assess performance of GFlowNets in their ability to approximate unormalized probability mass functions. However, the hypergrid and sequence environments are fundamentally limited for assessing generalization performance of GelowNets as their structure yields tasks that are easy to learn and approximate. Because of this, we introduce new graph-based benchmark environment where we can incrementally increase the difficulty of the reward while also ensuring that we can easily construct sufficiently challenging test set of unseen (left out) states to quantify generalization performance. This new environment provides us with means to more robustly test the generalization capabilities of GFlowNets compared to the commonly used benchmark environments (c.g. the hypergrid and sequence environments).
We describe the details of our experimental setups in $3 benchmark tasks with well defined and tractably computable p(x; 8), Vx E x Our main contributions are summarized as follows:
In
2
we describe our set of comprehensive
We propose a set of benchmark graph generation tasks of varying difficulty, useful for evaluating GFlowNets' generalization performance.
We reify and validate some hypothesized characteristics of GFlowNet generalization behaviour over discrete spaces. We accomplish this empirically using benchmark tasks.
We note that proposed empirical study helps disentangle some of the mechanismss of generalization in GFlowNets, but docs not necessarily determine the true underlying causal order of these mechanisms. Nouctheless, we believe this work can pave road-map for how to approach testing generalization of GFlowNeta and view this key step towards understanding important properties for this class of algorithms.
2

---

Published in Transactions on Machine Learning Research (04/2025)
We identify and present set of observations and empirical findings that form basis towards disentangling some of the mechanisms for generalization of GFlowNets.
1.1 Generative Flow Networks
Generative Flow Networks (GFlowNets, GFNs) are generative modeling framework used to learn to sample from an unnormalized probability distribution. We will refer to this unnormalized function as positive reward, R(s) > 0. Introduced in the discrete setting (Bengio et al., 2021), but since then extended to continuous settings (Lahlou et al. 2023), GFlowNets work by learning a sequential, constructive sampling policy. This policy, PF, is used to sample trajectories T = (so...., st) where states E S are partially constructed objects, making the state space a pointed directed acyelic graph (DAG) (S) where (s + +') A is a valid constructive step. There is a unique initial state 80-
While they can be expressed in multiple equivalent ways, will think of GFlowNets in this work through three main objects: the flow of state F(s) > O, and the forward and backward policies, Pr(s'|s) and Ps(s|s'). In perfect GFlowNet, these functions are such that for any valid partial trajectory (sn...,
F(Sn) II P = F(sm) t=
II Pa(0)184+1) 1=
(1)
where for terminal (leaf) states, F(s)= = R(s) by construction. If the above equation is satisfied, then starting at 8o and sampling from PF guarantees to reach a leaf state 8t = x with probability p(r) x R(x), E X. We use X' to denote the set of terminal (lcaf) states. Note that it may be useful think of Pp and PB as representing fractions of flows going forward and backward through the (DAG) network. It may also be useful to think of the flow going through edges: F(s 5') (ss)ss)))ss).
The so-called balance condition in 1 leads to varicty of learning objectives. In this work we primarily use the Sub-trajectory Balance (specifically SubTB(1)) introduced by Madan et al. (2022) and often considered standard, which takes the above conditions, parameterizes the log flow and logits of policies, taking the squared error over all possible subtrajcctorics of (So. ..., ST):
CSubTB(T)= = Σ log F(S.) Pr(8++1(ss) F(sm)
(2)
PB(
nEMST
We note that trajectory balance (TB) is special case of the above, where only n 0 and m T are used. We also note that F(s) is upper bounded (when PB of all its descendant edges is 1) by the sum of the rewards of all its descendant leaves. For a complete overview of GFlowNets, We define terms that we want to make as least ambignous as possible in
refer readers to Bengio et al. (0023
1.2 Related Work
Generalization in deep learning Generalization of deep neural networks (DNNs) in supervised learning
and deep learning has been extensively studied (Leshno et al. 1993 Pascanu et al. 2013 Zhang et al. 2017 Kawaguchi et al. 2017). Although it is still not entirely understood, certain mechanisms and intuitive notions for generalization have emerged as the favored schools of thought (Arpit et al. 2017 and inspire this work.
2021
Generalization in reinforcement learning Due to their close relationship with GFlow works studying generalization in Reinforcement Learning also inspire this work (Zhang et al. 2018 Packer al.
2018 Cobbe et al.] 2019: Bengio et al. 2020). For example, generalization behaviours differ when learning the same function through regression or through temporal credit assignment [Bengio et al. 2020 which in some form NelowNets make use of.
Generalization in GFlowNets It is uncontroversial that GFlowNets generalize, to some extent. They learn a distribution that matches the reward in-distribution (Bengio et al., 2021). and on test sets (Malkin et al. 2022), are affected by the choice of parameterization and flow distribution (Shen et al. 2023) and
3

---

Published in Transactions on Machine Learning Research (04/2025)
Table 1: Mean Absolute Error (MAE)
when training
GFlowNets for different GNN architectures.
Model
Constant
Counting
Neighbors
Cliques
GAT
0.08 = 0.00
0.14 0.01
0.30 0.01
0.32 0.03
GCN
0.48 +: 0.01
0.48
2.23
1.53
GIN
0.34 # 0.01
0.39 0.01
1.48 0.09 0.64 0.05
Table 2: Jenson-Shannon (JS) divergence when training GelowNets for different GNN architectures.
Model
Constant
Counting
Neighbors
Cliques
GAT
0.002 i 0.000
0.002 + 0.000
0.005 +: 0.001
0.008
0.001
GCN
0.031 + 0.001
0.037 = 0.001
0.339 = 0.017
0.240 + 0.049
GIN
0.018 + 0.000
0.018 + 0.000
0.247 = 0.007
0.061 + 0.007
10
councing cliques
10-3
constant
counting neighbors diiques
10
Diverpence
10-1
10
Au
10"
10-1
10.1
10-4
10"
20
40
60
80
100
Tra inling progress
e.xampples seen 167
eexamples seen 167
(a)
(b)
Figure 1: (a) Training a GNN on 3 different tasks with models of varying capacity. Most of the variance comes from varying capacity. Dashed lines for the highest capacity models. (b) Training GFlowNet (online and on-policy) on 4 different tasks. While ordering is mostly preserved, apparent difficulty depends on the choice of metric.
more generally are able perform much better than random search in large scale intractable problems. That being said, no study so far has tried to zoom in on whether anything unique about generalization within GFlowNets.
2 A Benchmark Task to Investigate Generalization Behaviours of GFlowNets
As first step, present novel benchmark environment built series of graph-based tasks. We use graphs as the foundation of the benchmark tasks as they are natural choice for compositional discrete objects, and wide range of combinatorial problems can be expressed as graph generation. We define tasks of varying difficulty on a fixed state space, thus holding the environment constant while the reward difficulty is varied. For completeness, we also conduct experiments on two common benchmark tasks in GFlowNet literature: the hypergrid and sequence tasks.
2.1 A Small Graph Environment
Consider space of graphs X and an action space A C X' x X of additive graph edits. We then define PF(-Is): A, + +Rt The action space has 3 actions: (i) add a new node, (ii) add new edge, or (iii) stop/terminate. When adding nodes, there is choice between two "colors" of nodes. This defines the state graph G = (X)A) for the graph environment; again note that states are themselves graphs.
sss,s') (s, s") E A}. The initial state so is the empty graph (no nodes and edges).
We setup an environment with all possible graphs of size 7 and less where the nodes can be one of two colors, for total of 72296 states. This allows us to compute exactly 0(x;0) for all x E X relatively quickly (in the order of 10 seconds GPU; see $.1.2
4

---

Published in Transactions on Machine Learning Research (04/2025)
Reward complexity: We define three different reward functions, which we hope to be of varying difficulty. The hardest. function, cliques, requires the model to identify subgraphs in the state which are 4-cliques of at least 3 nodes of the same color. neighbors, requires the model to verify whether nodes have an even number of neighbors of the opposite color. counting, simply requires the model to count the number of nodes of each color in the state. We fully define and show the distribution of log R(x) of the respective tasks in $ We can observe that the distribution of log R(x) increases in complexity relative to the hardness of the underlying task !
We verify that our intuitive notion of hardness for these tasks translates to graph neural networks (GNNs). To do so we train standard graph attention networks (Veličković et al. 2017] with varying capacity to regress to the (log) reward functions. We also tried GCNs (Kipf & Welling 2016) and GINs (Xu et al.) 2018). but found GATss to perform significantly better. Because of this, we elected to use GATs for our experiments in this work, as they present the best performance on this graph environment. See Tables D and 2 for the ablation over GNN architectures and soction D.1 for further details. We use a 90%-10% train-test split, and
show the resulting test. error in Figure [[a] We observe that the ordering seems consistent with the difficulty of the task.
Structural generalization: The so-called structural generalization hypothesis posits that function approxi- mators like DNNs capture regular patterns in the data (Arpit et al. 2017 Zhang et al. 2021), i.e. structure. This enables models to make accurate predictions on similarly structured imputs not present in the training set. We reify this intuition by introducing a structured test set, which we generate by picking states with at least nodes and adding all their descendants to the test set, until the test set is of the desired size; see C.13 for details. Generalizing to a "structured" test set this way poses more challenging problem than simply selecting test samples i.i.d. from the data distribution; this is relevant in domains such as molecular graphs Tossou et al. 2023), where the choice of split (e.g. based on scaffold) practically matters.
Metrics for Performance: To evaluate how well GFlowNets model the distribution p(z), we consider 2 distributional metrics: Jensen-Shannon (JS) divergence and mean absolute error (MAE) between log p(z) and the learned log p(:; 0); see $C.4 for details. Note that when generating graphs, taking into account isomorphic actions is essential to learning the right p(x;; 0) (see Ma et al.) 2023 for reference).
Hypergrid and Sequence Environments: While the hypergrid and sequence environments have been commonly used to sanity-check GFlowNet implementations and methods, we believe them to be too simplistic to leverage the generalization potential of the DNNs, as will be seen in our experimental results.
2.2 Difficulty of Tasks is Preserved When Training GFlowNets Online and On-policy
We verify that training GFlowNets on our proposed graph environment and defined tasks yields the expected task ranking. We train GFlowNets online with SubTB(1) and a uniform PB- To lower bound the modeling complexity, we add a fourth reward, one where R(x) = Vz E x Note that for this specific experiment there are no states hidden from the model, i.e. no test set, since the model is able to sample from its policy and explore the entirety of the state space. Results using test sets are presented in 4..1 Here, we only verify the difficulty of the "usual" GelowNet setup for these tasks.
In Figure I(b) we observe that the ordering of tasks is more or less conserved, but depends on the metric we use to measure the discrepancy between p(x) and p(x;0) f). Using a constant reward indeed lower bounds other rewards, but not by much. This suggests that there is, unsurprisingly, inherent difficulty in modeling the dynamics of the enviromment. We find that there are two main axes of difficulty with respect to the task (aside of course from the scale of the problem, which here is kept constant): (1) how difficult the reward is to model, and (2) how the reward is distributed, Hopefully, Figure [(a) and Figure 1(b] are convincing evidence of (1). We will come back to (2) later in $42
For example, the relative quantity of high reward states ("sparseness" of rewards over states) and the general distribution of log R(x) ("discontinuity" of reward distribution) factors that can drive learning difficulty.
NNoe that, hypothetically, this may be harder than other rewards. For constant reward, the model has to learn to put equal probebility mass cverywhere, which mcans being able model the entire state space. In contrast, more complex but "peakier" reward functions may be in some sense easier to "get right" if one cares more about modeling high-reward states, since there are presumably fewer of them.
5

---

Published in Transactions on Machine Learning Research (04/2025)
Readers are hopefully now on board that this benchmark is sufficiently complex, and is useful for empirical validation and hypothesis testing purposes. As mentioned, we also consider 2 non-graph environments, a hypergrid environment and sequence environment, with reasonably sized states spaces such that we can tractably compute p(z; 8) exactly. This allows us to investigate GFlowNet generalization behaviours in contexis other than graphs. We use lasks and reward functions used in prior GFlowNet work (Bengio el. al.
2021
Malkin et al. 2022: Jain et al. 2023). See C.2 for details.
3 A Method for Disentangling Generalization Mechanisms of GFlowNets
We consider 3 main experimental scttings: (1) distilling (regressing to) flow functions, (2) memorization gaps in GFlowNets, and (3) offline and off-policy training regimes. Each experimental setup built. on a series of simplifying assumptions, allowing us control for different moving parts and complexities that are present when training GFlowNets. We now present this experimental protocol, and then report and discuss our findings in
3.1 Distilling (Regressing to) Flow Functions
Just as we can compute the distribution p(2; 0) over X' exactly in this environment, we also compute edge flows F's s') and (s'sss exactly. We refer to these as the true flow and forward policies, in opposition to the approximated F(s - 6); and Pr(s' s;; 6). Note that we parameterize F(s - 8';0) and Pp(s'|s; 0) mappings from S + glm(s)) with nss) the number of children of s.
In this set of experiments, we train DNNs by regressing to the true flow 4 F(s + s') and forward policy P=(('(s). This removes the use of the GFlowNet training objective, and instead of the training distribution coming form trajectories sampled from Pp(s'[s; 8) we sample s from the training set, drastically simplifying the training procodure. As such, we basically control for any non-ideal factors within the GFlowNet training setup, such as shortcomings due to temporal credit assignment (Malkin et al., 2022).
To regress to log F(s + s') log (s both vectors, regress to each value independently and minimize the mean squared error. Note that the true (s(s'(s)= = softmax(log F(s - s')). Regressing to log F(s + s') is thus almost like regressing to log (s'(ss) but the model has to get the absolute magnitudes of each logit right, not just the relative ones (the softmax in Pr normalizes). Because of this, we expect that learning P=(s'|s) is easier, although learning flow magnitudes could help with generalization. We set PB to be the uniform policy to get unique and Pp.
Using this setup we are able to assess how fundamentally difficult these flow functions are to learn. Furthermore, can use our test set of unseen states. This allows us to assess generalization performance for learning p(z) (which we are able to tractably compute) by directly computing distributional errors. Because of the use of a test set, this is in some sense more challenging than when training GFlow directly, iee. online and on-policy, since standard training of GFlowNets allows the model to potentially explore the entire space and "see" the entire datase
Overall, in this setting we can probe the question of "do" GFlowNets generalize when learning F or PF and assoss Hypothescs 2 3 In the following sub-scction, we describe second approach that can help investigate some mechanistic intuitions of why flow functions might induce generalization in GelowNets, probing the question of "why do" GFlowNets generalize.
3.2 Memorization Gaps in GFlowNets
We would like to probe our hypothesis the contribution to generalization of learning flows in GFlowNets. We take the perspective of /Zhang et al. 2017 2021 and consider generalization the act of not memorizing. In other words, we can assess whether a model is generalizing or not by measuring the gap (in training
'In this sense, we are "distilling" the true flows into flow functions parameterized by DNN. &Readers may notice that because we've computed the regression targets F' and Pp exactly using the entire state space, some information the test set leaking into the targets. Since the purpose of this experiment to assess how hard these functions fundamentally are learn, we knowingly allow the model cheat.
6

---

Published in Transactions on Machine Learning Research (04/2025)
performance) between training it on structured data and training random unstructured data. We use this notion of memorization gap to examine how learning flows with or without structure impacts generalization.
We devise an experiment inspired from Zhang et al. (0117) 2021), where we train supervised models by regressing to R(s) and (s('lss) (using the framework described in the previ- ous section), but with various degrecs of "de- "structuring. Consider simply learning to predict R(s) with R(s; 0). If instead of regressing from 8 to R(s) we shuflle the reward labels of each state s, we end up with new (s, R(s)) pairings, This induces independence between the data pairings, i.c. s 1 R(s). Intuitively, training DNN with I. R(s) will force it to memorize, having removed structure in the mapping from to R.
Table 3: High level summary for the memorization gap ex- periments. Each row lists an individual experiment showing the corresponding data pair coupling and structure of learn- ing problem. These experiments consider models trained with
learning (not via online training using (1)TB(1)) (s(ssss denotes the policy logits derived from the shuffled rewards.
Learning (Regressing to)
Reward (data) Flow (environment)
Data Coupling
Structure
Structure
R
a R(s)
R
a1 Ris)
X
x
Pr
(f.60) Po(elee)
V
Pr
(8,69 Pa(sf19)
X
Pr (8, 1. Ppanti
X
Now consider regressing to Ps(''ss;; there are two ways to destructure this function. Recall that F(s) (and so implicitly P=(s'(s) is a function of the reward of all its descendants as well as of the transition structure the state space (the ways to get to those descendants). Let D(s) be the set of descendants of s. By shuffling R into R. we thus remove the dependence between and R(sa)Vsa D()), but keep the dependence between and how to get to D(s)- The second way to de-structure PF is to simply regress to random logits, which we will denote P
Prandom.
To recap, we regress to: R(s) using paired data, R(s) using shuffled data, PF(s'(s) ()) using paired data, PF (s'(s) using shuffled data, and (s'ls) using random logits Prandom from a magnitudo-preserving range. This is summarized in Table 3
Through this setup, we can assess the memorization gap that occurs when training models. Destructuring the reward but maintaining flow structure in the learning problem allows to assess the contribution that learning true flows has on generalization (i.e. not memorization) when training GFlowNets. Following from Hypothesis 2 if learning structured flows reduces the degree of memorization, perhaps flow prediction inherently acts as mechanism for generalization in GFlowNets.
3.3 Offline and Off-policy Training Regimes
Lastly, we investigate the effects of deviating from the self-induced training distributions of GFlowNets on generalization. To do this, we consider the setting of training GFlowNets of- fline and off-policy given a known dataset of final states X. We sample x ~ Px, where Px is some training distribution over X. We consider different distributions for Px that resemble dif- ferent practical approaches and techniques that are used for training GFlowNets (see Table 4 This setting controls for the effect of sampling from Pr |s; 0) in training GFlowNets.
Table 4: Details for different training distributions Px for offline and off-policy experiments.
Fx
Resemblance to Sumpling
Uniform Log-rewards
M(x)
L.d. from an ideal policy
R(a)
Prony for on-policy
p(a:0)
P=(*"]x:0)
Absolute crvor
(p(:.0) p(r)
absolute loss
Squared log-orvor
( logp())))
squared log-loss (e.g. SubTB(1))
Given a terminal I, we sample a trajectory 7 = .88....,;))) backwards using a uniform Pr Hence, we term this as offline training, since we are avoiding sampling x using Pr(s'|s; We use a standard GelowNet objective, SubTB(1), to train Pr(s'|s; 0) using those т. Note that this setup can remove the data non-stationarity that normally exists when using on-policy (())(s;() samples.
We also consider the online off-poliey setting, to assess to what degree deviating from (s) during training will affect learning p(x). Understanding this may be useful to develop novel GFlowNet algorithms,
Investigating the use of different Pas is another interesting direction that may give insight into the effects of the backward policy on the training dynamics of GelowNets (see Shen al. 2023). We leave this for future work.
7

---

Published in Transactions on Machine Learning Research (04/2025)
10a
15 Divergenca
10a
counting
neighbors
10
MAF MoFacceer MAF
10
counting neighbors
10-2
10 4.
cliques
10
-10-3
-10
10-3
-10-
10-
10a1
-10²
1f exammlesssee seen 167
examples seen
exammles seen
* exammples seen
(b)
Figure 2: (a) Training model to distill edge flows and policies. (i) doing so recovers the intended distribution, (ii) modeling PF appears easier than modeling F. (b) JS-divergence and MAE gaps between SubTB(1)
trained model and Pp distilled model. Distilling to Pr appears to yield lower distributional error, except for JS-divergence on the cliques task.
considering that in practice only few settings that deviate from on-policy training work (Rector- Brooks et al,, 2023). To achieve this, we consider a policy interpolation experiment where we sample trajectories from (s(s'(s) (1 a) (s('|s: f) + (P)((s')s), where 0 I 1 and Pu(('(s) denotes Pp that goes to every terminal stale with equal probability, i.e. such that p(x) 1. We do this because, as will see, any degree of deviation from (p(s'|ss [s; 0) during offtine training appears to have some negative effect. We observe this even for values of a close to 0. We discuss this in further detail section 4.4
This experimental setup helps us investigate the isolated effect of different data distributions Px on training dynamics and generalization by changing the data distribution (and non-stationarity) typically induced by P(s'|s;0) 0). We consider both settings with and without test set. This allows assess Hypothesis [ by observing the effects of various deviations from the normal self-induced training distribution coming from ())
4 Experiments
In this section, report and discuss observations and findings of the experimental methodology defined in $3 We conduct all experiments reported in this section over 3 random sceds. For training online and offline GFlowNets, we use SubTB(1) (see D.2 and uniform PB.
4.1 Distilling Flow Functions
We run the setup explained in $3.1 To evaluate the models, we look the distributional errors on p(z; 0). We 90%-10% train-test split. We report results in Figure 2(a) and Figure 2(b)
see that learning Pr and learning F yield similar difficulty in the sense that the distributional First, systematically close. Second, we see that the model is generalizing, getting sometimes even better
eтrors are
scores than a model trained online on the entire states space (i.c. with no test set) as seen in Figure 2 While this result should be obvious, confirms that the difficully of training a GFlowNet comes from both learning to model the flow functions themselves and performing temporal credit. assignment.
Obscrvation 4.1. Flow functions and flow policies are learnable and standard neural networks generalize when predicling them.
We confirm these observation in sequence and grid environments (sce EE.1 Further results $D.3 assess how well GFlowNets (and TB Malkin al. (2022) and FM (Bengio al. 2021 generalize in rank in the graph tasks.
8

---

Published in Transactions on Machine Learning Research (04/2025)
Distilled Divergence Distilled llop, kopl
GFN1S Divergence Online GFN log logp)
+
Distiled js Divergence Distiled -lops
Online GiN is Divergence Online GFN
Counting
Neighba
Cliques
Counting
Nei Norghbors
Cliques
10-1
10-4
10-1
10a
10
10
10
10a2
10a4
0.6
0.5
Y
(a)
(b)
Figure 3: (a) Training models distilled to Pp and models trained online/one coninico for a range of y (b) similarly for range of 3. Transforming the distribution of the reward does not significantly affect the generalization difficulty in approximating p(z).
Counting
Neighbors
Cliques
Nomalized Loss
Learning P:
Normalized Loss
True R
Normalized Loss
Learning
Shuffled
prandom
examples
1e7
examples
1e7
# examples
1e7
Figure 4: Memorization gap training curves for counting, neighbors, and cliques tasks. Maintaining flow structure in the learning problem (learning Pr under shuffled R) induces a smaller memorization gap relative to the fully de-structured setting. See Table for reference experimental setup.
4.2 Reward complexity and transformations
Consider monotonic transform H. IR that maps log (s)s Hly R(s)), Vs E such that log (s(s) is skewed function of y (see C..3 for further details). We use the setup in $3.1 and also train online GFlowNets in the setting of skewed reward distributions transformed by Hy as well as tempered distributions via a 3 parameter, log R'(s) ())= = 3 log Rs)). Results are in Figure 3(a) as we change and 3, both distilled and online models are not significantly affected.
Observation 4.2. Monotonic transformations, such skew and reward tempering, do not have detrimental impact generalization when learning p(x) with GFlowNets, for the range of hyperparemeters considered.
This implies that applying minor reward transformations, which a standard technique in GFlow Net training, is not detrimental to generalization about p(x) for a given task (of course, e.g. sparsifying a reward makes exploration harder), supporting Hypothesis 3 We repeat this experiment on the sequence environment and
find consistent results (see E.
4.3 Memorization Gaps in GFlowNets
We run the setup explained in $3.2 and Table 3 results are shown in Figure Additional results with R(s) ((,, a) and online trained GFlowNets are in $D.5 A typical observation in the memorization gop setup is for the training loss to initially plateau, until some phase change occurs (one can imagine parameters have self- arranged to create separate linear regions around each input point) and the training loss goes to 0. and PF induce curves with no plateau, while regressing to R and
We similar trends here, regressing prandom initially have very flat plateaus until phase change occurs. This shows very clear memorization gap, which expected.

---

Published in Transactions on Machine Learning Research (04/2025)
x- (p0x d) -P*M]
x- (log (og 90 ROp '000
.... X RiO
Tra ned using all states The ned using 90% states
15 Diveggoic
10-4
( hopol
[bo logat
10*
Ful joy 1og4
10f
10
2
2
Mcatples WAMRT
# esamples SHOT
examgles soeh
esamptes soch
(a)
(b)
Figure 5: Evaluation curves for offiine and off- off-policy trained GFlowNets on the neighbors task for different choices of 1x- (a) When training using the full dataset (no test set). (b) When training using subset of the full dataset (90%-10% train-test split). Complete experiments for all graph generation tasks and evaluation metrics are shown in D.6
Most interesting where Pr is partially de-structured, and we regress to the true Pr of shuffled reward R and retain the environment structure. While there no plateau, Pe becomes harder to fit, and there remains phase change. These results support Hypothesis 2 PF captures structure of both the reward function and the environment, since removing either induces memorization gaps.
Observation 4.3. The existence of clear memorization gaps, when removing reward structure but mainlaiséng environment structure, upholds that environment structure facililates generulization when learning flow functions and flow policies.
This finding implies that state flows may play a mechanistic role in why GFlowNets generalize to unseen states. We also show similar results for sequences and grids (see E.3
4.4 Generalization in Offline and Off-policy Training of GFlowNets
We the offline training sctup described in $3.3 and Table results on the neighbors task are shown in Figures 5(a) and 5(b) Results for all tasks are in $ DD.6 First, observe that performance is dependent on environment / task difficulty, the choice of xxx, and on the evaluation metric. For example, the best choice for Fx on the counting task is typically x ~ ll or x ~ log R(x), while this is not always the case for neighbors
and cliques (sce $D.6 Figure 13
Second, we observe that when training Pr(8'|s; f) offline and off-policy in the full data (no test set) regime, convergence rate heavily depends on the choice of Px (see Figure 5(a)), sometimes a catastrophic degree. We see this for our usual distributional metrics for approximating p(z)) [ However, when considering the MAF between log R(s) the approximated log R(s), we do not observe this lack of convergence (GFlowNets implicitly learn to predict the reward: log R(s) = log F(s;0) + log pp(sy|s; ()) Although training Pr (s |s; f) offline or off-policy may be tricky, learning Fs;; 0) jointly with PF(s'|s; 0) might be beneficial: we observing a reasonable estimate for log A(s) even when (s (s 0) fails to adequately model p(x) the model is learning something.
We then run the off-policy interpolations (as described in $3.3), with P. - -1 aF + aPu and plot the results in Figure 6. We observe trade-off between on on-policy and off- policy training when estimating p(z) versus R. We find that for harder tasks (neighbors, cliques), making the data more off-policy (a 1) makes p(x; 8) become worse and R better. When sampling on-policy (from Pp (s)]s;)), a = 0), approximating p(x) is reasonable, but approximating R is best when sampling uniformly off-policy (from Ps (s), a = 1).
Observation 4.4. Deviating from the on on-policy training distribution induced by Pr (s'1s; f) can negatively affect the ability of GFlowNels' to learn p(x)).
II possible that the drawbacks of this training regime overwhelm our ahility make meaningful conclusions on the choice of Px for offline training the neighhors and cliques tasks.
10

---

Published in Transactions on Machine Learning Research (04/2025)
counting
neighbors
cliques
Divergence
10
10
10
10
10
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
1e7
1e7
1e7
10-1
xx10--1
llogp lag
10a
100
10-1
60
10
1
x 1010
1.5
2.0
1.5
1c7
1e7
1c7
20-1
llog log
10-1
10-4
60-0
40-2
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
examples seen
1e7
examples seen
examples seen
0.0
0.2
0.4
0.6
Figure 6: Results for policy mixing interpolation experiments. Here, we mix (P'sss where S I 1. For a = 0, the model is fully on-policy, sampling trajectories from ( (s' 0). For a = 1, the model is sampling fully off-policy using Ps(sss)). This approximates the behaviour of training offline and off-policy when using x ~ (=)) (i.c. uniform sampling of z). Here we can observe the effects of deviating from on-policy samples during training. We see that at times, specifically for the difficult tasks small degrees of deviation from (s's'|s; 0) can lead to worsened performance when approximating p(x) (scen by sensitivity in distributional JS divergence and MAE metries). In contrast, approximating R appears to improve as the model samples more uniformly over X (if not entirely uniform, i.c. when a = 1). Interestingly, the counting task appears generally invariant to minor changes in a.
(neighbors and cliques),
This implies that training Gels oflline or off-policy may come with challenges. However, the model could be learning some informative structure in F(s) that helps it generalize (seen by reasonable log R(s)). We test this hypothesis in our second offline and off-policy training setup, and consider performance test set (Figure 5(b).. We observe that offline and off-policy trained GFlowNets reasonably approximate R on unseen states even when failing to adequately model p(x).
Observation 4.5. Learning Fss) can facilitate models learning to implicitly predict the reward on unseen states, even when Pp does not adequately approzimate p(x:) on these stales.
These results support Hypothesis training GelowNets online and on-policy is ideal, where Pr(s'|s; converges to sampling in proportion to R. We repeat these experiments for the sequence and hypergrid
11

---

Published in Transactions on Machine Learning Research (04/2025)
environments, where we observe that p(æ) is adequately modelled in the offiiee and off-policy cases (see F.4pp This may be due to the lesser complexity of these environments, which we establish by comparing the difficulty of our tasks to that of fitting a constant reward (see $E.1
5
Conclusion
In this work, we conducted an empirical investigation into the generalization behaviours of GFlowNett. We introduced set of graph generation tasks of varying difficulty to benchmark and measure GFlowNet generalization performance. Our findings support existing hypothesized mechanisms of generalization of GFlowNets as well as add to our understanding of why GFlowNets generalize. We found that GelowNets inherently learn to approximate functions which contain structure lavourable for generalization. In addition, we found that the reward implicitly learned by GFlowNets is robust to changes in the training distribution, but that GelowNets are sensitive to being trained off-policy.
Discussion: Our objective in this work to take scientific approach of starting from existing set of hypotheses and testing them through more comprehensive set of benchmark lasks. For example, many existing works claim that GFlowNets can effectively operate in the off-policy regime. However, through our investigation, we show that GFlowNets are not as robust as the tone of those claims would suggest. We demonstrated this by introducing a novel graph-based benchmark environment where task difficulty can be easily varied while also varying the off-policy sampling distribution. We repeated our empirical investigation using standard environments which are used in GelowNet literature (i.e. the hypergrid and sequence environments). Interestingly at times, and particularly for the off-policy results, there is a discrepancy of the observed behavior of GFlowNet generalization performance between the hypergrid and sequence environments with performance on our graph-basot environment. This result suggests the importance of taking into consideration the choice of benchmark environment when developing novel GFlowNet methods.
Our intention was to systematically test existing hypotheses regarding GFlowNets and their ability to generalize to unseen area of the state space. In testing these hypotheses, we confirmed existing knowledge while also finding contradictory outcomes from what has been previously understood. For example, while Observations 4.1-4.3 confirmed our existing received knowledge regarding GFlowNets and their generalization capabilities, Observations 4.4 and 4.5 were unexpected results. We believe these are interesting findings that warrant further investigation and could for example lead to improvements in off-policy training and exploration algorithms for GelowNets.
Limitations: We have shown that GFlowNNets tend to generalize when learning unnormalized probability mass functions for approximating p(x) over discrete spaces. In particular, we investigated the generalization behaviours of GFlowNets in the context of distributional errors for p(x). Because of the combinatorial nature of computing p(z) exactly, and likewise computing p(r; 9) from ()) we are limited in the fact that need to work within reasonably sized discrete spaces (as those we have proposed) to study Gel generalization for approximating p(x). Secondly, we don't explicilly test generalization in online and ou-policy trained GFlowNets. This in part consequence of requiring an environment and state space large enough such that a sufficient quantity of unseen states can be produced, while also being able to purposely hide the visited states from parameters updates of the GFlowNet. Lastly, although our proposed problems and environments have structure that induces difficult and interesting generalization tasks, they still may remain far from the structure of the real world.
Future Directions: We hope to have formed sound set of findings and observations that may lead to future research in understanding and formalizing generalization in GFlowNets. For instance, our set of findings could be used as starting points for formalizing some of our notions and intuitions for Get generalization (and the corresponding mechanisms) into mathematical thcory. Another direction that can stem from our work is to further empirically investigate GFlowNet generalization in the online and on-policy setting. Since we don't explicitly look at GFlowNet generalization in the online and on-policy training regime (i.c. since we don't hide any states from online trained GFlowNets); would be interesting in future work to explicitly investigate this. However, it is important to note that this is not necessarily a trivial task (as described in limitations).
12

---

Published in Transactions on Machine Learning Research (04/2025)
Acknowledgments
The bulk of this rescarch was done at Valence Labs as part of an internship, using the computational resources there. The authors are grateful to Yoshua Bengio, Berton Earnshaw, Dinghuai Zhang, Johnny Xi, Jason Hartford, Philip Fradkin, Cristian Gabellini, Julien Roy, and the Valence Labs team for fruitful discussions and feedback. In addition, we acknowledge funding from the Natural Sciences and Engineering Research Council of Canada and the Vector Institute.
References
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Einannel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International Conference Machine Learning. PMLR, 2017.
Lazar Atanackovic, Alexander Tong, Bo Wang, Leo J Lee, Yoshua Bengio, and Jason Hartford. DynGFN: Towards bayesian inference of gene regulatory networks with gflownets. Advances in Neural Information Processing Systems, 36:74410 74428, 2023.
Emmanucl Bengio, Joelle Pincau, and Doina Precup. Interference and gencralization in temporal difference learning. In International Conference on Machine Learning. PMLR, 2020.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381-2777 1-27344 2021.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet. foundations. Journal of Machine Learning Research, 24(210):1 55, 2023.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning, pp. 1282 1289. PMLR, 2019.
Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks. In Uncertainly in Artificial Intelligence, pp. 518-528, PMLR, 2022.
Tristan Deleu, Mizu Nishikawa- Mooiik, Jithendaraa Subramanian, Nikolay Malkin, Laurent Charlin, and Yoshua Bengio. Joint bayesian inference of graphical structure and parameters with single generative flow network. arXiv preprint 0rXii:2305.19300 2023.
Tristan Deleu, Padideh Nouri, Nikolay Malkin, Doina Precup, and Yoshua Bengio. Discrete probabilistic inference as control in multi-path environments. arXiv preprint arXiv:2402. 10309, 2024.
Moksh Jain, Emmanuel Bengio, Alex Garean, Jarrid Rector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, ct al. Biological sequence design with gflownets. In International Conference on Machine Learning, pp. 9786-9801. PMLR, 2022.
Moksh Jain, Sharath Chandra Raparthy, Alex Hernández- Garcia, Jarrid Rector-Brooke Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. In International Conference on Machine Learning, pp. 14631-14652 PMLR, 2023.
Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. ar preprint arXiuu1110.054088 1(8), 2017.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint XrXe::1609. 029007,027700 2016.
Salem Lahlou, Tristan Deleu, Pablo Lemos, Dinghuai Zhang, Alexandra Volokhova, Alex Mernándoz-Garcn Léna Néhale Ezzine, Yoshua Bengio, and Nikolay Malkin. A theory of continuous generative flow networks. Tn International Conference on Machine Learning, pp. 18269 18300. PMLR, 2023.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward networks with a. nonpolynomial activation function can approximate any function. Neural networks, 6(6):861-8))) 867, 1993.
Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. Deepergen: All you need to train deeper gens. arXiv preprint arXiv:2006.071 2020.
George Ma, Emmanuel Bengio, Yoshua Bengio, and Dinghuai Zhang. Baking symmetry into gflownets. NeurIPS 2023 AI for Science Workshop, 2023.
Kanika Madan, Jarrid Rector-Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, Andrei Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability. arxiv e-prints, page. arXiv preprint or arxi::2009.120 2022.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. Advances in Newral Information Processing Systems, 35:5955-5500 5967, 2022.
Sobhan Mohammadpour, Emnanuel Bengio, Emma Frejinger, and Pierre-Tuue Bacon. Maximum entropy gflownets with soft q-learning. In International Conference on Artificial Intelligence and Statistics, pp.
2593 2601. PMLR, 2024.
Andrei Cristian Nica, Moksh Jain, Emmanuel Bengio, Cheng- Hao Liu, Maksym Korablyov, Michael M Bronstein, and Yoshua Bengio. Evaluating generalization in gflownets for molecule design. In ICI.R2022 Machine Learning for Drug Discovery, 2022.
Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl,, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXie preprint ar 12282, 2018.
Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of gilownets with local credit and incomplete trajectories. In International Conference on Machine Learning, 2023.
Razvan Pascanu, Guido Montufar, and Yoshua Bengio. On the number of response regions of deep feed forward networks with wise lincar activations. arXiv preprint 1rXiv:1812.6008 2013.
Jarrid Rector-Brooks, Kanika Madan, Moksh Jain, Maksym Korablyov, Cheng-Hao Liu, Sarath Chandar, Nikolay Malkin, and Yoshua Bengio. Thompson sampling for improved exploration in gflownets. arXiv preprint trXi::2306.11 17693, 2023.
Julian Schrittwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, el al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839))60) :604 609, 2020.
Max W Shen, Emmanuel Bengio, Ehsan Hajiramozanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving gflownet training. arXiv preprint arXiv:2305.077 07170, 2023.
Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for sunersupe classification. arXiv preprint arXiv:2009. 03509, 2020.
Daniil Tiapkin, Nikita Morozov, Alexey Naumov, and Dmitry P Vetrov. Generative flow networks entropy-regularized rl. In International Conference on Artificial Intelligence and Statistics, pp. 4213 4221. PMLR, 2024.
Prudencio Tossou, Cas Wognum, Michael Craig, Hadrien Mary, and Emmanuel Noutahi. Real-world molecular out-of-distribution: Specification and investigation. 2023.
Ashish Vaswani, Noam Shazoer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, and Yoshua Bengio. Graph attention networks. arXie preprint 1rX0: 10903, 2017.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks?
arXiv preprint X 01100888000 2018.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint ArXi 07937, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communécations of the ACM, 64(3):107-115, 2021.
Dinghuai Zhang, Nikolay Malkin, Zhen Liu, Alexandra Volokhova, Aaron Courville, and Yoshua Bengio. Generative flow networks for discrete probabilistic modeling. In International Conference on Machine Learning, pp. 26412 26428. PMLR, 2022.
15

---

Published in Transactions on Machine Learning Research (04/2025)
A Vocabulary
The p(z;0) distribution, when using GFNs never explicitly model p(æ; 0); instead this measure is induced by the parameterization (F(s s0:0) or Pr(s((s;0) of a sequential constructive policy. We use p(a; 0) as the shorthand for the distribution over X induced by the chosen parmmeterizaionn with parameters 0. See also $C.1.2)
Online vs offline, model is trained online when it is trained from data generated during the training process - typically according to its own parameters, ie. for some model p(z;0), we may use X p(a; to train p(:; f).. Conversely a model trained offine is trained from a (usually fixed) data set, X D. It is possible to form a mixture of online and offline data. GFNs are compatible with this paradigm (unlike RL methods such as policy gradient methods).
On-policy vs off-policy, a model is trained on-policy if it is trained from data generated according to the unperturbed distribution p(2; (), whereas it is trained off-policy if it comes from any other distribution. For example, taking some actions at random would be considered off-policy (although a mild version), but so would taking samples from an entirely different distribution p(x; 0') or from a dataset.
Self-induced distribution, when training a model we refer to the distribution p(æ; 0) as "self- induced This is mainly to emphasize that this distribution changes as changes, which in online on-policy contexts is due to the model generating its own training samples (thus the "self" -induced).
True flows, we occasionally refer to true flows. What we mean by that is the exact calculation of F(s) or -(s as a function the reward R(s) and of PB- Note that for a fixed Ps there unique solution to F (Bengio et al.) 2021). When the reward is corrupted, we talk about the true flow of the corrupted reward R, i.c. F(s) is the exact calculation of the flow through but for terminal (sink) flows set to R.
B Limitations and Future Work
Impact Statement The outcomes of the research conducted in this work have practical implications for generative modeling in applications that include drug discovery, material design, and general combinatorial optimization in commercial settings. Because of this, we recognize the importance of considering safety and alignment in these closely related domains. We believe that advancements in GFlowNet research could result in models with improved generalization, in turn leading to models that are easier to align. Another important and useful application of GFlowNets is in the causal and reasoning domains. We believe that advancements in these areas may lead to more interpretable, easier to understand, and safer models. Lastly, we highlight that the applications of our findings for the advancements of GFlowNets may rely on building upon the hypotheses put forward in this work.
C Experimental Details
C.1 Details for constructing graph benchmark task
C.1.1 Reward functions
Here are the exact log-reward functions we use, as implemented in Python using the networkx (nx) and numpy (пр) libraries.
cliques counts the number of 4-cliques graph such that at least of the nodes of the cliques share the same color. We then subtract to that number the total number of cliques in the graph, but add the number of nodes. This is so that the maximal log-reward is O. We clip lewarss below -10.
16

---

Published in Transactions on Machine Learning Research (04/2025)
Counting
Neighbors
Cliques
1.0
1.2
0.8
0.6
0.4
0.4
0.2
11
-10 -8
-10
-6
-10
-8
-
lagR0x)
logRtd)
Figure 7: Distribution of normalized count of log-rewards over the state space for the given tasks: (left) counting, (middle) neighbors, and (right) cliques.
def cliques(g, x-4);
cliques list wlw.rithhs.......... cwique.. ffiddffiiuuuusseeeeeee The nunber of cliques each nade belongs to nun cliques bp..incouut (aun(eSSqua,, (})
colore {i: g [1] ["y*] for in nodes)
def color_ match(c):
return up bincount ([co((([[oool ([colore (i] for in c)) max() >-
cliques_ natch = (#loat(len(1)) == n) (1 if color ms)ch(s) alse 0.5) for in cliques]
return up wwx.mux (op. (wp..opp..ac..ooo www(cliques_matcee
up. sun(nun_cliquea) len(g) 1, -10)
neighbors looks at all the neighbors of every node, counting the number of nodes with an even number of neighbors of the opposite color. This total is modified function of the number of nodes in order to produce a nice log-reward distribution between 0 and -10.
def neighbors(g):
total, correct:
for n in nua nun_diff_col-
none [n] (*v"
for in g. g.naighbora(n):
nun_diff_coll += [s]g c)
total_correct += int aamm_diiff_ooll X = 0) (1 if nun cum_difffcomm _olr =- 0 else 0) return up. Cttarmnp...fataaee (o..fola33((tttal____11 (ttatattao_r_rrrttttteee correct noneg) lon(g-nodes)
-5)
7)
counting simply counts the number of red and green nodes, red nodes being "worth' more, and again modifies this count in order to produce nice 0 to -10 log-reward distribution.
def counting()):
np. bincount [odoz [*v"]
gl, minlangth-2)
return np. (1oott77aabb ncola[i] [1] / 2 4 10)
C.1.2 Computing p(x;;) 0), F(s), F(s + 8'), and Pf
We compute the probability of model with parameters 0 sampling x, p(x; 0) via what is essentially Dynamic Programming. We visit states s in topological order starting from For every child of s, s', (valid transition s + 8') we add (s 0) to the probability of visiting Po(s'), eventually accumulating visitation probability from all the parents s'. This is possible because we are visiting states in topological order. Note that we start with (s() and P. set to for every other state. p(s; computed as (rsttoos))
The above is fairly easy batch, requires one forward pass per state. Computing PF is the most expensive operation and can be batched, as long as values are accumulated by respecting topological order afterwards. We also actually store values on a log scale, using logaddexp operations for numerical stability.
To compute true flow functions, we do the reverse, visiting the DAG in reverse topological order. For every state 8, we add (ss'ss) units of flow to every parent s' of s, starting at leaves (due the reverse topological order) where F(s) R(s). We similarly set the value of the edge flow F(s > s') PB(ssls)) Pr is simply the softmax of edge flows. Note that we uniform Pn, as explained in the main text.
17

---

Published in Transactions on Machine Learning Research (04/2025)
C.1.3 Constructing test set
Why 6 nodes? Considering the maximal number of nodes is 7, 6 may seem "too close". On the other hand, picking a graph and exchuding its subtree means that this subgraph will never appear in the training set. This is quite aggressive, for example creating a test set 10% of the data only requires 35 such graphs whose subtrecs are exchuded. These graphs have an average of 272 205 descendants. Choosing graphs of 5 nodes would exclude average of 4700 graphs per pick, meaning that test set of 10% of the data would only stem from the exclusion of 2 or 3 graphs. This may not have the diversity desire, thus choice of 6 nodes.
Finally, we believe this is an interesting choice that relates to real-world applications of GFlowNets. In drug-discovery small molecules, given the enormous size of the state space, it likely for most subgraphs of a sufficient size to never appear in the training set (because model can only train for so many iterations in practice). In a graph generation context we are thus interested in how a model generalizes to subgraphs it's never seen. In this proposed benchmark, 67994 of the 72296 states graphs of 7 nodes; only excluding graphs of 6 or 7 nodes thus covers most of state space.
C.2 Details for hypergrid and sequence environments and tasks
C.2.1 (M x M) Hypergrid:
Grid environments have been used in GFlowNet, papers since their inception (Bengio et al. 2021 as a sanity check environment, presumably inheriting this custom from Reinforcement Learning. We continue the tradition and report results on 2D grid environment. In it, the agent has 3 actions, move in the a+ grid of size M (64 in experiments),
direction, move in the y+ direction, stop. The agent navigates and is forced to stop if any of the coordinates reaches M 1.
As reward signal we use functions used in past work [Bengio et al. 2021 Jain et al. 2023
C.2.2 Bit Sequences:
regressgee sequence environments have been used in variely of past GFlowNet work (Malkin al.
2022 Madan et al. 2022 Pan et al. 2023). In this work, we consider a bit sequence environment akin to that originally introduced by Malkin et al. (2022). We consider reward of the form R(x) exp( - miinmem d(z.m)) 10), where d is the Levenshtein edit distance between input sequence x and the closest "mode" sequence m, M C A' is the set of mode-soquencos, and I is the max sequence length. We seleet |M] = 60 sequences from M uniformly at random given the set X' of all possible bit sequences to length / = 15 to be the "mode" sequences (we do this once at the start of each run). This yields a discrete state space of 65, 535 states, comparable in size to our graph generation environment.
C.3 Monotonic reward transformation
We consider a monotonic transform of the form Hlooo R(x)) = log R(x) log R(x). As described earlier, the transform H,, maps log Rs(s) (log R(s)), Vs 8 such that log R'(s) skewed towards higher reward states function of some parameter 7 (sce CC.3 for further dctails. Note, H, is monotonic for values of log R(x)= 0 and 7 2 0. In graph and sequence generations tasks, log R(x) < 0, Vr E X, hence are able to use H, monotonic transform for log R(x). We show an example of H, applied to the cliques task in Figure for various value sof 7.
C.4 Evaluation metrics
To measure the generative modeling performance of GFlowNett we consider the Jensen-Shannon (JS) divergence and the MAE between the learned p(;; 0) and the true log 1(x). For divergence, we compute:
I p(<; = {L(p(=)] + [L( 0)
(3)
18

---

Published in Transactions on Machine Learning Research (04/2025)
log
0.8
log ((x))
logA'(4)
-4
0.4
-6
Y
V
==0.0
0.0
-10
-8
-6
4
-2
-10
-8
-6
-
lagR(x))
logR(x)
Figure Example of monotonic transform Hy. (left) distributions of log R(x) and log (x) for true log- rewards and skewed log-rewards (using 7 0.2). (right) plot of log R(x) versus log R' (x) for various values of 7. Points above the ? line yield skew towards higher log-reward values.
where Q = (p(x) + p(x; 0)). For the MAE, we simply take the absolute error [ogg((;; 0) - log p(x) and average over the cardinality of the state space.
C.5 Model architectures
For all graph experiments we use a modified graph transformer (Veličkovi el al. 2017 Shi et al. 2020 On top of the normal attention mechanism, we augment the input of each layer with the output of one round of message passing (using layers from the work of Li et al. 2020) with sum aggregation-wo found that this useful in tasks where counting was required. We 8 layers with 128-dimensional embeddings and 4 attention heads; we use this architecture after having tried difffernnt numbers of layers and embeddings in order to validate task; this is coincidentally represented in Fig, [(a)
For sequence tasks, we use vanilla transformer (Vaswani et al. 2017) with 4 layers of 64 embeddings and 2 attention heads. We did not search for hyperparameters, since this setup has been used in prior work (Malkin et al. 2022).
For grid tasks we use a LeakyReLU MLP with 3 layers of 128 units. The input one hot representation of each coordinate.
We use the Adam optimizer with learning rate 0.0001 for all models.
C.6 Implementation details
Our experiments are implemented in Pytorch and Pytorch Geometric. All experiments were run HPC cluster of NVIDIA A100 100GB GPUs for total of approximately 2000 GPU hours. Only 1 GPU is required for Eeach individual seed run of experiment, typically taking between 24 hours to 3 days to complete, depending on the experiment. Throughout the conception of this work, there were small set of preliminary and failed experimental runs as ideas were being formed.
D Additional Graph Task Experiments
D.1 Graph neural network architectures for GFlowNet graph task experiments
We tried a few GNN architectures. GATs appeared the best. While they are not the simplest GNNs, they may be the most general (because of the attention mechanism). We compared GATs, GCNs and GINs the experiment Figure (i.e. simply training GFlowNets online on-policy)s. We were careful to ensure that all models have almost-oqual numbers of parameters. We find GATs to be superior, but we acknowledge that this may not be the case in tasks beyond this benchmark, and that we have obviously spent more time tuning our GAT model than these. We believe unlikely that the choice of model would affect observations.
19

---

Published in Transactions on Machine Learning Research (04/2025)
D.2 Other GFlowNet objectives
We elected to consider SubTB(1) since it has been shown to yield improved results over TB and Detailed Balance (DB). Furthermore, we consider the Trajectory Balance (TB) and Flow Matching (FM) objectives in set of offline experiments to assess how GFlowNets assign probability mass to unvisited arcas of the state space (see D.3) In Table 5 we show that SubTB(1) performs best (compared to offine TB and offline FM) when ranking states in the graph tasks.
D.3 GFlowNets learn rank preservation of unseen states
Table 5: Comparing the rankings given by different ways of training a model on a dataset. Standard deviations
are over runs. In the last three rows the number in brackets is the number of optimal objects in the test set.
Task Supervised Distilled F Distilled Pr offline TB
offline subTB
offline FM
Test set Spearman correlation
cliques
0.92 0.02
0.92 i 0.01
0.93 1
0.88 0.01
0.90
0.86 0.03
neighbors
0.98 0.00
0.98 + 0.00
0.98 = 0.00
0.96 0.00
0.96 0.00
0.96 0.00
count
0.98 +: 0.00
0.98 + 0.00
0.98 + 0.00
0.95 0.01
0.95 0.00
0.96 0.00
Test set top-100 Spcarman correlation
cliques
0.66 = 0.02
0.74 = 0.10
0.83 + 0.04
0.67 + 0.10
0.69 + 0.11
0.53 0.13
neighbors
0.87 + 0.00
0.87 + 0.00
0.87 = 0.00
0.86 + 0.01
0.86 + 0.00
0.86 = 0.01
count
0.44 = 0.00
0.44 + 0.00
0.44 = 0.00
0.24 + 0.06
0.24 + 0.06
0.37 0.02
Avg Rank of optimal objects
cliques (15)
21.5 14.5
39.3 = 30.2
13.3 5.3
18.1 + 4.9
12.1 - 2.0
17.7 5.2
neighbors (48)
23.5 0.0
23.5 1
23.5
43.4 =
43.6 1
40.7
count (7) 3.0
3.0 +
3.0 +
62.7
61.1
25.6 +
We want to assess more precisely where GFNs put probability mass, in particular in states they 've never visited. More specifically though, GFNs are often used to find likely hypotheses, i.c. generate objects "close enough" to the argmax(es).
Consider the following scenario, which is a common way use GFNs. Some dataset of objects and scores D {(1
):) } is given to us. We train reward proxy to regress to R(x;; 0) Yi. and then train GFN R(x; 0) in order to generate as (and commonly, find the most "interesting" z). If we assume that supervised learning is "as good as it gets" to approximate R, then the task of finding arg max, R(x) by finding arg max, () should also be as good as it gets. This prompts us to ask, how close are GFNs to this idcal?
In the following experiment, we look at three measures. First, the Spearman rank correlation on the probabilities p(2;0) 0) (or R(x;)) 0) for the supervised model) of the test set, second we do the same for the top-100 graphs in the test set, and finally we look at the mcan rank of the set of optimal in the test set (e.g. if there are 7 optimal graphs in the test set and the model perfectly predicts their rank to be [0, 1, 2, 3, 5, 5] then the mean rank is (0 (0-14 1 2.. + 6)/7 =
These measures are intended to be proxies for how likely it is that a model trained on some dataset would be able to generate objects close enough to the optimal objects.
This third measure is also inspired by the following fact: it has been observed in a few setups that even though trajectory balance objectives appeared to fit the distribution much better than flow matching, flow matching can be more efficient at producing high-reward samples (which seems counter-intuitive)
We thus compare distilling edge flows to policies, as well as TB objectives to the FM objective. Specifically for GFN objectives (TB, subTB, FM), we use an oflline paradigm where we sample trajectories by sampling an x uniformly at random from D and going backwards with P3 uniformly at random (this is presumably a somcwhat idcal training condition).
We report the results in Table 5
The results are quite dense, let make few observations:
20

---

Published in Transactions on Machine Learning Research (04/2025)
When considering the entire test set, the supervised model is indeed the gold standard. Offline GFN models are slightly worse. This isn't too surprising, and is in fact reminiscent of results related to the difficulty of training value functions via TD in RL (Bengio et al. 2020).
When considering either the top-100 or the optimal objects, we see surprising results for the cliques lask (the hardest task)
for the top-100 objects, the distilled models do better, and the offline TB models are on-par with (and occasionally better than) the supervised model
for the rank of optimal objects, not only does distilled Pp do better, the offiine GFN methods do better as well.
Recall that some test set information is leaking into training; it may explain the advantage of distilled models, but not the advantage of offiine GFN models.
TB and subTB generally better at fitting ranking than FM, except when it comes optimal objects. This reproduces past observations, suggesting flow matching over-weights high-reward states, even outside of its training data; we unfortunately do not have coherent explanation:
ss) distillation being worse than Pp distillation means modeling F is probably not the advantage;
This phenomenon is consistent across training, although not on every task.
This phenomenon is amplified when not correcting for idempotent actions. This is expected Ma et al. 2023 but useful to measure.
We repeat these three measures with different train-test ratios, shown in Figure 17 where we see that the results are expected, and consistent with Table 5 To summarize: GFlowNets impticitly learn to put more mass high-reward objects than low-reward objects outside their training set.
The above is interesting, but is still just statement about ranking between unseen states. Obviously this says nothing about the total probability mass given to unseen states.
21

---

Published in Transactions on Machine Learning Research (04/2025)
D.4 Distilled flow experiments over only unseen states
constant counting neighbors
counting neighbers
10"
cliques
cliquis
-loppl
Joggo loqul
10
10-1
10-3
10-1
0.0
examples seen
167
examples seen
(b)
Figure 9: MAE error for only unseen states on constant, counting, neighbors, and clique graph generations tasks. (a) evaluation for model trained to distill edge flows and policies. (b) evaluation online trained GFlowNet via SubTB(I). We see that considering only the unseen states for MAE metric leads to comparable observation to the case of evaluating MAE over all states. This is expected since error on the unseen states should be driving the model's overall evaluation performance when evaluating on the entire state space. Note, for the online GFlowNet, there is no guarantee that the model has not "seen" the left out unseen states in this evaluation.
16a
10
MAEauuu MAEgi
constant
counting neighbars
cliques
-10-
-10
10-1
-109
asamples sean
Figure 10: MAF gap between online trained GFlowNett via SubTB(1) and distillation trained model regressing to Pp on only unseen states. We observe that when considering only unseen states, results are consistent with evaluation over the entire state space. This is also expected since error on the unseen states should be driving the model's overall evaluation performance when evaluating on the entire state space.
D.5 Additional memorization experiments
In Figure 11 we look reward corruption. This another setting considered by Zhang et al. 20077 achieve de-structuring of the form L R(s).
2021
In Figure
12 we look at online trained vs distilled GFlowNets with true shuffled rewards.
22

---

Published in Transactions on Machine Learning Research (04/2025)
True R Corrupted
Normalized Loss
examples seen 1e7
Figure 11: Memorization gap training curves for the constant reward on the graph generalion task. In this experiment, we consider the case where Gaussian noise corrupts the constant reward signal of the form R(x) = R(x) + e, ~ (0, thus inducing de-structuring. Here we consider a = observe behaviour consistent with that found in
Counting
Neighbors
Cliques
Distilling Pr
Online
Shuffied
Normalized Loss
examples seen
examples seen
exam ples Seen
Memorization gap training curves for counting, neighbors, and cliques tasks for distilled (regressing to Pr) and online trained GFlowNet. We observe results are consistent with findings in $4.3 for the online trained GFlowNet.
Figure 12:
23

---

Published in Transactions on Machine Learning Research (04/2025)
D.6 Full Experimental results for offline and off off-policy training of GFlowNets
jp(x:
[x0]
X- UOO
(logotx -logo
X- pix;
X- Rix!
counting
neighbors
cliques
10
10-2
10
10
15
1.5
2.0
167
1e7
10a
[log
10
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
1e7
1e7
1e7
10a
10a
lloe
10-1
10-2
0.5
1.0
2.0
2.5
0.0
1.0
1.5
2.0
0.0
0.5
1.5
2.0
2.5
examppees seen
167
examppee seen
167
exaample seen
167
Figure 13: Evaluation curves for offline and off-policy trained GFlow Nets on counting, neighbors, and cliques graph generations tasks for different choices of Px when training using the full dataset (no test set). Model performance is dependent on the choice of xxx When considering evaluation on the JS divergence and MAE metrics, depending on task and some choices of Fx, p(x) is not adequately approximated. However, regardless of task, offline and off-policy trained GFlow Nets appear to robustly learn R.
24

---

Published in Transactions on Machine Learning Research (04/2025)
X- pix: -01001
x-AOri
x plx:
x- U04
- ploqp0c, 8) p0opplgppppp
counting
neighbors
cliques
6x 10-1
10-
Divergence
5 10-1
10-3
x 10-2
4 10
x 10
1e7
1e7
4 10
2 10
10a
167
167
10a
109
[log [ogR]
10-1
10-1
examples seen
167
examples seen
167
examples soen
1w7
Figure 14: Evaluation curves for offline and off-policy trained GFlowNets on counting, neighbors, and cliques graph generations tasks for different choices of Px when training using a subset of the full dataset (90%~10% train-test split). In this setting, agnostic to the choice Px and graph generation task, we observe the GFlowNet models struggles converge when considering JS divergence and MAE distributional metrics for approximating p(x). However, we observe the Net models remains robust when learning R.
25

---

Published in Transactions on Machine Learning Research (04/2025)
(logpóx 0) - -logp [xa)a
counting
neighbors
cliques
10a
10
3x 10
2x
Test
10a
108
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
15
2.0
2.5
167
10a
10a
Test
10:1
10
Examples seen
107
examples soon
exomples sean
Figure 15: Continuation of Figure 14 but evaluated over only the unseen sates (test states). We observe performance is consistent with the evaluations over the entire state space (Figure 14
26

---

Published in Transactions on Machine Learning Research (04/2025)
D.7 Offline GFlowNets inadequately assign probability mass
subTB
prob-ablity
tomi probability
0.6
0.4
0.4
0.2
0.44
D-0.8
0.2
0.0
0.0
20
40
60
80
100
20
40
60
80
100
training progress
training progress
The total probability of the test set (dotted) and of the top-100 objects within the test set (full lines); means of 4 seeds. Each color is a different training/ test split ratio. As the model has access more data, it seems to overfit more; gives more total probability to very few states, but particularly, to high reward states. We also plot in the bottom right (stars) what the "true" total probability of the top-100 objects should be (it varies because of the random seeds and test set size, we show the seed average).
Figure 16:
Let's dig into the experiment of $D.3 We've ve established that offiine GFNs manage to rank unseen states fairly well, but this says nothing about how much probability it gives those states, only that the relative probabilitics are well bohaved.
In Figure 16 we show how much total probabilily (i.e. Eietest p( 0)) is allocated to the test set. Tn particular, we show the total probability of the whole test set, as well of the top-100 objects within the test set.
We believe the following result is unexpected: the training set grows larger, high-reward unseen objects take up more total probability.
This seems to be counter to our intuition on generalization; which suggests that as a model gets more training data, it should extrapolate better (which for a probability model should mean that p(x; 0) should get closer to p(x)).
This result seems be both terrible news and great news. First the bad news; this result suggests that naively applying GFN objectives to an offline training regime simply doesn't 't work. Indeed, intuitively because Z is free variable in GFNs, one of two things seems likely to happen: either the model ends up learning Z = Eietrain R(x,) and giving 0 probability to the test set, or the model ends learning an arbitrary large Z and giving probability to the training set. The good news; while the latter seems to happen (the training set ends up with fairly low total probability), there seems to be structure to which states receive the most probability. This again seems particularly useful in the scenario where we are looking for the most interesting (high-reward) x.
27

---

Published in Transactions on Machine Learning Research (04/2025)
Test Rank Correlation
Test top-100 Rank Correlation
Avg Rank optimal objects
1.0
0.8
10a
0.9
10a
0.8
0.4
10a
Offline SubTB
0.6
Supervised
10a
0.4
0.6
0.4
0.8
0.4
0.8
train set ratio
train set ratio
train set ratio
Figure 17: Rank-related metrics during offline GFN training and supervised regression as function of the size of the training set. Averages over 4 seeds (which influence the construction of the test).
E Sequence and Hypergrid Experiments
E.1 Online and distilled training
E.1.1
Online trained models
branin const corners
10a
10-2
15 Divergence
Blog tog
10-3
10-2
10-4
10-2
10-3
Figure 18: Training a GelowNet (online and on-policy) on different hypergrid tasks. Corners, the most frequently used task the hypergrid environment, appears to be most difficult for learning p(æ).
28

---

Published in Transactions on Machine Learning Research (04/2025)
10
10-1
10-3
Divergence
10
|log Ps logol
10
Manna wA
10-3
10-3
10-6
10
10
10"5
constant
10-8
edit distance
10-6
0.0
0.5
1.5
2.0
2.5
0.0
0.5
1.5
2.0
2.5
Number examples
1e7
Number examples
Figure 19: Training GelowNet (online and on-policy) on 2 different sequence tasks. The edit distance task appcars to be significantly more difficult than the constant task for learning p(x). Given the auto-regressive nature of the sequence environment, it is not unsurprising how well GelowNet performs the constant task (i.e. the sequence environment is not challenging on its own). In contrast, we reify that the edit distance reward is indeed a challenging task in this environment.
10-1
|log,e loggl
10
10-1
constant
edit distance
0.0
0.5
1.0
1.5
2.0
2.5
Number of examples
1e7
Figure 20: MAE error for only test states on the sequence task for online and on-policy. Note that for this experiment, the Lest states are not necessarily unvisited by the model
E.1.2 Distilled flows and forward policy models
10a
10a
const corners
10-2
10-4
10-1
Fostt
tost
train set size: 0.6
train set size: 0.9
10
8
10-1s
10
10-17
100
200
300
400
100
200
300
400
Epach
Figure 21: Training a MLP on different tasks to regress to (left) and R (right) in the hypergrid environment. Task difficulty between supervised models and online and on-policy trained GFlowNets appears consistent.
29

---

Published in Transactions on Machine Learning Research (04/2025)
Train ratio 0.9
Train ratio 0.9
10
10
Fsa
10
Divergence
|log -logp)
10
10-1
branin
10
const
currin
10
10
100
300
400
100
200
300
400
Validation Epoch
Validation Epach
10-4
10
10-2
i5 Divergence
[log Po loggp
10
10-1
branin
10-1
canst
corners
Fsa
currin
10
10-2
100
Validation Epoch
Validation Epoch
Figure 22: Training model to distill edge flows and policies 4 diffcrent hypergrid tasks for 90%-10% train-test split (lop) and 60%-40% train-test split (bottom). It generally seems that (a) doing so recovers the intended distribution fairly well, and (b) modeling Pr appears easier than modeling F, insofar as it recovers p(x) better, in the hypergrid environment.
10-2
10-1
10-1
Divergence
|logPa- log
10
constant
10-2
edit distance
10-5
1
10-7
PF
10-3
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
Number examples
Number examples
Figure 23: Training a model distill edge flows and policies on 2 different sequence tasks for 90%-10% train-test split. It generally seems that (a) doing recovers the intended distribution fairly well, and (b) modeling Pp appears easier than modeling F, insofar it recovers p(x) better, in the sequence environment for the edit distance reward. For the constant reward, there appears to be no apparent difference in difficulty between learning Pp or F.
30

---

Published in Transactions on Machine Learning Research (04/2025)
llogpe log.p1
constant edit distance
10-2
10-3
0.0
0.5
1.0
1.5
2.0
2.5
Number examples
1e7
Figure 24: MAE error for only test the sequence task for distilled model, regressed to Pp. Note that for this experiment, the test states unvisited by the model.
31

---

Published in Transactions on Machine Learning Research (04/2025)
E.2
Reward transformation
Distilled JS Divergence
-logpl
Online GFN Divergence
Online GFN [logge-loog
10-2
10
10
10-1
Figure 25: Training models distilled Pr and GFlowNet models trained online/on-polin for range of monotonie skew values 7 on the sequence task. Transforming the distribution of the edit distance reward to contain larger proportion of high reward values generally improves the performance for approximating p(æ)).
E.3
Memorization gap experiments
E.3.1
Memorization in hypergrids
branin
currin
corners
Leaming By Leaming
Pr(ll))
Shuffed
No-malizes lowd
Figure 26: Memorization gap training curves for branin, currin, and corners tasks on the hypergrid enviromment. Maintaining flow structure in the learning problem (learning Pr under shuflle Rt) generally induces smaller memorization gap relative to the fully de-structured setting. Note that for the branin reward, there is not apparent difference between learning the true R and the shuffled R, suggesting the reward alongside the hypergrid environment induces task with minimal difficulty.
32

---

Published in Transactions on Machine Learning Research (04/2025)
E.3.2
Memorization in sequences
String Sequence
Shuffled &
Nom Nomallzes slieed LoGs
Learning Fr Learning
10000
20000
30000
40000
Iterations
Figure 27: Memorization gap training curves for the edit distance task on the sequence environment. Maintaining flow structure in the learning problem (learning PF under shuffled Rt generally induces smaller memorisation gap relative to the fully de-structured setting.
E.4 Offline and off- pofico training
E.4.1
Offline and off-policy in hypergrids
x-pix:0
Rtd
x=|p00c p²(x)
x
(lagpóx; 8) logp"(())
branin
currin
corners
10a1
10-2
10-1
15 Divergous
10-2
10-4
10'
10:4
10-5
10a5
10-2
100
200
300
400
300
200
300
400
100
200
300
400
20a
[log 1og0)
10-1
10*
10
10-2
10
100
200
300
400
100
200
100
Volidation Epoch
Validation Epach
Figure Evaluation curves for offline and off-policy trained GFlowNets on branin, currin, and corners hyporgrid tasks for different choices of x when training using a subset of the full datasct (90%- 10% train-test. split). In this setting, agnostic to the choice of P and hypergrid reward, we observe the Gel models converge when considering both the JS divergence and the MAE distributional metrics for approximating p(x). This differs from the graph generation tasks where models struggle to converge on these metrics in this setting. For the more difficult task (corners), sampling r from sets that include the proxy-policy p(r; 0) result. in the best performance on JS divergence while sampling x ~ 24 and x ~ log 1(2; 0) logp(z) results in the best performance on the MAE metric.
33

---

Published in Transactions on Machine Learning Research (04/2025)
E.4.2
Offline and off off-policy in sequences
x Dogple; 0) -logp (x]?
x- |p0: 6) - px"
.... X- REA
x pixt
X- Upd
Divergence
[logps logpl
[logR= log
Test (log R - Rl
2x
2x10
10-1
6x
10a
4x10--
6x
3x10-1
4x 13m1
2x10-
3% 10ml
10-00-
examples
167
examples
167
167
examples
167
Figure 29: Evaluation curves for offline and off- off-policy trained GFlowNets edit distance sequence task for different choices of Pxx when training using subset of the full dataset (90%-10% train-test split). In this setting, agnostic to the choice of Px, we observe the GFlowNet models converge when considering JS divergence and MAE distributional metrics for approximating p(x). This differs from the graph generation tasks where models struggle to converge on these metrics in this setting.
34


Reviews:
[{"review_id": "fjQrNmtDjp", "reviewer": "Reviewer_58u1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper studies a relevant problem (generalization of GFlowNets) -- indeed, the favorable generalization properties have been repeatedly used in the literature to explain GFlowNet's success;\n\n- Overall, the paper is well-written and easy-to-follow;\n\n- The introduced benchmarks could be valuable for future research on the limitations/generalization of GFlowNets.\n\n**Weaknesses**\n\n- The hypothesis are somewhat vague:\n  - How do we quantify the \"narrow\" aspect in Hypothesis 1? Isn't that important to validate the hypothesis?\n  - The term structure in Hypothesis 2 is not clear  (referred to as \"patterns in the data\"). In addition, the paper says little about which \"structures\" enable GFlowNets to generalize.\n\n- I found surprising the performance gap between GAT and other GNNs. Any idea why is this? The paper says \"While they are not the simplest GNNs, they may be the most general (because of the attention mechanism)\" --- is GAT more expressive than GIN? In addition, I am not aware of prior work reporting such a stark difference in performance. In fact, some studies have highlighted the limitations of the original GAT model [1,2].\n\n- My main concern is about the generality of the findings, i.e., how broadly the conclusions apply. I wonder how much we can extrapolate the conclusions to different state graphs, training objectives, and GNNs (parametrization of the policy network). Can the authors elaborate on the dependence/sensitivity of the results on these choices?\n\n\n[1] Improving Attention Mechanism in Graph Neural Networks via Cardinality Preservation, IJCAI, 2020.\n\n[2] How attentive are graph attention networks?, ICLR, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper empirically investigates GFlowNets' ability to generalize to unvisited states during training. In particular, the authors introduce three hypotheses: (1) GFlowNets generalize only under a narrow set of distributions; (2) GFlowNets generalize because PF and F are not arbitrary, and generalization is mainly affected by the complexity of the reward. Aiming to validate these hypotheses, the authors design graph generation benchmark tasks with varying difficulty levels, expressed in terms of reward functions. The authors also discuss how GFlowNets generalize differently considering three settings: (1) distilling flow functions, (2) memorization gaps in GFlowNets, and (3) offline and off-policy training regimes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I don't have specific requested changes. That being said, I believe that addressing my concerns above (weaknesses) would help strengthen the impact of the paper.", "broader_impact_concerns": "I don't have any concerns regarding broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3VR6ERD1jo", "reviewer": "Reviewer_3T6g", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n+++ Very nice construction of benchmark and ablations, truly serving as an example on how to slice and ablate a model. it seems in particular the learning curve dynamics and memorization gap under ablations and permutation of setting is valuable\n\n++ Carefully scoped insights and conclusions, based on a *lot* of variation. (Some of the most interesting observations for me were in the appendix, e.g. the bias towards overfitting the argmax) and the rank relationship across training losses\n\nCode and explanation of the benchmark is transparent and gives the impression that it should be easy to extend  the benchmark\n\nWeaknesses\n\n1. the paper could be a bit more clear on some elements and take care in the details, in particular\n    \n    1. Fig 1a) “coincidentally” shows some ablation over capacity as noted in the appendix, but this is neither quantified further (is this 3x the GAT? what parameter count? was it run to saturation? why?)\n        \n    2. There is a lot of elements floating around (graphs as states, subtrees/graphs in the flow MDP the GFN operaties on, “supervised learning” vs “distillation” vs the different TB losses, learning F,log Pf, learning these under different permutations of online/offline/interpolated, with or without destructuring…) the authors already try to make this clear, but I think having a section in the appendix which uses a combinatorial coding for each setting/combination and spells out which setting exhibited what behaviour and supports which claim would be helpful (e.g., figure 5: offline-offpolicy-neighbours-regression on unpermuted R, varying PB amongst  the set… inducing different log Pf and reward distributions=> the variation across different ) Again, the authors already take care to define all of this but I think it can be improved. ideally, for each experiment and figure, it should be possible to look up the training method, what the training loss was taking to w.r.t and how exactly the curve value is computed (JS between what and what exactly, what specific p theta are we talking about) and what claim/insight is being supported.\n        \n    3. This is most pronounce for me in a) section 4.4. to confirm my understanding: you use the correctly computed pf to regress onto, but *sample* the training samples using off policy PF, leading to something like dataset skew? or is this section using subTB in an off policy way?  \n        and b) the logreward distributions shown, I can’t understand those plots, are they counts with an omitted e5 number? normalized frequencies? properly labeling the plots would resolve this issue. Maybe I'm misreading the plots, but the neighbors task also doesn’t look that nice log distributed to me tbh? the counting one does, but the neighors one looks loglognormal  with an additional peak?\n        \n    4. I think the paper can actually make some more confident claims around using GAT: all the tasks are of the form “does the action lead to an increase in number of features of type “does a local neighborhood exhibit a property which is a function of a center node interacting with it’s neighbors”, the feature computation at list  is *PERFECT* for GAT and wouldn’t surprise me if it could be formalized as a RASP program [https://arxiv.org/abs/2106.06981](https://arxiv.org/abs/2106.06981). That means for this benchmark, the Hypothesis class is somewhat likely to contain the ground truth function and thus *TRUE* generalization is possible (we can’t know for sure the hypothesis class contains the GT function without constructing a GAT that will compute this as it requires also distributing the chain of  actions to the final reward, but I think the idea of using the strongest possible approximator for this is exactly what one wants)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a synthetic task for studying the generalization properties of Gflownets (GFNs). The task consists of learning graph edits on graphs of up to 7 nodes, such that the resulting graph distribution follows the reward distribution induced by 3 + 1 debugging tasks\n\n1. Clique(G)=n_CC(G)-n_C(G)+n(G)\n    \n    where n_CC is the number of 4 cliques in which 3 or 4 nodes have the same color (max number of 4 cliques is 35, max number of clqiues is 120, for reference), then take exp => clipped to have logR be between -10 and up to 0.\n    \n2. count the number of nodes with an even number of neighbours of the opposite color \n    \n3. counting is number of red and green and assigns ground truths\n    \n4. (debugging) constant assignment of uniform reward everywhere\n    \n\nGiven the chosen graph state space yields a size of ~72k states, the authors can then derive ground truth state probabilities and flow transitions from the tasks, and perform a series of studies to investigate a set of key hypotheses around GFNs ( generalization under a narrow set of distributions, based on structure, driven by the complexity of the reward as distributed across the dynamics, not the distributional properties of the reward over end states).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None of these are critical, but I do think they'd make the paper _even better_:\n\n\nSuggestions:\n\n1. the clarification with making definitions unambiguously attributable\n    \n2. if computational power applies, one could validate the GAT strength as follows:\n    \n    1. assign a unique label (e.g. in a decimal one hot encoded way) to each graph and verify the model can learn this to zero training loss (meaning it can learn to identify each graph uniquely)\n        \n    2. verify the model can learn to regress the dynamic programming value function for each state to zero training loss\n        \n    \n    neither of these prove that the GAT can learn the GT function, but it comes as close as I can imagine without formalizing the function as a RASP + Message passing  \n    \n3. A nice to have to add would be a formalized notion of the type of generalization studied (e.g. in the form of excess loss compared between base distribution and test distribution, compared to the distributional shift under question, e.g. the change in shifted reward distribution as measured by W1 vs. the shift in p(x) recovery ) could help round of the already very good precision of claims in the benchmarks. some of the studies feel more like tests of Algorithmic stability  [https://proceedings.mlr.press/v70/liu17c.html](https://proceedings.mlr.press/v70/liu17c.html)  as they don’t check a generalization error and more the performance and learnability per se", "broader_impact_concerns": "theoretical study which does not require ethical implications, beyond the 2k GPU hours which we all inflict onto our planet with each paper", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BLu0NKLZAY", "reviewer": "Reviewer_ePyU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths__:\n - The paper is generally written very well and most points are communicated clearly and concisely.\n - The relative difficulty of the three problems defined in the graph environment are verified empirically and are sound.\n - Experiments are designed well to probe specific properties of GFlowNets.\n - Experiments retest and challenge conventional beliefs about GFlowNets on the new and more difficult task.\n - Interpretation of results is reasonable without extraordinary claims, and clearly identifies important points where some results are not completely conclusive but are still interesting.\n - The proposed synthetic environment and the results from the experiments are significant and are valuable for building our understanding of GFlowNets.\n - Many of the details are provided in the appendix and referenced appropriately.\n\n__Weaknesses__:\n - In the introduction, it is not immediately clear how experiment 1 supports hypotheses 2 and 3. It seems the experiment 1 does not directly support hypothesis 2, but is used as a stepping stone to support the hypothesis by using the distillation method in experiment 2. Distillation alone seems to only test generalization in general, but without considering experiment 2, it is not clear how it is testing the learning of structure. It is not revealed until the experiments how Hypothesis 3 is supported by the distillation (via monotonic transformation of the reward).\n - The explanation of experiment 2 (memorization in section 3.2) could be made more clear within the section. Between sections 3.1 and 3.2 there is a slight change in language where it is potentially not clear that the true flows are computed from the shuffled rewards. This point is actually made clear at the end of section A of the appendix on the definition of \"True flows\", which could at least be referenced within section 3.2 or clarified with an additional sentence.\n - In section 3.2 is the shuffling of rewards is amongst all states or just the terminal states?\n - \\tilde{P}_F(s'|s) in table 3 is not defined anywhere but can be assumed to be policy logits derived from the shuffled rewards.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This paper investigates generalization properties of GFlowNets using a new synthetic environment.\n - Three hypotheses are made about these generalization properties:\n    - GFlowNets generalize under a narrow set of distributions.\n    - GFlowNets generalize by learning structure.\n    - Generalization performance is driven mostly by reward complexity rather than distribution properties such as temperature and skewness.\n - Three core experimental settings are used to test these hypotheses:\n    - Distillation - using the tractability of the new synthetic dataset, GFlowNets can be trained and tested while controlling for non-ideal factors of the GFlowNet training setup.\n    - Memorization - various methods of ablation structure from the data / model are used to probe the generalization / memorization properties of GFlowNets.\n    - Offline and off policy training - generalization performance can be measured as a function of differing training distributions.\n - The proposed synthetic environment is based on a graph building problem with three levels of difficulty.\n    - This task is more difficult that conventional synthetic tasks: hypergrid and sequence tasks.\n    - Although the task is more difficult, important quantities for benchmarking (e.g., exact likelihoods, flows) are tractable.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My requested changes are mostly minor, as specified in the weaknesses section.\nI also recommend correcting the typo: \"asses\" -> \"assess\"", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that this was an interesting and thoroughly explored piece of work.  While entirely empirical, it sheds light on the behaviour of GFNs and points towards some promising avenues for future work.

A number of minor issues and clarifications were raised by the reviewers and the authors should aim to address these in the camera ready version.
Decision: Accept (poster)