Pessimistic Backward Policy for GFlowNets
Authors: Hyosoon Jang
Yunhui Jang
Minsu Kim
Jinkyoo Park
Sungsoo Ahn
Venue: NeurIPS 2024 poster (2024)
Keywords: Generative flow networks
generative models
reinforcement learning
Primary area: generative_models
TL;DR: We propose a pessimistic backward policy for GFlowNets to resolve the under-exploitation problem of backward policy-based flow matching.

Abstract: This paper studies Generative Flow Networks (GFlowNets), which learn to sample objects proportionally to a given reward function through the trajectory of state transitions. In this work, we observe that GFlowNets tend to under-exploit the high-reward objects due to training on insufficient number of trajectories, which may lead to a large gap between the estimated flow and the (known) reward value. In response to this challenge, we propose a pessimistic backward policy for GFlowNets (PBP-GFN), which maximizes the observed flow to align closely with the true reward for the object. We extensively evaluate PBP-GFN across eight benchmarks, including hyper-grid environment, bag generation, structured set generation, molecular generation, and four RNA sequence generation tasks. In particular, PBP-GFN enhances the discovery of high-reward objects, maintains the diversity of the objects, and consistently outperforms existing methods.

Pessimistic Backward Policy for GFlowNets
Hyosoon Jang', Yunhui Jang', Minsu Jinkyoo Park², Sungsoo Ahn' POSTECH
KAAIS
{hsjang1205, umi5510, sungsoo aho))ppctech.ac... kr. (min-su, j inkyoo. park ac. kr
Abstract
This paper studies Generative Flow Networks (GFlowNets), which learn to sample objects proportionally to a given reward function through the trajectory state transitions. In this work, we observe that GFlowNets tend to under-exploit the high-reward objects due to training on insufficient number trajectories, which may lead to large gap between the estimated flow and the (known) reward value. In response to this challenge, we propose a pessimistic backward policy for GelowNets (PBP-GFN), which maximizes the observed flow align closely with the true reward for the object. We extensively evaluate PBP-GFN across eight benchmarks. including hyper-grid environment, bag generation, structured set generation, molecular generation, and four RNA sequence generation tasks. In particular, PBP-GFN enhances the discovery of high-reward objects, maintains the diversity of the objects, and consistently outperforms existing methods.
1 Introduction
Generative Flow Networks [1, GFlowNets] models that sample compositional objects from Boltzmann distribution defined by some reward function. To this end, GFlowNets construct an object through trajectory of state transitions, e.g., iteratively adding molecular fragments to construct a molecule. They are attractive for their ability to sample a diverse set of high-reward objects, as demonstrated in molecular discovery [2, 3], biological sequence design [4], combinatorial optimization [5], and large language models [6].
In detail, GFlowNets aim sample from the Boltzmann distribution using a forward policy to decide the state transitions. However, this is challenging since the forward policy induces the distribution over trajectories, while the Boltzmann distribution is only defined the terminal state of trajectories, i.e. objects. Hence, directly matching the two distributions with respect to the terminal state requires an intractable marginalization of the forward policy over the exponentially sized trajectory space.
To circumvent this issue, GFlowNets employ an auxiliary backward policy that lifts the Boltzmann distribution to the trajectories via reversing the state transitions. In particular, the backward policy decomposes the unnormalized Boltzmann density of a terminal state into the unnormalized densities of trajectories, coined backward flow, associated with the terminal state. Then the forward policy learns the Boltzmann distribution by matching its unnormalized density, i.e., reward, coined forward flow, with the backward flow the observed trajectories. We call this training scheme flow matching."
The training objective of the flow matching has been investigated such as detailed balance [7] and trajectory balance [8], and sub-trajectory balance [9]. To facilitate training, improved credit assignment techniques have been explored [10, 11]. Additionally, exploration methods [12] have been proposed to collect more diverse trajectories. Moreover, exploitation methods such as focusing on the higher-reward trajectories from the backward policy [13] and sampling high-reward trajectories with local search [14] have been presented for the collection of higher- reward trajectories.
In this work, we refer to flow matching as the learning scheme that aligns the forward and backward flow.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

unabserved
2
(a) grid
conventional flow matchling
initial state
high reward
backward flow
forward flow
(c) flo matching with pessimistic backward policy
Figure 1: Flow matching for observed trajectories. (a) The task aims to reach the terminal state with reward-proportional probability from the initial state, by incrementing one coordinate random action. The black line indicates the two observed trajectories for each terminal state. (b-c) The arrow (-+) length indicates the amount of the backward forward flow. (b), the flow matching (z) between the observed backward and forward flows underestimates the high-reward object due to the low observed backward flow. (c), PBP-GFN succeeds with the observed backward flow that fully represents the true rewards.
In this work, we point out a pitfall of the flow matching objectives: when only a small portion of object-sharing trajectories is observed, GFlowNets tend under-exploit the object. This pitfall stems from the under-determination
the forward flow, due to training only on the observed backward flow that partially represents the true high reward. Consequently, the forward policy tends assign high probabilities to objects with high observed backward flow, rather than the high reward objects, illustrated in (a) and (b) of Figure 1. This is counter-intuitive as the forward policy favors objects with low rewards despite possessing the knowledge of other objects with higher rewards. While one could bypass this issue the cost of observing more trajectories [13], we pursue alternative direction in this work.
We propose simple remedy for the under-exploitation problem: pessimistic backward policy for GFlowNets (PBP-GFN). Our key idea is the maximization the observed backward flow to align the observed backward flow the true reward. Consequently. PBP-GFN resolves the under- exploitation problem which favors the object with high observed backward flow while neglecting the true reward, illustrated in (c) of Figure We also note that our algorithm preserves the asymptotic optimality to induce the target Boltzmann distribution by simply modifying the backward policy while preserving the true rewards [8]. Additionally, we analyze how algorithm reduces the error bound in estimating the true Boltzmann distribution.
We extensively validate PBP-GFN various benchmarks: hyper-grid benchmark [1], bag genera- tion [13], maximum independent set problem [5], fragment-base molecule generation [1], and four RNA sequence generation tasks [4]. In these experiments, observe that PBP-GFN (1) improves the learning of target Boltzmann distribution and (2) enhances the discovery of high-reward objects, while (3) maintaining the diversity of the sampled high-reward objects.
To conclude, our contributions can be summarized follows:
We characterize the under-exploitation problem stemming from an under-determined flow that only learns the observed flow for partially observed trajectories (Example 1).
To resolve this issue, we propose pessimistic training backward policy that aims to reduce
amount of unobserved flow for the observed objects.
Through extensive experiments, we show that our algorithm consistently improves the perfor- mance of GelowNets compared to prior works for designing the backward policy, even higher than other training algorithms for discovering high-reward objects.?
Code: https://githhn http://ttttt...oo//////ggg//ssst Backvard- batpree--d-aaaaadodaooooo---e-e Policy--oo for.iyy--fo--GFooleetttt GFlowNeets.

---

2 Preliminaries
Generative Flow Networks [1, GFlowNets] generate an object from the object space X' through trajectory 7 (5o, : of state transitions, where the terminal state is the object sT = z E X' be generated. Here, forward policy Pp ($t41|s,) makes the transition from the state s, the next state S141 and assigns a probability of PF( = II'=D' (=(5(11(5r) to the trajectory т.
Next, GFlowNets train the forward policy to sample objects from Boltzmann distribution defined by a reward function R(x) that satisfies:
PF (z) & R(z).
(1)
Here, P (z) is distribution of object z marginalized over exponentially sized non-terminal state spaces. To circumvent this intractability, GFlowNets train on the flow matching objectives.
Flow matching for training GFlowNets. To learn the Boltzmann distribution, the forward policy PF aims to align to a backward policy Pg. The backward policy (B(T(z> -I'' ( decomposes the reward into the unnormalized densities of object-sharing trajectories T(x) for an object i... R(x) = EveT(s) (Bx)))
To be specific, the forward policy learns to match the unnormalized densities to the backward policy, coined flow matching, over all trajectories in the trajectory space T:
VT E ET (pP)(T) R (T
where 7 (r) is a forward flow defined with learnable constant Ze. and backward flow. Equation (2) induces the forward policy following Boltzmann distribution, i.e., Zo PJP (x)) R(x), by marginalizing trajectory flows over set of trajectories T(x) inducing the object I, i.c., ErET(=) (rPp(T) Pp (I) and E)ET(E) P(x)PP((())
is
R(x).
To satisfy Equation (2). GFlow Nets minimize various training objectives. One such objective is the
trajectory balance [8, TB), defined as follows:
Zo Pr(T)) (Tu(T) = log R(x) (
which is minimized over trajectories observed during training, e.g., trajectories sampled from the forward policy. The set of observed trajectories stored in the replay buffer inducing the object x is denoted as (xx)) T(x). Note that training objectives for Equation (2) can also be defined transition [7, DB] or sub-trajectories [8, subTB].
3 Method
In this section, we introduce our pessimistic backward policy for generative flow networks (PBP-GFN). First, we show that forward policies trained with flow matching tend to under-exploit high-reward object x with partially observed trajectories B(x)) when the underdetermined forward flow only learns the small amount of observed backward flow for the high-reward object (Section 3.1). To address this issue, propose pessimistic training of backward policy that increases the proportion of observed flow for the object, which leads to an accurate estimation of the reward (Section 3.2).
3.1 Motivation: under-exploitation of objects with partially observed trajectorie
First, explain how conventional flow matching may suffer from the under-exploitation of observed
high-reward objects. To this end, we decompose the reward R(x) into two components: (1) observed backward flow Rz(z) ΣrEB(Z) R(x) Pa (T(zr assigned to the partially observed trajectories B(x). and (2) unobserved backward R(x) Rg (z) assigned the unobserved trajectories T(x) (B)))
Then (1) and (2) are paired with observed forrard flow and unobserved forward flow, respectively.
In detail, the hand, conventional flow matching aligns the observed forward flow to the observed ward flow for the first component, i.e., ΣTEB()) 7 (r) z RB(x). On the other hand, there exists degree of freedom for the unobserved forward flow [TTT(=) Zo PPPr()), it is challenging match the flow over unobserved trajectories, i.c. trajectories not in the buffer B.
3

---

21
22
33
Z/2
3'
Reward
Backward flow
2 Forward flow
(F3) & (2))(22) (22) & P (x))
PF & (r))(12)) I (T)
:observed
unobserved
R(x1) v R(22) R(as)
R(zi) < R(a2) < R(23)
(a) conventional flow matching
(b) PBP-GFN
Figure 2: Under-exploitatiin of objects with partially observed trajectories. The reward R(x) consists of (1) observed backward flow RB(ø) and (2) unobserved backward flow R(x) - (r)... (a) Conventional flow matching may assign higher probability to the lower-reward object as the observed forward flow is aligned only with small amount of observed backward flow. This fails to assign the accurate probability proportional to the reward. (b) PBP-GFN assigns more accurate probability proportional to the reward, by increasing the proportion of observed flow.
Overall, flow matching induces forward policy with the marginalized probability (x)(())
(p & [B(
E ZoPp(T) (BT())B(O))
(4)
Here, our key observation is that, for an observed object with a high reward R(x) but small
amount of observed backward flow [g(x), the unobserved backward flow R(x) Rs (x(x)) )) is likely to
be much larger than the forward flow of the unobserved trajectories [weT(w)) ((x) Zg (T). This
leads to the under-exploitation of the high-reward object и. since the forward policy assigns a higher probability another object x' with lower reward but a larger amount of observed flow RB(x), as illustrated in (a) of Figure As a result, the marginalized probability PF.s(x) may converge local optimum that yields a smaller expected reward compared to the target Boltzmann distribution.
To further motivate our proposal regarding the under-exploitation problem, we present failure case of flow matching converged to a local optimum contradicting the observed rewards in Example 1. We construct particular instance of Equation (4) where the forward policy underestimates the high-reward object compared to the lower one. We depict this example in (a) and (b) Figure 3.
Example 1. Consider two objects in and 22 with rewards of 1 and I respectively, where in is 2' reached by three trajectories ((T(x)) 3) and x2 is reached by one ([T(22) 1). Here, one trajectory for each object observed (B((z)] = [Bxz2] = 1). Then, the probability to induce object œ1 can be assigned as (21) 3 since the forward flow still matches the backward flows for the observed trajectory i.e., (F))(1)) PF.((T)) & R(zz)Pp(TTler = 1.2 This is lower assigned fully observed trajectories.
than PF,
Example 1 is counter- counter-intuitive, as a higher probability is assigned to the lower reward object 22 despite observing the higher reward object æ1. The forward policy (F also assigns zero probability unobserved trajectories. Consequently, the probability (F. cannot be corrected trajectories sampled from policy 7 (T) This hints the necessity of the remedy for the under-exploitation of objects due to the small amount of observed flow, with the fixed observation B.
more
3.2 Pessimistic backward policy for GFlowNets
Here, we propose a pessimistic training method for the backward policy in GFlowNets, coined PBP-GFN, which aims to resolve the inder-xplloitation problem of the flow matching with partially observed trajectories introduced in Section 3. address this challenge, the backward policy trained to reduce the amount of unobserved backward flow, being pessimistic about unobserved trajectories inducing the observed object. notable that the total backward flow for the object, i.e., reward, is preserved by shifting the unobserved backward flow into the observed backward flow.

---

(a) conventional flow matching (entire trajectories)
(b) conventional flow matching (partially observed trajectories)
(c) PBP-GFN
(partially observed trajectories)
Figure 3: Pessimistic backward policy for GFlowNets (PBP-GFN). The portion of the circle indicates the amount of flow, e.g., indicates the flow of I, and O indicates the half flow of i.c., the flow of 0.5. Additionally, the color of the flow indicates the flow inducing the same-colored reward, and the black and gray lines indicate the observed and unobserved trajectories, respectively. (a) Flow matching succeeds with the entire trajectories. One can observe that the true reward of Iy is and the reward of 22 is 0.5 by the amount of flow. (b) Flow matching fails with partially observed trajectories. (c) PBP-GFN assigns high probabilities to the backward transitions of observed trajectories to keep high probability high-reward objects.
Algorithm Learning pessimistic backward policy for GelowNett
1: Initialize the replay buffer (i.e., the set of observed trajectories) B, forward policy Pr, backward policy Pn. and parameter Zg.
2: repeat
3: Sample batch of trajectories (T()) ]k=1 from the behaviour policy.
4: Update B <- B U {r(k)) 1==1-
5: for n = 1,...,N .N do
> Learning pessimistic backward policy
6: Update Pn to minimize Eppp over B with stochastic gradients.
7: end for
8: Update PF, Za minimize LTB with (701) Jk=1 Jk=1
9:
until converged
To be specific, given a replay buffer B, the pessimistic training of backward policy Pa aims to increase the backward flow for the observed trajectories ending with the object æ. Specifically, it aims to maximize RB (z)x) = EreB(a) R(r) (T(x), thereby aligning the observed backward flow Rg(x) to the true reward R(x), reaching the upper bound (s(z) R(x). Consequently, flow matching with such a backward flow for partially observed trajectories induces an observed forward flow that accurately estimates the true reward, thereby preventing the under-exploitation of the rewards due to the small amount of observed flow (Example 1), as illustrated in (b) of Figure 2 and (c) of Figure 3.
Furthermore, it is worth noting how PBP-GEN better estimates the Boltzmann density, i.e., reward. The high-level idea that, given the fixed total flow, maximizing the observed forward and backward flows with PBP-GFN naturally minimizes the unobserved forward and backward flows, thereby reducing flow matching error the unobserved flows effectively. The detailed error bound in estimating the Boltzmann density is described in Appendix A.
Pessimistic training of backward policy. We train the parameterized backward policy PB to increase the backward trajectory flows in observed trajectories. R(x)) ΣtEB, (B(T(z), by assigning higher probabilities to the backward transitions (z(r(z) = I (s(ss(s(+1) of observed trajectories, i.c., Σre (B x)) 1. We achieve this by minimizing the negative log-likelihood:
fppp - E, EB))) [log PB(T(=)))
(5)
where x is the object induced by the trajectory r. It is notable that our approach only modifies the relative backward trajectory flows among trajectories inducing the same object and does not alter the total amount of backward flows, thereby preserving the asymptotic optimality of flow matching for learning the target Boltzmann distribution. Note that the training the pessimistic backward policy is practical in most cases, it only requires computing the stochastic gradients minimize bppp-
5

---

Subsequently, we train the GFlowNets with the learned pessimistic backward policy. The pessimistic backward policy is learned online with the forward policy, as new trajectories are observed for the training in each round. The training algorithm is described in Algorithm 1.² Note that the pessimistic training agnostic to the choice of flow matching objectives [7-9].
Related work
Generative Flow Networks (GFlowNett), GFlowNets [1, 7] train forward policy that sequentially constructs objects sampled from a Boltzmann distribution. They are closely related reinforcement learning in soft Markov Decision Processes (soft MDPs) [15-17] and variational inference [18]. Recently, there has been surge in research on improving the training of GFlowNets, such as introducing novel flow matching objective functions [8, 9, 12, 13], enhancing off-policy exploration [14, 19-21], incorporating order information for enabling preference-bar based optimization [22], and improving credit assignment [10, 11]. Moreover, GFlowNets
increasingly applied across wide range of fields such as molecular optimization [2. 3, 23], biological sequence design [4, 24], probabilistic modeling and inference [25, 26). combinatorial optimization [5, 27, 28], continuous stochastic control [29-31], and large language models [6].
Despite the advancements in training GFlowNets, there still exists the challenge of dealing with the vast number of trajectories. The number of trajectories grows exponentially with the increase in the number of state spaces and actions, making it impractical to observe all trajectories during training. This issue can be partially addressed by facilitating the discovery of unobserved trajectories [12]. However, the problem of probability not matching the rewards remains unless sampled trajectories comprehensively cover all possible flows.
Training GFlowNets with auxiliary backward policy. GelowNets train a forward policy to align with the auxiliary backward policy, which inverts the construction process the object. Therefore, the choice of the backward policy directly impacts the training of GFlowNets and is vital to the improvement of the sampling performance. Despite erucial role, the choice of backward policy has gained limited attention with only few works [8. 13, 15], and none of these works tackle the under-exploitation of high-reward objects caused by unobserved backward flow.
For instance, while the uniform [8] and the MaxEnt [15] backward policies assign a fixed probability to the backward transition for enhancing exploration, our pessimistic backward policy learns the backward transition probability for enhancing exploitation. Next, conventional [8] and sub-structure
[13] backward policies may enhance the exploitation by learning the backward flow to align with the forward flow or to improve the credit assignments. However, they do not directly reduce the unobserved backward flow and do not resolve the under-exploitation stemming from that.
Experiment
We evaluate our method on various domains, including a hyper-grid [1], bags [13], structured sets [5], molecules [1], and RNA sequences [13, 14]. As base metrics, we consider the number of modes, e.g., samples with rewards higher than specific threshold, and the average 1001000 score, which are measured via samples collected during training. We report the performances using three different random seeds. In these experiments, one can observe that:
PBP-GFN improves learning of the target Boltzmann distribution (Figures 4 and 6).
PBP-GFN enhances the discovery of high-reward objects (Figures 5, and 8).
PBP-GFN maintains the diversity of sampled high-reward objects and promotes the discovery of distinet diverse modes (Figures 7(c) and 9).
5.1 Synthetic tasks
In synthetic environments, i.e., hyper-grid environment, bag generation, and maximum independent set, we first show how our method (PBP-GFN) improves the performance compared to the prior methods that proposed various designs the backward policy, on both the trajectory balance [8. TB] and detailed balance-based implementations [7, DBJ. As baselines, we consider the conventional
We describe the detailed implementations in Appendix B.

---

(a) Target
(b) TB
(c) Uniform
(d) MaxEnt
(e) PBP-GFN (our)
Figure 4: The target distribution and empirical distributions of each model trained with 10y trajectories. The empirical distributions computed rescaled products of the distribution three runs. Our method (PBP-GFN) consistently discovers all modes over three runs and learns the target Boltzmann distribution correctly within the relatively small number of trajectories.
18.5
3.0
16.5
2.5
Avg MIS size
17.5
Num moses
1.5
0.5
1.0
Avg of Top
2.0
1.5
Mactre
17.0
18.0
15.5
DB Hiiitmm
PoP 0FM
0.0
1.0
1.5 (x10*)
2.0
35
10
1.5
2.0
16.5
10 Epoch
15
20
14.5
10 Epech
15
20
Active
(x10P)
Bag generation
(b) Maximum independent set problem
Figure 5: The performance comparison with the prior backward policy design methods. The solid line and shaded region represent the mean and standard deviation, respectively. The PBP-GEN shows superiority in generating diverse high-reward objects, compared to the considered baselines for designing the backward policy.
backward policy trained with the TB or DB [7], the uniform backward policy [8], and the maximum entropy backward policy [15, MaxEnt].
Hyper-grid [1]. We first consider the hyper-grid, where the target Boltzmann distribution is defined over the 16 x 16 x 16 grid illustrated in Figure 4(a). We also consider the 20 x 20 20 x 20 hyper-grid. The actions are incrementing one coordinate by one or terminating. The high-reward regions, i.e., modes, are defined as near the corners of the grid that separated by regions with very small rewards. this task, we consider the TB-based implementation following the prior work [8]. The detailed experimental settings described in Appendix B. In this task, measure the L1 distance between the target Boltzmann distribution and the empirical distribution PF (z), with the measurable likelihood of the Boltzmann distribution.
Bag generation [13]. next consider a simple bag generation task, where the action is adding an item to a bag. The bag yields a high reward when seven repeated items are included, i.e., modes. We apply our method the prior TB-based implementation on this task [13] and compare it with TB and MaxEnt. The detailed setting is described in Appendix B.
Maximum independent set [5). We also consider solving maximum independent set problems, where the action is selecting a node and the reward is the size of the independent set. At each epoch, the GFlowNets train with the of training graphs, and sample solutions for each validation graph and measure the average reward and the maximum reward following Zhang et al. [5]. We apply our method to the prior DB-based implementation of this task [5]. The experimental setting described in Appendix B. Note that the MaxEnt is equivalent to the uniform backward policy in this task.
Results. In Figure and Figure 6, depict the empirical sampling distribution and the L1 distance from the target Boltzmann distribution for each method in the hyper-grid environment. Here, one can see that our method (PBP-GFN) captures all modes and converges to the target Boltzmann distribution faster than the baselines. These results can be attributed to the capabilities of PBP-GFN, which enables us effectively learn from the large amount of correct forward flow.

---

30.0
TB
1.03
Num modes [x102
25.0
MaxEnt
Unitor SJOTB
Avg of Top-100
0.55
20.0
1.0
Similarity
15.0
AZC
GAFN
10.0
0.45
2.0 Active round (x10P)
0.86
0.9
0.94
0.98
1.02
Active round (x10²)
Avg of Top-100
(a) Number of modes
(b) Top-100 performance
(c) Tanimoto similarity
The performance on molecular generation. The solid line and shaded region represent the mean and standard deviation, respectively. The PBP-GFN shows superiority compared to the baselines in generating diverse high reward molecules while yielding similar Tanimoto similarities compared to other baselines with prior backward policy designs.
Figure 7:
In both bag generation and maximum independent set problems, one can see that our approach also shows (1) superior performance (2) or faster convergence compared to the baselines as illustrated in Figure 5. One can reason this result stems from the capabilities of PBP-GEN that facilitate the learning of correct Boltzmann dis- tribution. Furthermore, it is worth noting that our method makes improvements over both TB and DB-based implementations.
MaxEnt
[x100+1
PBP-CFN
3
Molecular generation
10
10
10*
Trajectories sampled
Next, we evaluate our method in the fragment-baset molecule generation [1], where the action is adding molecular building block. The reward the binding energy between the molecule and the target protein computed by pre-trained oracle [1]. Here, the mode is defined as high-reward molecule with low Tanimoto similarity [32] measured against previously accepted modes.
(a) 16 16 x 16 grid
Uniform
Maxfint
[x10
We consider TB-based implementation for our method and com- pare with various baselines including GFlowNets and reinforcement learning algorithms: DB, sub trajectory balance subTB]. TB, defined with uniform and MaxEnt backward policies, generative augmented flow networks [10, GAFN], and Advantage Actor-Critic [33, A2CJ. For the evaluation metric, analyze the trade-off be- tween the average score of the top 100 samples and the diversity of these samples. Additionally, to measure diversity, compute the average pairwise Tanimoto similarity following prior works. The detailed setting is described in Appendix B.
Trajectories sampled (x10*)
(b) 20 x 20 x 20 x 20 grid
Figure 6: L1 distance between Boltzmann distribution. PBP. GFN shows fastest learning tar- get distribution with respect to the observed trajectories.
Results. We depict the results in Figure One can see that our method, i.e,, PBP-GFN, outperforms the baselines in enhancing average score of unique top-100 molecules and the number modes found during training. These results highlight that PBP-GFN also can make improvements for environments with huge state space. Furthermore, one can see that our method yields low Tanimoto similarities between top-100 molecules with respect to the average reward. This verifies that our algorithm not only generates high-scoring samples but also diverse molecules.
Sequence generation
We consider four RNA sequence generation tasks that aim to discover diverse and promising sequences that bind to human transcription factors [4, 34, 35), where the action is appending prepending an amino acid. baselines, we consider the same baselines as the fragment-based molecule generation. In this task, conduct experiments on the following four benchmarks.

---

Num moddes (c 0c10g
Num modess (x309)
Avg of Toga 00
100
20
3.
SATN
Jung
285
30
A3C
Active round (x101)
Active round (x10²)
Active round (x10*)
Active round (x10')
(a) TFBind8
(b) RNA-A
1.6
1.0
12
1.0
(x 4.104)
1.2
Avg of Tap 100
Avg of Top- 100
0.0
Num modes
0.4
0.0
3.7
0.
0.6
round (x10')
(x10
(x10")
round (x100)
(c) RNA-B
(d) RNA-C
Figure 8: The performance on RNA sequence generation. The solid line and shaded region represent the mean and standard deviation, respectively. The PBP-GFN shows superiority compared to the baselines in generating diverse high reward sequences.
2-hamiming modes
(-MaddEnt
2.hammiin modes
2-hamming ng modes
suOTB DB
(x10*)
(x10')
(x10*)
(a) RNA-A
RNA-B
(c) RNA-C
Figure 9: The number of 2-hamming ball modes discovered during training. The solid line and shaded region represent the mean and standard deviation, respectively. The PBP-GFN shows superiority compared to the baselines in discovering diverse distinct modes.
TFBind8. This is the task to generate length-eight RNA sequences. The reward is computed by wet-lab measured DNA binding activity to Sine Oculis Homeobox Homolog 6 [34]. The mode is determined based on whether it is included in predefined set of promising RNA sequences [13].
RNA-Binding. This task generates length-14 RNA sequences. In this task. consider three different target transcriptions: RNA-A, RNA-B, and RNA-C [14, 36]. The mode is defined as an RNA sequence with reward higher than the threshold. In this task, also consider the 2-hamming ball modes [36], which defined as the local maximum among its intermediate neighborhoods defined by modifying components of the sequence.
Results. The results are presented in Figure 8. One can see that FBP-GFN shows faster convergence superior performance compared to the considered baselines in enhancing the average score of unique top-100 sequences and number modes during training. Furthermore, in Figure one see that method better discovers the diverse distinct modes that are separated far from each other, compared to the basclines.
5.4 Ablation studies
Comparing overall generated sample quality. To further analyze the overall sample quality, we provide the relative mean error [13] which measures the distance between the mean values of the empirical generative distribution and the target Boltzmann distribution. We present the results in Figure 10. One can that our method yields the lowest errors.

---

0.0
-02
-0.4
-0.
Relative mean errar
Relative mean error
Relative error
-0.4
-0.3
-0.2
Relative -0.6
TB
-0.6
CAPh 10P-01
(x10")
round (x10')
Active
(x10")
(x10')
TFBind8
RNA-A
RNA-B
(d) RNA-C
Figure 10: The relative mean error comparison. The solid line and shaded region represent the
mean and standard deviation, respectively. Our PBP-GFN yields the closest error to zero.
Num modes (x:101)
1.35
Too 100
1.0
Num modees (x10*)
0.9
Avg Top 100
0.0
0.99
0.9
T3
TB wm sutter
0.7
Iscalwardh CRN
PSP-07M
15 wilt baller
w/o botfer
Active found (x10*)
Active (x10P)
Active round (x10*)
Active (x10P))
(a) Comparison with local search
(b) Comparison with reward- swward-priorttized buffer
Figure 11: Comparison with off-policy sampling methods for exploitation. The considered task RNA-C. The solid line and shaded region represent the mean and standard deviation, respectively. One can see that PBP-GFN (I) shows competitive performance compared to GFNs with local search and reward-priont prioritized buffers, and (2) improves performance when combined with them.
Comparison with other exploitation methods. We further validate PBP-GFN by comparing or combining the pessimistic backward policy with local search [14] and reward- wwirrd-rioritized buffer [13] that are off-policy sampling methods and orthogonal to our pessimistic backward policy. We present the experimental results in Figure 11. One can observe that our method (1) shows similar performance compared to them and (2) consistently improves the performance when combined with them.
6 Conclusion
In this work, we identify the under-exploitation problem in flow matching due the large amount of unobserved flow lifted by the backward policy. To resolve this, we introduce pessimistic training of the backward policy for GFlowNets (PBP-GFN) that reduces the probabilities for unobserved backward trajectories leading to observed trajectories. Our PBP-GFN shows successful alternative to the prior backward policies and has demonstrated improved performance across eight benchmarks.
Limitation. As exploitation method, our PBP-GFN makes a trade-off between obtaining high-reward trajectories and diversified trajectories, i.e., there no free lunch in the exploitation- exploration trade-off. Although our approach maintains the diversity of high-reward sampled objects in the considered benchmarks, this may not hold for some environments where exploration is sig- nificant. We discuss such a setting in Appendix C. To relax this issue, one can reduce the learning rate for the pessimistic backward policy or incorporate an exploration-focused off off-policy sampling method. One can further control the trade-off by interpolating PBP-GFN with explorative GFNs, e.g.. MaxEnt, which can be an interesting future work direction.
Acknowledgements
This work partly was supported by Institute of Information & communications Technology Plan- ning& Evaluation (ITTP) grant funded by the Korea Dovernment(MST)) (No. JITP-2019-0-01900 Artificial Intelligence Graduate School Program(POSTECHH), the National Research Foundation of Korea(NRFF grant funded by the Korea government(MSTI) (No. Science Research Program through the National Research Foundation of Korca(NRF) funded by the
3366), Basic
10

---

Ministry of (dUS) (No. 2022RI IAA3A5052559)4 and GRDC(Global Research Devel- opment Center) Cooperative Hub Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Science and ICT(MSIT) (No. RS-2024-00436165))
References
[1] Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in 8, 15
Neural Information Processing Systems, 34:27381-27399 2021. 1, 2,
[2] Moksh Jain, Sharath Chandra Raparthy, Alex Hernández-Gard Garcia, Jarrid Rector- settor-Brooos, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. In International Conference on Machine Learning, pages 14631-14653. PMLR, 2023. 6,
[3] Miruna Cretu, Charles Harris, Julien Roy, Emmanuel Bengio, and Pietro Liò. Synflownet: Towards molecule design with guaranteed synthesis pathways. In ICLR 2024 Generative and Experimental Perspectives for Biomolecular Design (GEM) Workshop, 2024. 1,6
[4] Moksh Jain, Emmanuel Bengio, Alex Sermandez-Gariia Jarrid Rector- Brooos Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning,
pages 9786-9801. PMLR, 2022. 2, 6,
[5] Dinghuai Zhang, Hanjun Dai, Nikolay Malkin, Aaron Courville, Yoshua Bengio, and Ling Pan. Let the flows tell: Solving graph combinatorial problems with gflownets. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 11952 11952-11969 11969. Curran Associates, Inc., 2023.
1, 6, 7, 15
[6] Edward Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference in large language models. In International Conference on Learning Representations, 2024. 1,6
[7] Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. Journal of Machine Learning Research, 24(210):1-55, 2023. 1, 6,
[8] Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. Advances in Neural Information Processing Systems, 1155955-5667, 2022. 1, 2, 3, 6, 15
[9] Kanika Madan, Jarrid Rector-Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, Andrei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability. In International Conference on Machine Learning, pages
PMLR, 2023. 1, 6,
[10] Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of gflownets with local credit and incomplete trajectories. In International Conference on Machine Leurning, pages 26878-260 26890. PMLR, 2023. 1, 6,
(11) Hyosoon Jang, Minsu Kim, and Sungsoo Ahn. Learning energy decompositions for partial inference of gflownets. International Conference on Learning Representations, 2024. 1, 6
[12] Ling Pan, Dinghuai Zhang, Aaron Courville, Longbo Huang, and Yoshua Bengio. Generative augmented flow networks. In International Conference on Learning Representations, 2023. 1,
[13] Max Walt Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving gflownet training. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2023. 1, 2, 6, 7, 9, 10, 15, 16
[14] Minsu Kim, Taeyoung Yun, Emmanuel Bengio, Dinghuai Zhang, Yoshua Bengio, Sungsoo Ahn, and Jinkyoo Park. Local search gflownets. In International Conference on Learning Representations, 2024. 1, 6, 9, 10, 16
[15] Sobhan Mohammadpour, Emmanuel Bengio, Emma Frejinger, and Pierre-Luc Bacon. Maximum entropy gflownets with soft q-learning. International Conference on Artificial Intelligence and Statistics, pages 2593-2601. PMLR, 2024. 6,

---

[16] Tristan Deleu, Padideh Nouri, Nikolay Malkin, Doina Precup, and Yoshua Bengio. Discrete probabilistic inference as control in multi-path environments. arXiv preprint arXiv:2402.10309, 2024.
[17] Daniil Tiapkin, Nikita Morozov, Alexey Naumov, and Dmitry P Vetrov. Generative flow networks as entropy-regullarized rl. In International Conference on Artificial Intelligence and Statistics, pages 4213-4221. PMLR, 2024. 6
[18) Nikolay Malkin, Salem Lahlou, Tristan Deleu, Xu Ji, Edward Hu, Katie Everett, Dinghuai Zhang. and Yoshua Bengio. Gflownets and variational inference. In International Conference on Learning Representations, 2023, 6
[19) Jarrid Rector-Brooks Kanika Madan, Moksh Jain, Maksym Korablyov, Cheng-Hao Liu, Sarath Chandar, Nikolay Malkin, and Yoshua Bengio. Thompson sampling for improved exploration in gflownets. In ICML 2023 Structured Probabilistic Inference & Generative Modeling (SPIGM)
Workshop, 2023. 6
[20] Elaine Lau, Stephen Zhewen Lu, Ling Pan, Doina Precup, and Emmanuel Bengio. Qgfin: Controllable greediness with action values. arXiv preprint arXiv:2402.052.44 2024.
[21] Minsu Kim, Joohwan Ko, Dinghuai Zhang, Ling Pan, Taeyoung Yun, Woochang Kim, Jinkyoo Park, and Yoshua Bengio. Learning to scale logits for lemperature-conditional gflownets. arXiv preprint 2xXi:2331000282: 2023. 6, 16
[22] Yihang Chen and Lukas Mauch. Order- prder-preserving gflownets. In The Twelfth International Conference on Learning Representations. 6
[23] Yiheng Zhu, Jialu Wu, Chaowen Hu, Jiahuan Yan, Tingjun Hou, Jian Wu, et al. Sample- efficient multi-objective molecular optimization with gflownets. Advances in Neural Information Processing Systems, 36, 2023. 6
[24] Pouya M Ghari, Alex Tseng, Gökeen Eraslan, Romain Lopez, Tommaso Biancalani, Gabriele Scalia, and Ehsan Hajiramezanali. Generative flow networks assisted biological sequence editing. In NeurlPS 2023 Generative Al and Biology (GenBio) Workshop, 2023. 6
[25] Dinghuai Zhang, Nikolay Malkin, Zhen Liu, Alexandra Volokhova, Aaron Courville, and Yoshua Bengio. Generative flow networks for discrete probabilissic modeling. In International
Conference on Machine Learning, pages 26412 26412-26423 2-26428. PMIR, 2022. 6
[26] Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks. In Uncertainty in Artificial Intelligence, pages 518-528. PMLR, 2022.
[27] David Zhang, Corrado Rainone, Markus Peschl, and Roberto Bondesan. Robust scheduling with gflownets. In International Conference on Learning Representations, 2023. 6
[28] Minsu Kim, Sanghyeok Choi. Jiwoo Son, Hyeonah Kim, Jinkyoo Park, and Yoshua Ben- gio. Ant colony sampling with gflownets for combinatorial optimization. arXiv preprint AxXi:2403307044: 2024. 6
[29] Salem Lahlou, Tristan Deleu, Pablo Lemos, Dinghuai Zhang, Alexandra Volokhova, Alex Hernández- Garcia, Léna Néhale Ezzine, Yoshua Bengio, and Nikolay Malkin. A theory of continuous generative flow networks. In International Conference on Machine Learning, pages
18269-18300. PMLR, 2023. 6
[30] Dinghuai Zhang, Ricky Tian Qi Chen, Cheng- Hao Liu, Aaron Courville, and Yoshua Bengio. Diffusion generative flow samplers: Improving learning signals through partial trajectory optimization. In International Conference on Learning Representations, 2024.
[31] Marcin Sendera, Minsu Kim, Sarthak Mittal, Pablo Lemos, Luca Scimeca, Jarrid Rector-Brook Alexandre Adam, Yoshua Bengio, and Nikolay Malkin. On diffusion models for amortized inference: Benchmarking and improving stochastic control and sampling. arXiv preprint arXiv:2402.05008. 2024. 6
[32] Dávid Bajusz, Anita Rácz, and Károly Héberger. Why is tanimoto index an appropriate choice for fingerprint-based similarity calculations? Journal of cheminformatics, 7(1):1-13, 2015. 8
[33] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The
12

---

33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1928-1937, New York, New York, USA, 20-22 Jun 2016. PMLR.
[34] Luis A Barrera, Anastasia Vedenko, Jesse V Kurland, Julia M Rogers, Stephen S Gisselbrecht, Elizabeth J Rossin, Jaie Woodard, Luca Mariani, Kian Hong Kock, Sachi Inukai, et al. Survey of variation in human transcription factors reveals prevalent dna binding changes. Science, 3516680))1450-1455 2016. 8, 9, 16
[35] Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Bench- marks for data-driven offline model-based optimization. In International Conference on Machine Learning, pages 21658-21668 PMLR, 2022. 8
[36] Sam Sinai, Richard Wang, Alexander Whatley, Stewart Slocum, Elina Locane, and Eric Kelsic. Adalead: A simple and robust adaptive greedy search algorithm for sequence design. arXiv preprint (rXiv:201000214// 41, 2021. 9
[37] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019. 15
13

---

A Proof for error bound
We demonstrate that PBP-GFN yields lower error bound between the true marginalized distribution
Pe (z) and the target Boltzmann distribution PB (x) R(a) defined with the = =>EEx R(z),
compared to the conventional GFlowNets. To this end, derive the following bound:
-
P3 (x) < 2 2 Σ PB(T)+E
(6)
TEB(a)
where the e depends on the enror in the likelihood for trajectories over the observed trajecto-
ries, ErEB(E)) (Pr(T) (Pp(T) - Pn(r)). In this equation, our pessimistic backward policy maximizes ErE((z) PT( by maximizing the likelihood of EreB(.)) PB(T(x:) This better reduces the error
bound compared to the conventional backward policy.
The error bound in Equation (6) can be derived as follows:
Σ Pf(x) (x)
Σ
Σ
PF(T)- E Pn(T)
TET(2)
TET(#)
PF(T)
E PB(T)
+ +Σ
(x
Σ PF(T))
(x(x) E
PD(T)
TEB(x)
TEB(2)
TEB(2)
TEB(x)
Σ
Σ
PF(T) Σ PB(T)
+ +Σ
(x( E PF(T)
+ +
(x( E PB(T)
rEB(x)
TEB(Z)
rEB(2)
rEB(w)
Σ
Σ PF(T) Σ PB(T)
+ +
Pf Σ PE(T)
+ +E
(x(x)- Σ
PB(T)
TE)(w)
TEB(2)
TEB(x)
TEB(w)
Σ
Σ PP(T) Σ PB(T)
+2 +2 Σ
Σ PF(T)+ Σ PB(T)
TEB(a)
TEB(2)
TEB(x)
TEB(a)
= 2
Σ PB(T)
TEB(2)
where the error is associated with the error in trajectory flow matching the observed trajectories Σ)EB(A) (PF(T) - PB(T)).
14

---

B Experimental details
In all experiments, the backward policy is designed to have the same architecture as the forward policy, c.g., a feedforward network with the same hidden dimensions, but does not share the parameters with the forward policy. For all experiments, we set the learning rate for the pessimistic training of the backward policy as 1e-3. Our overall implementations for each benchmark follow the prior studies. Note that we consider both on-policy and off-policy settings. We use a single GPU of NVIDIA GeForce RTX 3090.
Hyper-grid environment. The implementations follow the prior study by Malkin et al [8]. We consider 16 x 16 x 16 hyper-grid, where the starting state is (0, 0, 0) and the action is incrementing one coordinate terminating. The reward is computed follows:
D
D
R(s) = Ro - 0.5
II
0.5 E (0.25, 0.5) 8d
+ 2 III 1
0.5 E (0.3, 0.4) 111
88
11 --1
d=1
d=1
where H = 16, D = and Ro is 1e-3 in our settings.
The forward policy is implemented with the feed-forward neural network that consists of two layers with 256 hidden dimensions and is trained with a learning rate of 1e-3. 3. The learning rate for Zg is 0.1. In this task, the training is on-policy. The GFlowNets are trained with the 64 trajectories sampled from the current policy in each round. We train the pessimistic backward policy to minimize EAAP for these 64 trajectories in each round.
Bag generation. The implementations follow the prior study by Shen et al. [13]. The action includes one of seven types of entities in the current bag with a maximum capacity of 15. If it contains seven or more repeats of any items, it has a reward 10 with 75% chance, and 30 otherwise. The threshold for determining the mode is 30.
The forward policy is implemented with the forw-fowward neural network that consists of two layers with 16 hidden dimensions and is trained with learning rate of 1c-4. The learning rate for Zo is 1e-2. In this task, the training is off-policy and consists of online and offline rounds. The online round uses 32 trajectories sampled from the forward policy with an exploration rate 0.1 and the offline round uses 32 trajectories sampled from the backward policy conditioned on the high-reward objects [13]. We store these trajectories (sampled during online and offline rounds) into the buffer B. In each round, we train the pessimistic backward policy to minimize 4риP with trajectories sampled from the buffer B where the N in Algorithm is eight. The buffer stores trajectories sampled during the previous 20 rounds.
Maximum independent set. The implementations follow the prior study by Zhang et al. [5]. In this task, the action is selecting a node construct the maximum independent set. The reward is the set size and the temperature. The number of training graphs and validation graphs are 4000 and 500, respectively. The graph contains around 200 to 300 nodes. Furthermore, the reward re-scaled with the temperature which is annealed during training, starting from 1 and ending at 500.
The forward policy is implemented with the graph isomorphism neural network [37] that consists of five layers with 256 hidden dimensions and is trained with learning rate of 1e-3. In this task, the training on-policy transition-basse with the 64 transitions s + s' in trajectories sampled from the current policy. We train the pessimistic backward policy to minimize the negative lig-liiel log PB (s(s") for these transitions within each training step.
training [5]. In each training step, the GFlowNets are trained
Molecule generation. The overall settings is similar to the prior study [1], and our implementations are built upon the released codes. 45 This task aims to generate a molecule, where the action is adding fragment. The number of available fragments is 72. The reward is computed by pre-trained function [1]. The reward is scaled to have the maximum value nearing 1.0, and the reward exponent set to 64.0. The mode is defined as a molecule with a reward higher than 0.97 and Tanimoto similarity lower than 0.65, measured against previously accepted modes.
The forward policy is implemented with the graph attention transformer that consists of four layers with 128 hidden dimensions and two attention heads, which is trained with a learning rate of 1c-4.
hhtp:://githubb... hhaps//gihhhh.co//cuusionoo com/eeunsiinnnaaamagftooittta
MIT License, Copyright (c) 2020 Recursion Pharmaceuticals
15

---

The learning rate for Ze is 1e-3. In this task, the training is off- off-policy, where 64 trajectories are sampled from the lagged forward policy Pr. whose parameters are updated F 0.95 0.05F in each round. We store these trajectories into the buffer B. In each round, we train the pessimistic backward policy to minimize Eppp with trajectories sampled from the buffer B where the N in Algorithm is eight. The buffer stores trajectories sampled during the previous 20 rounds.
TFBind8. The implementations follow the prior study by Shen et al. [13]. The action appending or prepending an amino acid to the sequence with maximum length of eight. The number of amino acids is four. The reward is pre-computed based on wet-lab measured DNA binding activity Sine Oculis Homeobox Homolog 6 [34], which is scaled between 0.001 to 1.0. The reward exponent is set to 3.0. The mode is determined based on whether included predefined set of promising RNA sequences [13].
The forward policy is implemented with the feed-forward neural network that consists of two layers with 128 hidden dimensions and is trained with a learning rate of 1e-4. The learning rate for Zo 1e-2. In this task, the training is off-policy and consists of online and offline rounds. The online round uses 16 trajectories sampled from the forward policy with an exploration rate 0.01 and the offline round uses 16 trajectories sampled from the backward policy conditioned on the high-reward objects [13]. We store these trajectories (sampled during online and offline rounds) into the buffer B, In each round, we train the pessimistic backward policy to minimize εуиp with trajectories sampled from the buffer B where the N in Algorithm is eight. The buffer stores trajectories sampled during the previous 20 rounds.
The implementations follow the prior study by Kim et al. [14]. The action appending or prepending amino acid to the sequence with maximum length of 15. The reward is scaled between 0.001 to 1.0, and the reward exponent is set to 8.0. The mode is determined based on whether is included in a predefined set of promising RNA sequences [21].
RNA-Binding.
The forward policy is implemented with the feed-forward neural network that consists of two layers with 128 hidden dimensions and is trained with learning rate of 1c-4. The learning rate for Ze 1e-2. In this task, the training is off off-policy and consists of online and offline rounds. The online round uses 32 trajectories sampled from the forward policy with an exploration rate 0.01 and the offline round uses 32 trajectories sampled from the backward policy conditioned on the high-reward objects [13]. We store these trajectories (sampled during online and offline rounds) into the buffer B. In each round, we train the pessimistic backward policy to minimize fPPP with trajectories sampled from the buffer B where the N in Algorithm is eight. The buffer stores trajectories sampled during the previous 20 rounds.
16

---

(a) conventional flow matching (entire trajectories)
(b) conventional flow matching (partially observed trajectories)
(c) PBP-GFN (partially observed )rajecorer))))
A synthetic example for the case where PBP-GEN may reduce exploration. The portion of the circle indicates the flow amount, and the color indicates the flow inducing the same- colored reward. The black and gray indicate the observed and unobserved trajectories, respectively. (a) The true reward of Tı, I2, and 23 are 0.5, 0.25, and 0.75, respectively. (b-c) Compared the conventional GFN, PBP-GFN may yield relatively low probability to the unobserved high-reward trajectory, by assigning a relatively high flow the observed high-reward trajectory.
Figure 12:
C Exploration-exploiatioo trade-off
As the case where our PBP-GFN may reduce exploration, we consider scenario where an unobserved high-reward trajectory may largely overlap with observed low-reward trajectory. Then, to explore the high-reward trajectory, one should assign a relatively high probability to the low-reward trajectory, Le., the opposite of the requiring exploitation. We exemplify and analyze this scenario in Figure 12, which illustrates how pessimistic training may reduce exploration by enhancing the exploitation of observed high-reward trajectories.
Despite the potential reduction of exploration, we would like to clarify that our method is still effective exploitation is significant in most environments. There is free lunch in the exploitation- exploration trade-off. One can further control the trade-off by interpolating PBP-GFN with explorative GFNs, e.g., MaxEnt, which can be an interesting future work direction.
17

---

D Broader impact
Improving the performance of GFLowNets can significanlyy influence various domains, particularly in biology (e.g., molecule and RNA sequence generation as discussed in Section 5). However, these improvements also pose potential risks including the creation of harmful drugs and the misuse of synthesized molecules.
18

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] is generally preferable to "[No]" it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and introduction accurately reflect the paper's contribution and characterizing the emper-exploitation problem in conventional flow matching of GFlowNets and resolving the issue by pessimistic training of backward policy. We also clarify the scope of the validation of our method: improved performance in enight generation benchmarks.
scope:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
19

---

Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of the work Section 6. summary, our PBP- GFN makes trade-off between exploitation and exploration, i.e., obtaining high-reward trajectories and diverse trajectories
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't I acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide detailed proof of how PBP-GFN improves the learning of target Boltzmann distribution in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
20

---

Answer: [Yes]
Justification: We provide the source code of our PBP-GFN including all the experiments in the attached code folder, which enables the reproduction of our experimental result.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the source code of our PBP-GFN including all the experiments in the attached code folder. In addition, the data is provided in the source code of baselines that we have cited.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
Soaeee//ooouuummssseeeeeeeeeeese conooocyy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: // //nips. wwwbbl//ggiidee/dddddtiiiisseese
for more details.
21

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the experimental settings including the model architecture, the number of sampled trajectories, and the reward for each task in Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yss]
Justification: Our experimental results include the standard deviation over three different random seeds. The standard deviation is described with the shaded region in each plot.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
22

---

Answer: [Yes]
Justification: We provide our compute resource, a single RTX 3090 GPU.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / wtw.rpeuuripp/eeee/eeeeee wwtppccccppccceecccccccccccccc htcp/ppiic/cccccccieeeeeeeeeee /tw.cccccccceeeeeeese
Answer: [Yes]
Justification: authors.
Our work conforms NeurlPS Code of Ethies, including the anonymity of the
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discussed the positive and negative social impacts last of Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and funetioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
23

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We cited the original paper that produced the code package or dataset in Section 5.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please refer the references.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Our paper does not release any assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not release any new assets.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve any crowdsourcing or research with human objects
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "xvV0Wxptua", "reviewer": "Reviewer_u41F", "summary": "This paper addresses the under-exploitation of high-reward objects in Generative Flow Networks (GFlowNets) due to the limited observation of trajectories during training. The authors propose PBP-GFN (Pessimistic Backward Policy for GFlowNets), which modifies the backward policy to maximize the observed backward flow, aligning it closer to the true reward. They argue that this pessimistic training scheme encourages the forward policy to assign higher probabilities to high-reward objects, even if they are under-represented in the observed trajectories. Experiments across eight benchmarks, including hypergrid, molecule, and RNA sequence generation, demonstrate that PBP-GFN improves the discovery of high-reward objects while maintaining object diversity.", "questions": "Please refer to weaknesses section above.", "limitations": "N/A.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper tackles an important issue in GFlowNet training – the potential for under-exploration of high-reward objects due to the sparsity of observed trajectories.  This issue is particularly relevant in complex domains with vast trajectory spaces.\n* The proposed solution, PBP-GFN, is conceptually simple and intuitive. Maximizing the observed backward flow to align with the true reward directly addresses the identified problem of under-exploitation. \n* The paper provides extensive experimental results across a variety of tasks, showcasing the effectiveness of PBP-GFN in enhancing the discovery of high-reward objects.", "weaknesses": "* While the intuition behind PBP-GFN is clear, the paper lacks a strong theoretical analysis to support its claims. The \"error bound\" analysis in Appendix A is limited in scope and doesn't offer a comprehensive understanding of the algorithm's convergence properties or its impact on the target Boltzmann distribution. \n* The core idea of modifying the backward policy in GFlowNets has been explored in prior works [1, 2, 15]. PBP-GFN, while achieving promising empirical results, doesn't present a significant conceptual departure from these existing approaches.  Its primary contribution appears to be a specific strategy for maximizing the observed backward flow, but the paper lacks a detailed comparison and analysis of how this strategy differs from or improves upon existing techniques.\n\n* The paper primarily focuses on metrics like the number of modes discovered and the average top-100 score. While these metrics are relevant for measuring exploration and exploitation, they provide a limited view of the overall quality and diversity of generated samples.  Evaluating PBP-GFN on more comprehensive downstream task-specific metrics would strengthen the paper's claims.\n\n*  While the paper claims that PBP-GFN maintains object diversity, the pessimistic training scheme inherently introduces a bias towards the observed trajectories. This bias could potentially lead to a reduction in exploration and the generation of degenerate or less diverse samples, especially in domains with significant uncertainty or where observed trajectories are not truly representative of the target distribution.  A deeper analysis and empirical evaluation of this potential bias would be useful.\n \n* The performance of PBP-GFN heavily relies on the quality and representativeness of the observed trajectories.  In scenarios where the initial observed trajectories are biased or incomplete, PBP-GFN could amplify these biases, hindering the discovery of truly novel and high-reward objects. The paper doesn't address this sensitivity or propose mitigation strategies. \n\n* The paper mainly compares PBP-GFN with other backward policy design methods. However, it doesn't provide a thorough comparison with other exploitation-focused techniques in GFlowNets, such as local search GFlowNets [3] or those focusing on higher-reward trajectories [2]. This limited comparison weakens the paper's claim of achieving superior performance in discovering high-reward objects.\n\n[1] Trajectory balance: Improved credit assignment in gflownets\n[2] Towards understanding and improving gflownet training.\n[3] Local search gflownets\n[4] Maximum entropy gflownets with soft q-learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer u41F,\n\nWe express our deep appreciation for your time and insightful comments. In what follows, we address your comments one by one.\n\n---\n\n**W1. The paper lacks a strong theoretical analysis to support claims.** \n\nWe agree that our paper lacks a strong theoretical analysis since we mainly focus on empirical performance improvements. We note that a strong theoretical analysis would be challenging (though valuable) since it requires thinking about the optimization landscape of GFlowNets, where no theoretical results have been made. Hence, we chose to provide insights in Appendix A by analyzing our PBP-GFN under strong assumptions. \n\n---\n\n**W2. Compared to existing backward policies [1,2,3], this work (A) does not present a significant conceptual departure and (B) lacks a detailed comparison.**\n\nWe respectfully disagree with (A). Unlike existing backward policies that focus on under-exploration or credit assignment issues, PBP-GFN is designed to tackle a new under-exploitation problem stemming from the unobserved backward flow.\n\nNext, to resolve (B), we will incorporate the following detailed comparisons with the existing backward policies in our future manuscript.\n\n- While **uniform backward policy [1]** and **MaxEnt backward policy [3]** assign a fixed probability to the backward transition for enhancing exploration, our pessimistic backward policy learns the probability of the backward transition for enhancing exploitation. \n- While **conventional backward policy [1]** and **sub-structure backward policy [2]** may enhance the exploitation by learning the backward flow to align with the forward flow or to improve the credit assignments, they do not directly reduce the unobserved backward flow and do not resolve the under-exploitation stemming from that.\n\n---\n\n**W3. Current metrics are insufficient to capture overall quality and diversity. Downstream task-specific metrics would strengthen the paper's claims.** \n\nWe would like to clarify that we have already incorporated downstream task-specific metrics following prior studies [1,3,4]. We consider the Tanimoto similarity and edit distance to measure the diversity of molecules and RNA sequences, respectively. \n\nTo further address your comments on the overall sample quality, we provide the relative mean error [2] which measures the distance between the mean values of the empirical generative distribution and the target Boltzmann distribution. We present the results in **Figure A** of our rebuttal PDF. One can see that our method shows superior performance.\n\n---\n\n**W4. PBP-GFN may reduce exploration in domains with significant uncertainty or where trajectories are not representative of the target distribution. A deeper analysis and empirical evaluation of this are recommended.**\n\nTo address your comments, we consider a scenario where the observed trajectories are not representative of the target distribution. In this domain, an unobserved high-reward trajectory may largely overlap with an observed low-reward trajectory. Then, to explore the high-reward trajectory, one should assign a relatively high probability to the low-reward trajectory, i.e., the opposite of the case requiring exploitation. \n\nWe examplify and analyze this scenario in **Figure B** of our rebuttal PDF, which (1) illustrates how pessimistic training may reduce exploration by enhancing the exploitation of observed high-reward trajectories and (2) provides an empirical evaluation of the exploration. One can observe that PBP-GFN may reduce the exploration in considered settings.\n\nDespite the potential reduction of exploration, we would like to clarify that our method is still effective as exploitation is significant in most environments. As discussed in **Limitation** section, there is no free lunch in the exploitation-exploration trade-off. One can futher control the trade-off by interpolating PBP-GFN with explorative GFNs, e.g., MaxEnt, which can be an interesting future work direction. \n\n---\n\n**W5. PBP-GFN can amplify biases in initially observed trajectories, hindering the discovery of novel objects, but the paper does not address this or mitigation strategies.**\n\nAs you mentioned, our method may struggle to discover novel objects when heavily biased towards initially observed trajectories. However, we would like to clarify that such failure cases are rare in practice, as demonstrated in extensive experiments. \n\nFurthermore, to mitigate the bias towards initially observed trajectories, one can reduce the learning rate of the pessimistic backward policy in the initial training rounds. Additionally, incorporating exploration-focused sampling, e.g., a noisy policy, can address this issue. We will add a discussion about this potential risk and mitigation strategies in our future manuscript.\n\n---\n\n**W6. The paper lacks a comparison with other exploitation techniques, e.g., local search or reward-prioritized buffer.**\n\nFirst, we clarify that local search and reward-prioritized buffers are sampling methods for off-policy training, that are orthogonal to our pessimistic backward policy, i.e., the methods are not directly comparable. One can combine both approaches to improve performance further. Note that we have already used the reward-prioritized buffer to implement our method and baselines (**Appendix B**).\n\nNevertheless, to address your comments, we verify the effectiveness of PBP-GFN by comparing or combining the pessimistic backward policy with local search and reward-prioritized buffer. We present the results in **Figure C** of our rebuttal PDF. One can observe that our method (1) shows similar performance compared to other techniques and (2) consistently improves the performance when combined with them.\n\n---\n\n[1] Trajectory balance: Improved credit assignment in GFlowNets, NeurIPS 2022\n\n[2] Towards understanding and improving GFlowNet training, ICML 2023\n\n[3] Maximum entropy GFlowNets with soft Q-learning, AISTATS 2024\n\n[4] Local search GFlowNets, ICLR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "10gYZKinuN", "reviewer": "Reviewer_smvU", "summary": "This paper proposes a pessimistic backward policy for GFlowNets (GFN), which maximizes the expectation of observed backward trajectories. This paper points out the under-exploitation of high-reward objects for previous GFN training methods and provides a deep analysis of this problem. Extensive experiments validate the superior performance of the proposed method.", "questions": "- How do you estimate Eq. 5 in practice? This is an important component of the method, but it is not described.\n- What's the meaning of $N$ in algorithm 1? Why does this method need multi-round gradient updates?", "limitations": "Limitations have been discussed in this paper.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-written. The illustrations of the proposed method are clear, and the toy examples provided are easy to understand.\n- This paper proposes a simple yet effective method to address the problems of under-exploitation in GFN training.\n- Extensive experiments on 8 benchmarks validate the efficacy of the method.", "weaknesses": "Does the auxiliary objective introduced in Eq. 5 affect the original GFN training objective? Since higher probabilities are assigned to the backward transitions of observed trajectories, could this impact the original assumptions of TB, DB, and other objectives? Do you observe instability during training after some episodes, once the model has seen a sufficient number of samples?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer smvU,\n\nWe express our deep appreciation for your time and insightful comments. In what follows, we address your comments one by one.\n\n---\n\n**W1. Does the pessimistic training objective affect the original assumption of GFlowNets training objective?** \n\n\nOur pessimistic training objective does not affect the original assumption of GFlowNet objectives, e.g., DB, TB, and subTB. The assumption of them, i.e., flow matching over entire trajectories $\\tau \\in \\mathcal{T}$ constructs target Boltzmann distribution, is valid for any design of backward policy, i.e., degree of freedom [1,2]. Our pessimistic training objective only modifies this backward policy within the GFlowNet objectives. \n\n---\n\n**W2. Did you observe any instability during training after the model had seen a sufficient number of samples?**\n\nIn our experimental setup, we observed no particular instability after the model had seen a sufficient number of samples. \n\n---\n\n**Q1. How do you estimate Equation (5) in practice?**\n\nIn practice, we estimate the gradients of **Equation (5)** using the stochastic gradients computed with mini-batch randomly sampled from the buffer $\\mathcal{B}$, as described in **Algorithm 1**. We will update our future manuscript to better reflect this point.\n\n---\n\n**Q2. What's the meaning of $N$ in Algorithm 1, and why does pessimistic training require multi-round gradient updates with $N$?**\n\nAs you mentioned, $N$ is a hyperparameter that specifies the number of update rounds for training the pessimistic backward policy. We require this as we use stochastic gradients to minimize **Equation (5)**, which involves multi-round updates over multiple mini-batches.\n\n---\n\n[1] GFlowNet Foundations, JMLR 24\n\n[2] Trajectory balance: Improved credit assignment in GFlowNets, NeurIPS 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xp0mpmDoQC", "reviewer": "Reviewer_CTvM", "summary": "The authors present a problem with GFlowNets training that originates from not having seen backward trajectories for a particular terminal state. The authors show that because of the lack of observed trajectories, the backward flows underestimate the probabilities for the observed flows, resulting in a (forward) policy that may not match the reward distribution. The authors propose to mitigate the problem by increasing backward flows for the observed trajectories. The authors provide experimental results in eight different environments, showing the superior performances of their method in terms of mode discovery and distribution fitting.", "questions": "1. What is the hypermeters for GAFN in Figure 9? Were the hyperparameters tuned for the experiments?\n2. The example in Figure 3 raises a question: if $P_B(x|\\tau)[i]$ is maximized, $P_B(x|\\tau)[j] \\approx 0; j \\neq i$, thereby matching it will cause $P_F(x|\\tau)[j] \\approx 0; j \\neq i$. In smaller state-space, it may not be a problem, but in a larger state-space, this should be an issue as it may cause $p(x) \\approx 0$ where x necessitates action j whose probability we've just made approximately 0. Curious to hear what you think about this/whether you've done experiments in large-scale environments.", "limitations": "Yes. Limitations are adequately addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Well-written motivation for the problem described.\n2. The solution presented is well-motivated and clearly described.\n3. The experiments include well-studied environments and baselines, showing the method's performance in exploitation.", "weaknesses": "1. The discussion about lacking backward flow given the unseen trajectories is valid. In practice, one can mitigate this by having a uniform backward policy with **a reward-prioritized replay buffer** [1], which also can increase the probability a trajectory yielding high rewards, thereby tackling the mentioned problem. I feel adding this baseline will improve the paper.\n\n2. As an exploitation method, at least one experiment with a larger state-space should be useful to show its performance in larger state-spaces. Hypergrid with a larger dimension and horizon can be one option for this.\n\n\n[1] Shen, Max W., et al. \"Towards understanding and improving gflownet training.\" International Conference on Machine Learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer CTvM,\n\nWe express our deep appreciation for your time and insightful comments. In what follows, we address your comments one by one.\n\n---\n\n**W1. One can mitigate under-exploitation using a reward-prioritized replay buffer. Adding this as a baseline will be helpful.** \n\nWe clarify that our experiments on the bag and RNA sequence have already incorporated the reward-prioritized buffer into all baselines and our method (**Appendix B**). Since the pessimistic backward policy and the reward-prioritized buffer are orthogonal, they are not directly comparable and one can combine them to train GFlowNets.\n\nNevertheless, to further address your comments, we also considered PBP-GFN without a reward-prioritized buffer and compared it with baselines using a reward-prioritized buffer. We present results in **Figure C(b)** of our rebuttal PDF. One can observe that the pessimistic backward policy (1) yields similar performance improvements compared to the reward-prioritized buffer, and (2) further improves their performance when combined with it.\n\n---\n\n**W2. As an exploitation method, at least one experiment with a larger state-space could be useful, e.g., larger hyper-grid.** \n\nWe first clarify that our molecule generation environment considers a sufficiently large state space sized up to $10^{16}$ [1]. To further address your suggestion about a larger hyper-grid, we conducted additional experiments on $40\\times 40 \\times 40 \\times 40$ hyper-grid. To the best of our knowledge, this is larger than any hyper-grid experiment conducted in the literature. We present the results in **Figure D(b)** of our rebuttal PDF. One can observe that our method still shows superior results compared to considered baselines. \n\n---\n\n**Q1. What are the hyperparameters for GAFN?**\n\nFor GAFN, we searched for the coefficient $\\alpha$ of intrinsic rewards within $\\{1 \\mathrm{e-}1, 1 \\mathrm{e-}2, 1 \\mathrm{e-}3\\}$. We designed the random network to have the same architecture as the policy network, but with an output dimension of one, and used a learning rate of $1 \\mathrm{e-}3$ for this network.\n\n---\n\n**Q2. In a large state space, PBP-GFN may cause $P_F(x)\\approx 0$ when $x$ necessitates an action $j$ but PBP-GFN have made probability of $j$ approximately $0$. Curious to hear what you think about this and whether you've done experiments in large-scale environments.**\n\nWe think that such a potential reduction in the exploration of unobserved objects is natural due to the exploration-exploitation trade-off. As PBP-GFN enhances the exploitation, this may reduce exploration for the unobserved objects in the large state space (discussed in **Limitation** section). However, we would like to clarify that one can simply relax this issue by reducing the learning rate for the pessimistic backward policy or incorporating an exploration-focused off-policy sampling method to observe $x$ with $P_F(x)\\approx 0$.\n\nFurthermore, it is worth noting that our method has already shown good performance in a large state space, e.g., molecule generation sized up to $10^{16}$. This implies that our method is still effective in environments with large state space, as exploitation is also significant for them.\n\n---\n\n[1] Trajectory balance: Improved credit assignment in GFlowNets, NeurIPS 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tN43yDRJ5x", "reviewer": "Reviewer_WPBt", "summary": "The paper successfully identifies and addresses the under-exploitation problem in the flow matching of GFlowNets. The proposed method, Pessimistic Backward Policy (PBP-GFN), adjusts the probabilities of backward trajectories to improve exploration and exploitation, achieving superior performance in various benchmarks.", "questions": "- When minimizing the negative log-likelihood in Eq.5, how to ensure, or do we need to ensure that the sum of the backward probability on a state is 1, and why the total amount of backward flows will not change claimed in line 157? In the provided codebase \"PBP.train_proxy\", it seems the sum of log_pb_actions will change, and the total number of flows will change. \n\n- This paper includes MIS from [1]. Can you include more difficult graph combinatorial optimization problems from [1], such as maximum cut? (Maximum clique is not necessary since it is related to MIS)\n\n[1] Zhang, D., Dai, H., Malkin, N., Courville, A., Bengio, Y., & Pan, L. (2023). Let the flows tell: Solving graph combinatorial optimization problems with gflownets. arXiv preprint arXiv:2305.17010.", "limitations": "Please see the **Weaknesses** part.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper first identifies the significant issue in conventional GFlowNets—under-exploitation due to unobserved flow in backward trajectories.\n- The paper provides a clear and detailed explanation of the methodology, including the training process and the loss function used for the backward policy.", "weaknesses": "- The PBP-GFN introduces additional complexity to the training process of GFlowNets. Pessimistic training of backward policy might be computationally expensive or hard to tune in different settings. The author should discuss this in the updated version of the paper. \n- In the experiment sections, this paper does not compare with some new GFlowNets methods, such as [1, 2], that claim improvement over the baselines. I wonder (1) Are PBP-GFN's results better than [1, 2]? (2) Can we combine the pessimistic backward policy with strategies, such as those in [1, 2], to further improve performance? I understand that each paper deals with GFlowNets from a different perspective, therefore, I am ok if not all answers are yes. \n- There are some missing references that are also quite relevant to the topic. [3,4]\n- I would raise my scores if my concerns are resolved. \n\n[1] Kim, M., Yun, T., Bengio, E., Zhang, D., Bengio, Y., Ahn, S., & Park, J. (2023). Local search gflownets. arXiv preprint arXiv:2310.02710.\n\n[2] Jang, H., Kim, M., & Ahn, S. (2023). Learning Energy Decompositions for Partial Inference of GFlowNets. arXiv preprint arXiv:2310.03301.\n\n[3] Chen, Y., & Mauch, L. (2023). Order-Preserving GFlowNets. arXiv preprint arXiv:2310.00386.\n\n[4] Tiapkin, Daniil, et al. \"Generative flow networks as entropy-regularized rl.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer WPBt,\n\nWe express our deep appreciation for your time and insightful comments. In what follows, we address your comments one by one.\n\n---\n\n**W1. The PBP-GFN introduces additional complexity, which may be expensive or inpractical in some settings. Can the authors provide a discussion on this point?** \n\nWe clarify that the training of the pessimistic backward policy does not require significant overhead and is practical in most cases, as it only requires computing the gradients of the negative log-likelihood for the given trajectories. This requires similar or higher complexity than the conventional backward policy but less than the sub-structure backward policy. Especially, this complexity (i.e., the computation of gradients) would be minor for real-world problems with expensive reward evaluations, e.g., molecular docking tools or wet-lab experiments. We will update our future manuscript to reflect these points.\n\n---\n\n**W2. Compared to independent lines of research for exploitation, e.g., local search GFN and LED-GFN, (A) does PBP-GFN show superior results, and (B) can we combine PBP-GFN with these approaches to further improve performance?** \n\nTo address your comments, we conducted new experiments by (A) comparing PBP-GFN with local search and LED-GFN and (B) combining PBP-GFN with them. Note that (B) is trivial since our pessimistic backward policy is orthogonal to local search and LED-GFN that address off-policy sampling and local credits. \n\nWe present the results in **Figure C(a)** and **Figure D(a)** of our rebuttal PDF. One can see that PBP-GFN (A) shows similar performance compared to local search GFN and LED-GFN, and (B) further improves performance when combined with them.\n\n---\n\n**W3. There are some missing references that are also quite relevant to the topic [2,3].**\n\nThank you for your valuable suggestions! We will incorporate them into the related works section of our future manuscript.\n\n---\n\n**Q1. (A) How to ensure the sum of the backward probability on a state is $1$, and (B) the provided codebase \"PBP.train_proxy\" seems change the sum of `log_pb_actions`, so the total backward flows will change.**\n\nWe would like to clarify that (A) is trivial. When we implement the backward policy using softmax, i.e., $\\sum_{s\\in\\text{parent}(s')}P_B(s|s')=1$, the summation of the backward probability on a state satisfies $\\sum_{\\tau\\in\\mathcal{T}(x)}P_B(\\tau|x)=\\sum_{(s_0\\rightarrow\\cdots\\rightarrow s_T)\\in\\mathcal{T}(x)} {\\prod_{t=0}^{T-1}P_B(s_t |s_{t+1})=1}$, by iterating the law of total probability over entire trajectories. Note that we do not constrain $\\sum_{\\tau\\in\\mathcal{B}(x)}P_B(\\tau|x)$ for observed trjajectories $\\tau \\in \\mathcal{B}(x)$ to be equal to one, but maximize it.\n\nFor (B), we would like to clarify that the sum of `log_pb_actions` corresponds $\\log P_B(\\tau|x)=\\sum_{t=0}^{T-1} \\log P_B(s_t|s_{t+1})$ for the given trajectory. The change of this does not affect the total amount of backward flow, i.e, $R(x)\\sum_{\\tau \\in \\mathcal{T}(x)}P_B(\\tau|x)$, as $\\sum_{\\tau \\in \\mathcal{T}(x)}P_B(\\tau|x)=1$ is always guaranteed over the entire trajectories.\n\n\n---\n\n**Q2. This paper includes MIS. Can you include more difficult graph combinatorial optimization problems from [1], such as maximum cut?**\n\nIndeed, we previously tried other graph combinatorial optimization problems from [1], e.g., the maximum cut problem, but could not reproduce the official results even when using the official implementation. For example, the official score for the maximum cut problem is around $700$, but running their implementation yields a score of $2000$ during the initial training round. To alleviate your concerns, we report the result of PBP-GFN applied to the maximum cut problem in **Figure D(c)** of our rebuttal PDF.\n\n---\n\n[1] Let the Flows Tell: Solving Graph Combinatorial Problems with GFlowNets, NeurIPS 2023\n\n[2] Order-Preserving GFlowNets, ICLR 2024\n\n[3] Generative Flow Networks as Entropy-Regularized RL, AISTATS 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers (**u41F**, **smvU**, **CTvM**, and **WPBt**) and area chairs,

We are deeply grateful for the time and effort you spent reviewing our manuscript. In what follows, we summarize our rebuttal PDF and planned revisions.

---

### Summary of rebuttal PDF

Our rebuttal PDF provides the following contents:

- **Figure A** provides experiments on a new metric for the overall quality of generated samples (**u41F-W3**).
- **Figure B** presents a synthetic example for the case where our PBP-GFN may reduce exploration (**u41F-W4**). 
- **Figure C** and **Figure D(a)** present the results of comparing and combining PBP-GFN with methods that improve exploitation (**u41F-W6**, **CTvM-W1**, **WPBt-W2**).
- **Figure D(b)** and **Figure D(c)** present the results on additional benchmarks (**CTvM-W2**, **WPBt-Q2**).

---

### Summary of planned revisions

We will update our future manuscripts to incorporate the following contents that have been addressed in our rebuttal:

- Clarification of the detailed conceptual differences with existing methods for backward policy modification (**u41F-W2**)
- Clarification of detailed implementations for pessimistic training (**smvU-Q1**)
- Additional references relevant to the topic (**WPBt-W3**)
- Additional experimental results measuring overall sample quality (**u41F-W3**)
- Additional experimental results comparing with other methods that improve exploitation (**u41F-W6**, **CTvM-W1**, **WPBt-W2**)
- Additional experimental results on the large-scale hyper-grid (**CTvM-W2**)
- Analysis for the case where our PBP-GFN may reduce exploration (**u41F-W4**) 
- Discussion of the strategies for mitigating risks in exploitation (**u41F-W4,5**)
- Discussion of additional time complexity (**WPBt-W1**)

Meta-review: This paper explores the under-exploitation problem in the flow matching of GFlowNets and proposes PBP-GFN.

There were some concerns about the lack of theoretical analysis, which should not be a reason for rejection. The authors addressed most of the concerns raised by the reviewers in their rebuttal. Most importantly, the authors compared their method with a couple of baselines suggested by the reviewers. Even though the proposed method achieves comparable performance to these baselines, they are orthogonal, and the authors show that one can gain more by combining PBP-GFN with these baselines. I recommend an acceptance.
Decision: Accept (poster)