Local Search GFlowNets
Authors: Minsu Kim
Taeyoung Yun
Emmanuel Bengio
Dinghuai Zhang
Yoshua Bengio
Sungsoo Ahn
Jinkyoo Park
Venue: ICLR 2024 spotlight (2024)
Keywords: GFlowNet
molecule optimization
biological sequence design
local search
reinforcement learning
Primary area: generative models
TL;DR: This paper suggest a powerful GFlowNet training method using local search.

Abstract: Generative Flow Networks (GFlowNets) are amortized sampling methods that learn a distribution over discrete objects proportional to their rewards. GFlowNets exhibit a remarkable ability to generate diverse samples, yet occasionally struggle to consistently produce samples with high rewards due to over-exploration on wide sample space. 
This paper proposes to train GFlowNets with local search, which focuses on exploiting high-rewarded sample space to resolve this issue. Our main idea is to explore the local neighborhood via backtracking and reconstruction guided by backward and forward policies, respectively. This allows biasing the samples toward high-reward solutions, which is not possible for a typical GFlowNet solution generation scheme, which uses the forward policy to generate the solution from scratch. Extensive experiments demonstrate a remarkable performance improvement in several biochemical tasks. Source code is available: \url{https://github.com/dbsxodud-11/ls_gfn}.

Published as a conference paper at ICLR 2024
LOCAL SEARCH GFLOWNETS
Minsu Kim" & Taeyoung Yun KAIST
Emmanuel Bengio Recursion
Dinghuai Zhang Mila, Université de Montréal
Yoshua Bengio
Sungsoo Ahn POSTECH
Jinkyoo Park KAIST, Omelet
Mila, Université de Montréal, CIFAR
ABSTRACT
Generative Flow Networks (GFlowNets) are amortized sampling methods that learn a distribution over discrete objects proportional to their rewards. GFlowNets exhibit remarkable ability generate diverse samples, yet occasionally strug- gle to consistently produce samples with high rewards due to over-exploration on wide sample space. This paper proposes train GFlowNets with local search, which focuses on exploiting high-rewardes sample space to resolve this issue. Our main idea is explore the local neighborhood via backtracking and recon- struction guided by backward and forward policies, respectively. This allows bias- ing the samples toward high-reward solutions, which not possible for typical GFlowNet solution generation scheme, which uses the forward policy to gener- ate the solution from scratch. Extensive experiments demonstrate remarkable performance improvement in several biochemical tasks. Source code is available: https:/ / / githhu..... com/ wbtp//uddddddddoooooddoodddddddd 11/1s_ggfiil
INTRODUCTION
Generative Flow Networks (GFlowNets, Bengio et al., 2021) are a family of probabilistic models designed to learn ewward-proportions distributions over objects, in particular compositional objects constructed from a sequence actions, e.g., graphs or strings. It has been used critical appli- cations, such as molecule discovery (Li et al., 2022; Jain et al., 2023a), multi-objective optimiza- tion (Jain et al., 2022b), biological design (Jain et al., 2022a), causal modeling (Deleu et al., 2022; Atanackovic al., 2023; Deleu et al., 2023), system job scheduling (Zhang et al., 2022a), and graph combinatorial optimization (Zhang et al., 2023b).
GelowNets distinguish themselves by aiming produce diverse set of highly rewarding samples (modes) (Bengio et al., 2021), which is especially beneficial in a scientific discovery process where need increase the number of candidates who survive even after screening by the true oracle function. In pursuit of this objective, the use of GFlowNets emphasizes exploration to uncover novel modes that differ significantly from previously collected data points.
However, GFlowNets occasionally fall short in collecting highly rewarding experiences as they be- come overly fixated on exploring the diverse landscape the vast search space during training. This tendency ultimately hinders their training efficiency, GFlowNets heavily relies on experien- tial data collected by their own sampling policy (Shen et al., 2023).
Contribution. In this study, we introduce novel algorithm, local search GFlowNets (LS-GFN), which is designed to en- hance the training effectiveness of GFlowNets by leveraging lo-
msddl
cal search in object space. LS-GFN has three iterative steps: (1) we sample the complete trajectories using GelowNet trajec- tories; (2) we refine the trajectories using local search; (3) we train GFlowNets using revised trajectories. LS-GFN is promis- ing, as we synergetically combine inter-mode global exploration and intra-mode local exploration. GelowNets induce inter-mode
Trainiing
Intra-mode Dyploration
Intro-mode Cyploration
GRowWic
Figure 1: Strategy of LF-
Correspondence Minsu Kim <min-su @ kaist.cckr>>

---

Published as a conference paper at ICLR 2024
exploration via the iterative construction of solutions from scratch. As shown in Figure 1, local search serves as means to facilitate intra-mode exploration.
Our extensive experiments underscore the effectiveness of the proposed exploration strategy for GFlowNets. To assess the efficacy of our method, we apply it to six well-established benchmarks encompassing molecule optimization and biological sequence design. We observe significant im- provement in the mode seeking and average reward of GFlowNets with our local search. The pro- posed method outperforms niques employed by various reinforcement learning baselines as well as sampling baselines, in terms of both the number of modes discovered and the value of top-K K rewards.
only prior GelowNet methods but also reward-maxinen maariimaaiiiizttoon tech-
2 RELATED WORKS
Advances and extension of GFlowNets. A GFlowNet is generative model that learns particle flows on a directed acyclic graph (DAG), with directed edges denoting actions and nodes signifying states of the Markov decision process (MDP). The quantity of flows it handles effectively represents the unnormalized density within the generation process. GFlowNets, when introduced initially by Bengio al. (2021) for scientific discovery (Jain al., 2023b), employed flow matching condition for their temporal difference (TD)-like training scheme. This condition ensures that all states meet the requirement of having equal input and output flows. Subsequent works have further refined this objective, aiming for more stable training and improved credit assignment. Notably, Malkin et al. (2022) introduced trajectory balance which predicts the flow along complete trajectories, resembling a Monte Carlo (MC) method to achieve unbiased estimation. Madan et al. (2023) proposed subtra- jectory balance, which is akin to TD(A) (Sutton, 1988), to train GFlowNets from partial trajectories. Furthermore, Zhang et al. (2023c) proposed quantile matching to better incorporate uncertainty in the reward function.
GFlowNets exhibits insightful connections with various research domains, enriching the synergy be- tween these areas. In the study by Zhang et al. (2022b; 2023a), the connection between GelowNets and generative models such as energy-basse based models (LeCun et al., 2006) and denoising diffusion probabilistic models (Ho et al., 2020) is investigated. Meanwhile, Malkin et al. (2023) shed light on the relationship between hierarchical variational inference (Ranganath al., 2016) and GFlowNets, providing comprehensive analysis of why GFlow Nets deliver superior performance. Additionally, the works of Pan al. (2022: 2023a;b) offer valuable insights into the integration of reinforcement learning techniques.
Improving generalization of GFlowNets in high reward space. In prior attempt overcome the low-reward exploration tendency of GFlowNets, Shen et al. (2023) suggested strategies such as prioritized replay training to target higher-reward regions, and structure- based credit assignment to identify shared structures among high-reward objects. Furthermore, they suggested new edge flow parametrization method called SSR, which predicts edge flows as a function of pairs of states rather than of a single state. Although Shen et al. (2023) share similar objective to our research, we dis- tinguish ourselves by employing a local search approach to steer GFlowNet towards the exploration of highly rewarding regions.
3 PRELIMINARIES
In this section, we introduce the foundational concepts underpinning GFlowNets, a novel generative model tailored for compositional objects denoted as x X. We follow the notation from Bengio et al. (2023). GFlowNets follow a trajectory- -ased generative process, using discrete actions to iteratively modify a state which represents a partially constructed object. This can described by directed acyclic graph (DAG), G = A). where S is finite set of all possible states, and A is subset of S x S, representing directed edges. Within this framework, we define the children of state E S as the set of states connected by edges whose head is s, and the parents of state the set of states connected by edges whose tail is s.
We define complete trajectory T (so ... s.) T from the initial state su to terminal state s, = x E X. We define trajectory flow as F(T) : T + Ryo. which represents the unnormalized density function of T. We define state flow as the total amount of unnormalized probability
2

---

Published as a conference paper at ICLR 2024
Z Rx)) R(x2)
Terminal state
F(5o + st)
$1
X1
((s))
Source
Sink
Initial state
I R(x2)
Sink
Terminal state
Illustration GFlowNet a toy environment with two objects 21, u2 is the total amount of flow in the source, and the sink is the storage for the flow of the terminal state æ1 and ar. The generative probability is: p(r1)= = R(x1)/ (R(x1) + R(x2)), and p(z2) = R(x2)/ (R(x)) + R(x2)).
Figure 2:
R(x1) + M(22)
flowing though state s: F(s) EtET:Er F(T), and edge flow as the total amount of unnormalized probability flowing through edge + 8:: F(s - 5) -= F(T).
We define the trajectory reward R(T) the reward of the terminal state of the trajectory
R ( (5g - .... 8n =)) R(x), i.e., the reward is determined only by the terminal state, and intermediate states do not contribute to the reward values.
We define the forward policy to model the forward transition probability PF(s'(s) from to child s'. Similarly, we also consider the buckward policy Pn(s(s') for the backward transition s' --+ where s a parent of
PF and PB are related the Markovian flow F' as follows:
Pr(s'(ss F(s -5) Pa(s|s') F(8-88) F(5))
F(s)
The marginal likelihood of sampling x E X' can be derived as PT (x) ErsT PF(T) where > x denotes a complete trajectory that terminates at x. The ultimate objective of GFlowNets is to match the marginal likelihood with the reward function, P. (z) R(z). PF. is also called terminating probability. See Figure for a conceptual understanding of GFlowNets.
The trajectory balance algorithm is one of the training methods that can achieve Pp (x) R(x). The trajectory balance loss LTB works by training three models, learnable scalar of initial state flow Ze ss F(so) Σ F(T), forward policy PF(5t- (s: 0), and backward policy PB Pp( to minimize the following objective:
Trajectory balance.
7o IT Pr (81
LIB(T;
log R(x) IT; PB (6t-1(s;;6)
Replay training of GFlowNets. One of the interesting advantages GFlowNets that they can be trained in an off-policy offline manner (Bengio al., 2021). To this end, prior methods often rely on training with replay buffers, which iterate two stages. (1) collect data = {ri...., TM by using the GFlowNet's forward policy Pr or exploratory policy, and (2) minimize the loss computed from samples from the replay buffer D. This training process leverages the generalization capability of the flow model to make good predictions visiting the entire trajectory space of T is intractable. This generalization capability highly relies on the quality of the training dataset D. This study investigates how make high-quality replay buffers D using local search method during the sampling step.
unseen trajectories, which is critical since
LOCAL SEARCH GELO (LS-GFN)
Overview. Our method is simple augmentation (Step B) of existing training algorithms for GFlowNets (Step A, and Step C). Our local search in Step B refines the candidate samples by
3

---

Published as a conference paper at ICLR 2024
Sampling
Refining
ACTGAA
ACTGAA
CTTGAA
MM
M
Pottoual
Priturel
MX
Iteration (f times)
Step C: Training (T times iteration)
Figure 3: Illustration of Local Search GFlowNet LFLG algorithm.
partially backtracking the complete trajectory using the backward policy PB and then reconstruct- ing with the forward policy Pp. This procedure is done multiple times to iteratively refine and construct highly rewarding samples.
Our algorithm trains Pp and PB by repeating these three steps (see also Fig. 3):
Step A. We sample a set of trajectories {rı,..., TM] using Pr(r).
Step B. We refine the M trajectories parallel for / iterations. For each iteration, we generate [r 3 from (71,-.. TM using local search by using Pn's i destroying and Pr's backtracking, and add (Tí.
.-. Tus . accept Tm + 7m (i.e. make transition) or reject Tm < Tm (i.c. make staying) with
into the training dataset D. Then, we choose whether
filtering rules for m = 1, ,,, M.
Step C. We train the GFlowNet by using training dataset D. We use reward prioritized sampling over D and use the sampled trajectories to minimize GFlowNet loss function such as trajectory balance.
4.1 STEP A: SAMPLING
We construct complete trajectory through sequential process of generating actions from scratch by using forward policy (T( This approach enables global exploration over different modes. It worth noting that within the GFlowNet literature, various techniques have been proposed to use (r( for exploration (Pan et al., 2022; Rector-Brooks et al., 2023). In this work, we employ the e-noisy method. It selects random action with probability and follows (s(s'(s) to sample the action with probability -.
4.2 STEP B: REFINING
After completing Step A, we have an initial candidate set samples T1, T2, "TM. In Step B, we make I iterations of local search for M candidate trajectories in parallel.
Taking a reprresentative among the M candidate trajectories, let = (so + + S, = r). Inspired by Zhang et al. (2022b), backtrack K Ktstep from complete trajectory into partial trajectory using PB. Subsequently, employing Pr. we sample K- K-step forward trajectory to reconstruct a complete trajectory from the partial trajectory.
Tback
8n
x)
Trecom
(5/--
x¹)
Note the --+ stands for a backward transition from one state to its parent state. We call the local search refined trajectory r':
(80
recan
We define the transition probability ((T'(T) along with its reverse counterpart ((T(7') as follows:
q(T'lT)= (=(TTock( (F) (Trooon), q(rlT')= = PB (Trocon (x') Pe (Tdossr))))
(4)
We now need to determine whether to accept or reject r' from g(F'lT). We present two filtering strategies, one deterministic the other stochastic.
Deterministic Filtering, We accept r' with following probability:
A (T, r') (((TT)>>(())
(5)
4

---

Published as a conference paper at ICLR 2024
ACTGAA
reject:
Pa(Tback)
CTTGA
Pr
Py
Pr
Ps(Frocon)
accept: fransit
ACTGAA
Backtracking with P
Reconstruction with F,
Filtering
Figure 4: Illustration of the 3-step refinement process of LS-GFN.
Stochastic Filtering. We accept r' using the back-and-forth Cectropolis-Hastinee (Hastings, 1970; Zhang et al., 2022b) acceptance probability:
R(T') ((r AT,,7) min [1. R(T) g(T|T')
Deterministic filtering offers advantages the context of greedy local search when aiming to maximize the rewards of candidate samples. In contrast. stochastic filtering can be viewed proc-progs sampling method within Markov chain Monte Carlo (MCMC) that helps maintain the sampling objective of GFlowNets, where seek generate samples from the distribution p(x) x R(x) while promoting diversity. In this work, we use deterministic filtering as a default setting but closely analyze its pros and cons Appendix B.2. Our overall process refinement, including backtracking, reconstruction, and filtering, is illustrated in Fig. 4.
Note that we gather both accepted and rejected trajectories and compile them into a training dataset D. To ensure that highly rewarded trajectories from DD receive priority during training, we use swward-based prioritized replay training (PRT) method (Shen et al., 2023). This approach increases the likelihood of using accepted trajectories within the training process.
4.3 STEP C: TRAINING
In this work, we use the TB objective function as the default objective for training on dataset D. In TB, train three models, model of initial state flow Zg. forward policy Pp (p 0), and a backward policy ( as follows:
((
Zo IT Pp (81186115 [Log
R(x)) 11 PB (8t-1 (5c 0): )
C(0: Ero(r)
Note that the transition probability ((r'(r) in Equation (3) is defined with PF(s++ 1s; 0) and P 0) where this means the local search capability also evolves with GFlowNets and im- proves throughout the training process.
Algorithm Local Search GFlowNet (LS-GFN)
1: Set <-
b Initialize training dataset.
2: for 1,...,, T do
b Iteration of training rounds
3: Sample nI.... TM ~ PF(T:0)
Step A: Sampling
4: for L...,,,,
Step B: Refining
5: for 1,... . do
6:
Propose T'm g(: Tm: 0) by Equation (4).
7:
Update D - D }
8:
Accept (Tm + Tim) or reject (Tm + Tm) by Equation (5).
9:
end for
10:
end
11:
Use the Adam optimizer achieve: arg min L(e; D).
Step C: Training
12:
end for

---

Published as a conference paper at ICLR 2024
We use reward-based prioritized replay training (PRT) (Shen et al., 2023), where we set PD(T) to sample a batch of trajectories B = [TI..... TM/ 90th percentile of the D and the 50% of the B is sampled from the below 90th percentile of the D.
where 50% of the is sampled from the above
Our method can similarly accommodate various other objective functions, such as DB, and SubTB. See Algorithm I for the detailed pseudocode of method.
5 EXPERIMENTS
We present our experimental results on biochemical tasks, including molecule optimization and bi- ological sequence design. In these settings, generating diverse samples with relatively high rewards is crucial for robustness to proxy misspecification (Bengio et al., 2023). To this end, we measure the accuracy of GelowNets using the relative gap the target reward distribution following Shen al. (2023). We also measure the number of modes discovered by GelowNets.
5.1 TASK DESCRIPTION
Let X be the of all objects that can be generated (i.c., the terminal state space), and T be the complete trajectory space which consists of all possible trajectories that can incrementally construct any z X. As different trajectories TI,..., TN can represent identical X. [T] 2 |.)
We consider two molecule optimization and four biological sequence design tasks:
QM9. Our goal generate a small molecule graph. We have 12 building blocks with 2 stems and generate a molecule with 5 blocks. Our objective is to maximize the HOMO-LUMO gap, which is obtained via a pre-trained MXMNet (Zhang et al., 2020) proxy.
sEH. Our goal is to generate binders of the sEH protein. We have 18 building blocks with 2 stems and generate molecule with 6 blocks. Our objective to maximize binding affinity to the protein provided by the pre-trained proxy model provided by (Bengio et al., 2021).
TFBind8. Our goal to generate a string of length of nucleotides. Though an autoregressive MDP is conventionally used for strings, we prepend-append MDP (PA-MDP) (Shen et al., 2023), in which the action involves either adding one token to the beginning or the end partial sequence. The reward is DNA binding affinity human transcription factor (Trabucco al., 2022).
RNA- RNA-Binding. Our goal is to generate string of 14 nucleobases. We consider the PA-MDP to generate strings. Our objective is to maximize the binding affinity to the target transcription factor. We present three different target transcriptions, L14-RNAI, L14-RNA2, and L14-RNA3, introduced by Sinai et al. (2020).
5.2 BASELINES
We consider prior GelowNet (GFN) methods and awward-maximization methods lines. Prior GFN methods include detailed balance (DB, Bengio et al., 2023), maximum entropy GFN (MaxEnt, Malkin et al., 2022), trajectory balance (TB, Malkin al., 2022), sub-trajectory balance (SubTB, Madan et al., 2023), and substructure-guiie 2023). For reward- maximization methods, we consider Markov Molecular Sampling (MARS, Xie et al., 2020), which is campling-based method known to work well in the molecule domain, and RL-based methods which include advantage actor-critic (A2C) with entropy regularization (Mnih et al., 2016), Soft Q-Learning (SQL, Haarnoja et al., 2018), and proximal policy optimization (PPO, Schulman et al., 2017).
base-
trajectory balance (GTB, Shen et al.,
5.3 IMPLEMENTATIONS AND HYPERPARAMETERS
For GFN implementations, strictly follow implementations from Shen et al. (2023) and implement only non-existing methods by ourselves. For all GFN models, we apply prioritized replay training (PRT) and relative edge flow policy parametrization mapping (SSR) from Shen et al. (2023). We run experiments with T 2, 000 training rounds for QM9, sEH, and TFBind8 and T = 5, 000 training rounds for RNA-binding tasks.

---

Published as a conference paper at ICLR 2024
Table 1: Accuracy of GFlowNets. Mean and standard deviation from 3 random seeds reported.
Method
QM9 (T)
sEH (1)
TFBind8 (1)
L14-RNA1 (1)
L14-RNA2 (T)
L14-RNA3 (T)
DB
93.16 = 0.94
95.26 a: 0.37
77.64 = 0.70
28.25 0.54
16.99
00221
DB LS-GPN
95,41 + 1,94
93.77 + 0.48
75.59 0.09
29.86 = 0.24
18.19
0.03
MaxPot
96.95 + 0.44
100.00 + 0.00
84.66 0.63
33.53 0.19
0.26
32.49 + 1.59
MaxFint IS-GPN
100.00 + 0.00
100.00 + 0.00
97.67 + 1.14
88.04 + 1.94
56.93 + 1.05
74.28 + 3.71
SabTB (0.9)
93.49 + 0.62
98.98 + 0.19
76.53 1.08
29.38
28.18
18.77
SubTB (0.9) LS-GFN
100.00 a: 0.00
100,00 + 0.00
76.54 =: 0.55
41.16 0.41
25.01 + 0.31
21.24 a 0,12
97.84 0.63
100.00 a: 0.000
85.63 0.35
33.47
0.35
32.70 0.59
TB LS-GFN
100.00 + 0.00
100.00 + 0.00
97 05.05 = 0.58
87,28 3.25
56.63 + 0.56
75.75 4 3,10
QM9
sEH
TFainds
1000
100.0
8009
80.0
80.0
0010
80.0
6000
20.0
50.0
50.0
10
10
13'
Trahinng Rounce
(TB LS-GFN)
MadEnt
SubTB (0
OTB
L14-RNA1
L14-RNA2
L14-RNA3
100.0
60.0
76.0
12.0
10
12"
12 13'
10'
10' 16'
Traiiing Rounce
(TB LL.GGFN)
MaxFre
SubTB
QTR
Figure S: Accuracy of GFlowNet various tasks. Ours stands for TB LS-
To ensure fairness in sample efficiency across all baselines, we maintain a consistent reward eval- uation budget for each task. This budget denoted as B, is determined by the number of candi- date samples per training round (M), and the number of local search revisions (I) resulting in B = M x (I + 1) = 32 for all baselines. for LS-GFN, we set M 4, and I 7 as default. We provide a detailed description of the hyperparameters in Appendix A.2.
5.4 EVALUATING THE ACCURACY OF GELOWNETS
We first evaluate how well our method matches the target reward distribution. As suggested in Shen et al. (2023), we measure the accuracy of training GelowNet p(ar; by using a relative error between the sample mean of R(x) under the learned distribution p(2; 0) and the expected value of R(x) given the target distribution p²(x)= = R(x)/ Evex R(x):
Acc (p (r; 0)) = 100 x min E Epp(:0) [R (x))] Epr(.) [R [Rx)] .1).
For all experiments, we report the performance with three different random seeds. We provide details of our experiments in Appendix A.
Table 1 presents the results of our method when integrated with different GFN training objectives. Note that our local search mechanism is orthogonal training methods, so we can plug our method into various objectives. As shown in the table, our method outperforms baselines and matches the target distribution in most cases. This highlights the effectiveness of local search guided by GFN policies on finding high-quality samples.
Figure shows the performance of our method and prior GFN baselines across training. We only plot results of prior GFN methods without local search and our method integrated with TB for clear visualization. For monitoring, we collect 128 on-policy samples every 10 training rounds and accumulate them, following Shen et al. (2023). Note that samples for computing relative error from the target mean have never been used for training. As shown in the figure, our method converges to the target mean faster than any other baselines.
7

---

Published as a conference paper at ICLR 2024
Table 2: The number of discovered modes. Mean and standard deviation from random seeds reported
Method
QM9
sEH
TFBind8 (t)
L14-RNAI (T)
L14-RNA2
LI4-RNA3
DB
635 5
217 =
304 =
5
420
120
DB LS-GPN
5
326 13
0
11+3
13+1
3=0
MaxEnt
701 +
316
8+2
7+3
Maxfnnt + LS-GPN
7
4831 = 148
317 2
33+2
31
1
(0.9)
5+1
SubTB (0.9) + LS-GFN
787 a 2
2434 a 60
314 =
16.4
13
7
TB
706
TB LS-GFN
793 4
5228 141
316 0
32
2
1
QM9
SEH
TFbind8
300
100
3800
240
100
200
1500
60
Trahhing
LS-GFN)
MexEnt
(0.9)
L14_RNA1
L14_RNA2
L14_RN [N__
Training
LS--FN)
MaxEnt
SUSTB (0.9)
GTB
Figure 6: Number of modes discovered over training. Ours stands for TB + LS-GFN.
Note: We conducted a local search only for training, not at the inference phase for fair comparison. For every inference- metric, such the relative metrie, compare method with other GFN baselines without the local search refining process.
5.5 EVALUATING THE NUMBER OF MODES DISCOVERED
In this experiment, we systematically assess our training process's ability to uncover numerous distinctive modes. In biochemical tasks, modes are defined as high-scoring samples that exceed specified reward threshold, and are distinctly separated based on predefined similarity constraint. To achieve this, evaluate both the reward magnitude and the diversity of generated samples. Detailed statistics regarding these modes can found Appendix B.4. To ensure the reliability of results, report performance across all experiments using three random seeds.
In Table 2, we present the outcomes of incorporating our method into various GFN training objec- tives. We see that our approach exhibits remarkable performance in mode diversity when compared to previous GFN techniques. This underscores the efficacy of our local search mechanism facil- itating exploration within intra-mode regions during training. Notably, our method showcases the most substantial improvements in RNA-binding tasks, where objects are comparatively longer than in other tasks. Complementing these findings, Figure visually represents the progression of mode discovery throughout training. Our method not only identifies the highest number of modes among the compared techniques but also stands out for its accelerated mode detection, underscoring its ef- ficiency. This is relatively surprising, since one common downside of training on higher rewards is to make the model greedier and less diverse (Jain et al., 2023c).
5.6 COMPARISON WITH REWARD MAXIMIZATION METHODS
We evaluate method against established techniques, including reinforcement learning baselines (PPO, A2C, SQL) and a sampling baseline (MARS). We use three metrics: number of modes dis- covered, mean rewards of the top 100 scoring samples out of evaluation samples accumulated across
8

---

Published as a conference paper at ICLR 2024
Number of Modes
Top-100 Reward
Unique Fraction
0000
0.03
100
Top- 0D Roward
0.03
Umgun Fracton
0.75
-00
200
0.47
025
.00
300
1200
1000
2000
-00
200
1200
1000
2000
400
000
1200
1000
2000
Trakiing Rounds
Ours LS-GFN)
MARS
A2C- Entropy
Soft Q-Leerning
PPO
Figure 7: Performances in QM9 task. The average among 3 independent runs is reported.
Number of Modes
Top-100 Reward
Unique Fraction
0.90
1.30
20
Top- Raward
0.81
Unique tracten
0.75
16
0.50
0.38
0.00
1200
3003
5000
Training Rounds
Ours (TB LSSGFNN)
TB
MARS
A2C- Entropy
Soft O-Leaming
PPO
Figure 8: Performances in L14-RNAI task. The average among independent runs reported.
training, and sample uniqueness. Sample uniqueness maximized at 1.0 when all samples are dis- tinct, while it will be zero when all samples are identical.
As shown in Figures 7 and 8, our method surpasses reward-maximizen wottddmaaiiizziinn methods in terms of mode- seeking capabilities. Reward maximization methods can lead to a high fraction of duplicated sam- ples, falling into non-diverse local optima. Our method consistently surpasses existing techniques in terms of the number of modes identified, which is only possible when both strong exploration and exploitation are achieved by the model.
We interpret these results by recalling the importance of the structure of both trajectory space (T) and object space (X)). Some inefficiencies reinforcement learning (RL) arise from the failure to account for symmetries, wherein multiple trajectories lead the generation of identical samples. GelowNets, which make use of this symmetry in their training objective, may very well waste less time visiting the same state from different paths, since they are trained know they are the same outcome. See Appendix B.1 for detailed results on the other four tasks.
5.7 ADDITIONAL EXPERIMENTS
Comparison between deterministic filtering and stochastic filtering, See Appendix B.2.
Experiments for hyperparameter T. We did experiments for the hyperparameter / we introduced, which is the number of revisions with local search; see Appendix B.3 for details.
Experiments for the number of modes metric. We investigated different ways of counting modes and closely compared LS-GFN with other algorithms; see Appendix B.4.
Experiment for acceptance rate. We measured the acceptance rate A (T,7') during training, re- flecting the success of the local search compared GFlowNet's sampling. We observed an in- teresting phenomenon: the rate is fairly steady, sigmifying consistent evolution between GFlowNet (PF(T;0) and the local search (i.c., PB (T 6) and PP (Trecon; 0)); see Appendix B.6.
6 DISCUSSION
In this paper, proposed novel algorithm: Local Search GFlowNet (LS-GFN). We found that LS-GFN has the fastest mode mixing capability among GFlowNet baselines and RL baselines and has better sampling quality than GFlowNets. Our method had been consistently applied to exist-

---

Published as a conference paper at ICLR 2024
ing GFlowNets algorithms with simple modifications. These results suggested that combining the inter-mode exploration capabilities of GFlowNets and intra- mode exploration through local search methods is powerful paradigm.
Limitation and Future Works. A limitation of LS-GFN lies in the potential impact of the quality of the backward policy its performance, particularly when the acceptance rate of the local search becomes excessively low. One immediate remedy is to introduce an exploratory element into the backward policy, utilizing techniques like e-greedy or even employing uniform distribution to foster exploration within the local search. A promising avenue for future research could involve fine-tuning backward policy enhance the local search's acceptance rate.
ACKNOWLEDGEMENT
We thank Nikolay Malkin, Hyeonah Kim, Sanghyeok Choi, Jarrid Rector- Brooor, Chenghao Liu, Ling Pan, and Max W. Shen for their valuable input and feedback this project.
REFERENCES
Lazar Atanackovic, Alexander Tong, Jason Hartford, Leo Lee, Bo Wang, and Yoshua Bengio. Dyngfn: Bayesian dynamic causal discovery using generative flow networks. arXiv preprint arXiv:2302.04178, 2023.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381-27394, 2021.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J. Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. Journal of Machine Learning Research, 24(210):1-55, 2023. URLhttps //jmlr.eeee orgg/.orrrggeee wwwrrg/ppprs///22 22 0364 html.
Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks. In Uncertainty in Artificial Intelligence, pp. 518-528. PMLR, 2022.
Tristan Deleu, Mizu Nishikawa-Toome Jithendaraa Subramanian, Nikolay Malkin, Laurent Char- lin, and Yoshua Bengio. Joint bayesian inference graphical structure and parameters with a single generative flow network. arXiv preprint wrXh:2235./99365 2023.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming, 1861-1870. PMLR, 2018.
WK Hastings. Monte carlo sampling methods using markov chains and their applications.
Biometrika, pp. 97-109, 1970.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-666 6851, 2020.
Moksh Jain, Emmanuel Bengio, Alex Hernandez-Gareia. Jarrid Rector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning, pp. 9786-9801. PMLR, 2022a.
Moksh Jain, Sharath Chandra Raparthy, Alex Hemnández-Gareia, Jarrid Rector-Brooks, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. In International Conference on Machine Learning, 2022b. URL https:://////....t...t //pp///pp..../ CorpusID:25309771
Moksh Jain, Tristan Deleu, Jason Hartford, Cheng- Hao Liu, Alex Gomandez-Garrch and Yoshua Bengio. Gflownets for ai-driven scientific discovery. Digital Discovery, 2(3):557-577, 2023a.
10

---

Published as a conference paper at ICLR 2024
Moksh Jain, Tristan Deleu, Jason Hartford, Cheng-Hou Hao Liu, Alex Hernández-Garchhn and Yoshua Bengio. GFlowNets for AI-driven scientific discovery. Digital Discovery, 2023b.
Moksh Jain, Sharath Chandra Raparthy, Alex Hemándezz-Garciá, Jarrid Rector-Brooks Brooks, Yoshua
Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. In International Conference on Machine Learning, pp. 14631-14653, PMLR, 2023c.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Shibo Li, Jeff M Phillips, Xin Yu, Robert Kirby, and Shandian Zhe. Batch multi- active learning with budget constraints. Advances in Neural Information Processing Systems, 35:995- 1007, 2022.
Kanika Madan, Jarrid Rector-Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, An- drei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergenee and stability. In International Conference on Machine Learning, pp. 23467-23483. PMLR, 2023.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. Advances in Neural Information Processing Systems, 35:5955-5967, 2022.
Nikolay Malkin, Salem Lahlou, Tristan Deleu, Xu Ji, Edward J. Hu, Katie Elizabeth Everett, Dinghuai Zhang, and Yoshua Bengio. GFlowNets and variational inference. International Con- ference on Learning Representations (ICLR), 2023.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928 1988.197.60 PMLR, 2016.
Ling Pan, Dinghuai Zhang, Aaron C. Courville, Longbo Huang, and Yoshua Bengio. Generative augmented flow networks. International Conference on Learning Representations (ICLR), 2022.
Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of GFlowNets with local credit and incomplete trajectories. International Conference on Machine Learning (ICML), 2023a.
Ling Pan, Dinghuai Zhang, Moksh Jain, Longbo Huang, and Yoshua Bengio. Stochastic generative flow networks. Conference on Uncertainty in Artificial Intelligence, 2023b.
Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In International conference on machine learning, pp. 324-333. PMLR, 2016.
Jarrid Rector-Brooks, Kanika Madan, Moksh Jain, Maksym Korablyov, Cheng-Hao Liu, Sarath Chandar, Nikolay Malkin, and Yoshua Bengio. Thompson sampling for improved exploration in gflownets. arXiv preprint arXiv:2306.1 17693, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1xXiv:1707.06344 2017.
Max W Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving GFlowNet training. In International Conference on Machine Learning, pp. 30956-30975. PMLR, 2023.
Sam Sinai, Richard Wang, Alexander Whatley, Stewart Sloeum, Elina Locane, and Eric Kelsic. Adalead: A simple and robust adaptive greedy search algorithm for sequence design. arXiv preprint 11X1v:2010.02141, 2020.
Richard Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3:9 44, 1988.
11

---

Published as a conference paper at ICLR 2024
Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Benchmarks for data-driven offline model-based optimization. In International Conference on Machine Learn ing. pp. 21658-21676. PMLR, 2022.
Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. Mars: Markov molecular sampling for multi-objective drug discovery. In International Conference on Learning Representations, 2020.
David Zhang, Corrado Rainone, Markus Peschl, and Roberto Bondesan. Robust scheduling with gflownets. In The Eleventh International Conference on Learning Representations, 2022a.
Dinghuai Zhang, Nikolay Malkin, Zhen Liu, Alexandra Volokhova, Aaron Courville, and Yoshua Bengio. Generative flow networks for discrete probabilistic modeling. In International Confer- on Machine Learning, pp. 26412 26428. PMLR, 2022b.
Dinghuai Zhang, Ricky T. Q. Chen, Nikolay Malkin, and Yoshua Bengio. Unifying generative models with GFlowNets and beyond. International Conference on Machine Learning (ICML) workshop of Beyond Bayes: Towards Universal Reasoning Systems, 2023a.
Dinghuai Zhang, Hanjun Dai, Nikolay Malkin, Aaron Courville, Yoshua Bengio, and Ling Pan. Let the flows tell: Solving graph combinatorial optimization problems with gilownets. arXiv preprint arXiv:2305. 17010000 2023b.
Dinghuai Zhang, Ling Pan, Ricky T. Q. Chen, Aaron C. Courville, and Yoshua Bengio. Distribu- tional GelowNets with quantile flows. arXiv preprini 2302.05793, 2023c.
Shuo Zhang, Yang Liu, and Lei Xie. Molecular mechanics-drive graph neural network with multi- plex graph for molecular structures. arXiv preprint arXiv:2011.07457 2020.
12

---

Published as a conference paper at ICLR 2024
A EXPERIMENTAL SETTING
A.1 DETAILED IMPLEMENTATION
For the GetowNets policy model, we use an MLP architecture with relative edge flow parameteriza- tion (SSR) suggested in Shen et al. (2023). Given pair of states ss,s"), we encode each state into a one-hot encoding vector and concatenate them to pass as an input of the forward/backwwrd policy network. The number of layers and hidden units varies across different tasks, which is listed in Ta- We use the same architecture with different parameters to model forward and backward poli- cies. We initialize log Zg to 5.0. Following Shen et al. (2023), we clip gradient norms to a maximum of 10.0 and policy logit predictions to a minimum -50.0 and maximum 50.0. To implement find that introducing separate neural net- Ex
DB and SubTB, which require state flow predictions, work for mapping fo (s) - R+ is more useful than SSR, fo(s) Σ o' Echill(s) fo 8"). Please refer Figure 9.
800
600
Sumber of N/000
302
300
100
Triening
1615
(a) Nomber of
(h)
(0.9)
Figure 9: Experiments on the different parametrization state flow DB and SubTB.
A.2 HYPERPARAMETRRR
For hyperparameters of GFlowNets, we do not change the initial setting proposed by Shen et al. (2023). For all tasks, we use ADAM (Kingma & Ba, 2015) optimizer with learning rate 1 10 for log Zo. 1 x 10-4 for forward and backward policy. We use different reward exponent 3 to make p(:; f) x R R²(x) and reward normalization constant suggested in Shen et (2023) except for the RNA task, which is newly suggested by us. For the RNA task, we use reward exponent of 8 and scale the reward to a maximum of 10.
Table 3: GFlowNet hyperparameters for various tasks
Tasks QM9 sEH
Number of Layers
Hidden Units
Reward Exponent (3)
Training Rounds (T')
1024
2,000
1024
2,000
TFBind8
128
2,000
RNA-binding
128
5,000
For LS-GFN, we have set the number of candidate samples as M and the local search interaction
to 7 = as default values. In contrast, other GFN models without local search employ a default value of M 32 ensure fair comparison of sample efficiency.
A.3 HYPERPARAMETER TUNING FOR RI. BASELINES
To implement RL baselines, we also employ the same MLP architecture used in GFlowNet baselines. We find optimal hyperparameter by grid search on the QM9 task terms of the number of modes. For A2C with entropy regularization, we separate parameters for actor and critic networks and use
a learning rate of 1 x 10-4 selected from {1 x 10-5 x 10-4 x 10 10-4, -,5 5 x 10-3 - x 10-33
with entropy regularization coefficient x 10 selected from {1 x 10 10-4 x 10
10 10-2). 10 10-4.11
For Soft Q-Learning, we use learning rate of 1 x 10 -4 selected from {1 x 10 - 10 5 10 rate x 10 selected from {1 x 10 5 regularization coefficient x 10-2 selected from (1 10-4 10 1 10-2).
10-3) For PPO,
employ entropy regularization term and use learning
x 10 M
10-5, 1 x 10
x 10 10-3, 10-3) with entropy
13

---

Published as a conference paper at ICLR 2024
B ADDITIONAL EXPERIMENTS
B.1 CLOSER COMPARISON WITH RL BASELINES
We also assess our approach against RL baselines across four additional tasks, as detailed in Chapter 5.6. In Figures 10, 11, 12, and 13, we present the comprehensive results. These findings demonstrate that our method outperforms RL baselines, particularly in the detection of diverse modes. While most RL methods yield subpar unique fraction by producing duplicated samples concentrated in narrow, highly rewarded regions, our approach excels seeking remarkable modes, resulting in a wide variety of highly rewarded samples.
Number of Modes
Top-100 Reward
Unique Fraction
1001
Hambe do
3000
14p-100 Fossord
6.45
005
2000
097
Training
Curs (1B LSS-Gin
18
MARSS
A2C 22C-Erre Entrogy
Solt Q-i namming
PPO
Figure 10: The sEH task.
Number of Modes
100-100 Reward
Unique Fraction
Namber ooddde
0.50
0.7
-00
0:44
020
400
300
1200
1000
2000
400
302
1200
3000
2000
300
1200
3000
2030
Trint Rourdbs
Clax (TB 1 S-GGiN)
1R
MARS
AFC Frioooy
Soft warning
190
Figure 11: The TFbind8 task.
Number Modes
Top-100 Reward
Unique Fraction
of Moree
Top 100 Deeoord
0.73
Lviqua
0.00
0.2
1830
2000
-000
1200
2005
3003 Rounds
4000
1000
2002
4000
8030
Taaling
(TB LS-GPN)
MARS
Endrpp
Q-Lamening
100
Figure 12: The L14 14.2 task.
Number Modes
Top-100 Reward
Unique Fraction
1.00
1.00
Number Modee
Ttp 000 Rownert
Poacken
0.00
0.80
000
Hounds
Oura (TB LB-GRN)
MARB
ADC Eidroy
Q-Eaaaring
PPO
Figure 13: The L14_RNAY task.
14

---

Published as a conference paper at ICLR 2024
B.2 CLOSER COMPARISON BETWEEN DETERMINISTIC FILTERING AND STOCHASTIC FILTERING
We also compare the different filtering strategies we proposed in the methodology section. We conduct experiments on the QM9, sEH, and TFbind8 tasks with TB as an underlying GFN training method. For evaluation, we generate 2048 samples from the trained model. Experiment results are reported in Table 4. As depicted in Table 4, the stochastic filtering strategy yields a wider range of solutions, emphasizing diversity, whereas the deterministic strategy places greater emphasis on maximizing high-scoring rewards. Consequently, these two filtering strategies can be selected based on distinct objectives or purposes.
Table Analysis on Different Filtering Strategies
Task
Filtering Strategy
Accuracy
Top 100 Reward
Top 100 Diversity
Uniq. Fraction
QM9
Stochastic
100.00 0.00
0.59 0.01
0.43 0.00
0.97 0.00
Deterministic
100.00 0.00
0.61 0.02
0.42 0.00
0.96 0.01
sHH
Stochastic
100.00 = 0.00
6,84 = 0.01
0.30 # 0.00
1.00 = 0.00
Deterministic
100.00 = 0.00
6,87 = 0.01
0.29 + 0.01
1.00 # 0.00
TFbind8
Stochastic
99.23 1.09
0.97 0.00
1.98 +: 0.02
0.96 +: 0.00
Deterministic
100.00 0.00
0.97 0.00
1.94 +
0.95 0.00
B.3 ABLATION STUDY OF AND M
We investigate the effect of the number of revision steps on reward and diversity. When we set the number of revision steps as O, steps as batch size, we generate single sample and apply local search repeatedly. We conduct experiments on the QM9 task with TB as an underlying GFN training method. Table presents the performance across different numbers of revision steps. As shown in the table, we confirm that the mean the top 100 rewards consistently increases as the number of revision steps increases due to strong local exploration, while the unique fraction of samples gradually decreases.
typical GFN method. When we set the number of revision
Table 5: Effect of the number of revision steps Reward and Diversity
1
M
Num. Modes
Accuracy
Top 100 Reward
Top 100 Diversity
Uniq. Fraction
0
32
699 + 14
98.46 + 2.17
0.57 0.01
0.43 + 0.00
0.98 0.00
1
16
752 7
99.85 0.16
0.57 0.01
0.43 0.00
0.98 0.00
3
8
781
100.00 0.00
0.59
0.42 0.00
0.97 0.00
4
100.00 0.00
0.60
0.43 0.00
0.97
2
800
100.00 0.00
0.61
0.42 0.00
0.96 0.01
1
793
100.00 0.00
0.62
0.00
0.95 0.01
15

---

Published as a conference paper at ICLR 2024
B.4 EXPERIMENTS ON SEVERAL NUMBER OF MODES METRIC
How to define mode is not a trivial problem. All samples whose reward is above a certain threshold cannot be considered modes. Therefore, conduct experiments several different metrics for defining modes.
First, for molecule optimization tasks, we use the Tanimoto diversity metric. We define mode as follows. For all samples whose reward is above a certain threshold level, we only accept samples that are far away from previously accepted modes terms of diversity metric.
For biological sequence design tasks, we define mode as a local optimum among its intermediate neighborhoods. We can define the neighborhood as n- hamming ball, which means that we can make x from zosighbor by modifying n components of the sequence following the definition intro- duced by Sinai al. (2020).
Figure 14 shows the performance of our method and prior GFN methods in terms of a number of modes. As shown in the figure, our approach outperforms other baselines when the definition of the mode is changed. We also find that when we eliminate a similar sample from the modes, GTB shows promising results among all the other prior GFN methods. Figure 15 also exhibits similar trend.
Diversily 0.00
Diversily 0.50
Diversily 2 0.75
30
400
20
200
420
850
1820
2002
820
1230
2002
1200
1022
2002
Trshhing Rounds
(TB LS-OFN)
MaxEnt
D6
SubTB (0.9)
Figure 14: Experiments on several number of modes metrics. Experiments are conducted on QM9. The diversity is measured by 1 Tanimoto similarity.
No Constraint
1-Hamming Ball
2-Hamming Ball
58020
50
40030
12.5
50000
15
7.5
20020
5.0
10000
0.0
2000
4200
3000
Training Rounds
Ours (TB LS-GFN)
TB
MaxEnt
D6
SubTB (0.9)
Figure 15: Experiments on several number of modes metrics, Experiments are conducted on L14 RNAI.
16

---

Published as a conference paper at ICLR 2024
B.5 ABLATION STUDY OF K
We investigate the effect of the number of destruction and reconstruction steps on the performance of method. For default, I. We conduct an ablations study of K on RNA task. As shown in the Figure 16, we find that when we increase K. we can generate more diverse samples while we can achieve higher reward by decreasing K. When k = 4, we achieve the highest number of modes discovered across training.
K [(L 1)/2], where L the total length of the object
Number of Modee
Top-100 Reward
Unique Fraction
6.7
1,0005
Top: 100 Roward
2-3075
0,750
030
03000
1230
3000
00C2
1000
2503
3000
500
9000
2003
4000
Training Rourds
K044
K-10
K-14
Figure 16: Ablation study on k. The average value among 3 independent runs is reported.
17

---

Published as a conference paper at ICLR 2024
B.6
LOCAL SEARCH ACCEPT RATE EXPERIMENTS
0.55
0.00
m
Accept Rate
0.45
Accept Rate
0.40
0.40
MM
0.30
~/ wm Mpd mm
0.25
1000
4000
Training Rounds
Training Rounds
Stodtaelic Fillenn
Deberbraa:: Filerrg
Stechool: I Basrin
Deternitdee:Taatty
(a) Accept Rate
QM9
(b) Accept Rate L14 2J4_RNAI
Figure 17: Experiments the local search accept of different filtering strategies.
In Step B of enhancing the sampled trajectories from PP(T) through a local search guided by P (Táestr)) and PT((Teco)), assess the acceptance rate and decide whether to accept or reject the new suggestion generated by the local search.
Recapping, in deterministic filtering, accept r' with the following probability:
A (T, = [(((r)>>(())
Additionally, in stochastic filtering, we accept r' based on the Metropolis- acceptance prob- ability:
R(T') ((Fl(r) min 1, R(F) ((T|T')
A min
The acceptance rate, denoted A (r,T'), gauges how effectively local search enhances the perfor- mance compared to Pr(t). An intriguing experiment involves tracking the acceptance rate during training to observe the dynamic interplay between PF(T) and the local search mechanisms (i.e., PB((Testron and PF(Treso)) The ideal outcome would manifest as a stable acceptance rate, signi- fying that (F(T) evolves efficiently during training, it receives valuable support from the local search, which in turn evolves effectively with the aid of well-trained P3 PB (Tdessory) and Pp (Treco))
As demonstrated in Figure 17, the acceptance rate remains consistently stable, serving as confirma- tion that our LS-GFN training maintains stability while evolving both (F(T) and the local search components (Pa Fdestroy) and (T(Teccm)) mutually supportive manner.
The acceptance rate in deterministic filtering is lower compared to stochastic filtering due to its stricter acceptance criteria. These rates consistently fall below 0.5 each training iteration, indi- cating that only small proportion of successfully refined trajectories contribute significantly to the improvement of the GFlowNet training process.
18


Reviews:
[{"review_id": "jazpW7vg2h", "reviewer": "Reviewer_wUkH", "summary": "- This paper introduces local-search generative flow networks (LS-GFN), a method to improve exploitation in generative flow networks (GFNs).\n- The authors argue that, while GFNs explore effectively, their learning is hampered by exploitation — that is, they sample candidate objects that are mostly good but fail to obtain the highest rewards.\n- To address this limitation, they propose to refine candidate objects by backtracking\n- They validate LS-GFNs on real-world biochemical testbeds, where LS-GFNs significantly outperforms alternatives.\n- The paper also includes a well-written introduction to GFNs (Section 3) which covers all the necessary background for the paper.", "questions": "- Why call it “destroy”  since the original trajectory isn’t always discarded? A more intuitive name could be “backtrack”, “rewind”, or anything that doesn’t suggest destruction.\n- The top plots in Figure 5 are strange: it looks like some curves go beyond 100% accuracy. Could you either fix them so we can still see the curves on the plot, or explain what is happening?\n- How can LS-GFNs recover from a biased backward policy? In other words, assume the forward policy is fine but the backward policy always backtracks to states which yield the same (high reward) candidate objects — how can LS-GFNs overcome this lack of exploration?\n- Please confirm that lines 7 and 8 in Algorithm 1 aren’t swapped. If they aren’t (which partially addresses my question above), wouldn’t swapping them and extending $\\mathcal{D}$ with $\\tau_m$ further improve exploitation? Maybe this should be added as an ablation as well.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Novelty: the method is novel as it cleverly takes advantage of both the probabilistic forward and backward policy of GFNs.\n- Significance: the proposed methods exhibits excellent empirical results and drastically outperforms the other baselines. I wish there were more of them, but more on that below.\n- Clarity: the paper is well written and provides an exceptionally clear and concise introduction to generative flow networks. Moreover the method is quite simple (I think I could replicate the authors’ results) so it’s likely to be adopted by the community.", "weaknesses": "- While the method itself is simple and well presented, it remains a little unclear the relative importance of different components. For example, the exposition focuses on the trajectory balanced objective — does this mean LS-GFNs don’t work with different training objectives? The paper also mentions building on top of Shen et al., 2023 which introduces prioritized replay training (PRT) to GFNs, but this method doesn’t appear as a baseline — so how much of the improvements are due to PRT and how much is due to the local search?\n- In addition to the hyper-parameter $I$ (number of revisions with local searches), I wished the number of backtracking steps and the acceptance rate were also studied — how much tuning did they require in order to get these good results?\n- Wall-clock time is never mentioned — how much overhead does the refining process of LS-GFNs incur? How about in terms of sampled states (instead of training rounds)?\n- One baseline I wished were included: given a first trajectory, resample the last K steps and only keeping the best candidate. This is akin to beam search in LLMs or go-explore in RL. Other nice-to-have baselines include top-p and top-k sampling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JXChBRaaIB", "reviewer": "Reviewer_xEGb", "summary": "The paper proposes Local Search GFlowNet (LS-GFN) for training GFlowNets with local search to enhance the training effectiveness. The proposed algorithm explores the local neighborhood through destruction and reconstruction guided by backward and forward policies, respectively. Extensive experiments demonstrate significant performance improvements in several biochemical tasks.", "questions": "1. What about the sampling complexity of local search? Can you conduct experiments under the same sampling complexity or compare your algorithm with an upper bound sampling times?\n2. How to decide the local search interaction I? What about the results with different I?\n3. As PRT is adopted in LS-GFN training, is PER used for RL methods in te experiments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces an algorithm which combines inter-mode global exploration with intra-mode local exploration in GFlowNets training.\n2. The paper is well-written and easy to understand. The proposed algorithm and experimental setup are clearly described.", "weaknesses": "1. The paper does not declare the sampling complexity of the proposed method. The local search may require more sampling, which can lead to an unfair comparison. \n2. The limitations of the proposed algorithm and potential drawbacks are not discussed in detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DN3NBQldAd", "reviewer": "Reviewer_4fxw", "summary": "This work proposes an improvement to the GFlowNet training procedure that biases samples towards the high-reward terminal states. The idea is to train a back-tracking model that would go back from the completed flow trajectory in DAG, and then use the forward model to sample the removed part again.\nThe method boasts impressive performance on biological sequence design when combined with different objectives for training GFlowNets, and achieves the best performance with Trajectory Balance method.", "questions": "- How is the number of steps K picked for backtracking? If it's fixed, is there a way to pick it automatically?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The presentation is clear\n- The idea is simple and easy to implement\n- The evaluation is comprehensive and showcases the strength of the idea", "weaknesses": "- The proposed method adds some computational overhead", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)