Evolution guided generative flow networks
Authors: Zarif Ikram
Ling Pan
Dianbo Liu
Venue: Accepted by TMLR ()

Abstract: Generative Flow Networks (GFlowNets) are a family of probabilistic generative models recently invented that learn to sample compositional objects proportional to their rewards. One big challenge of GFlowNets is training them effectively when dealing with long time horizons and sparse rewards. To address this, we propose Evolution guided generative flow networks (EGFN), a simple but powerful augmentation to the GFlowNets training using Evolutionary algorithms (EA). Our method can work on top of any GFlowNets training objective, by training a set of agent parameters using EA, storing the resulting trajectories in the prioritized replay buffer, and training the GFlowNets agent using the stored trajectories. We present a thorough investigation over a wide range of toy and real-world benchmark tasks showing the effectiveness of our method in handling long trajectories and sparse rewards.  We release the code at http://github.com/zarifikram/egfn.

Published in Transactions on Machine Learning Research (06/2025)
Evolution guided generative flow networks
Zarif Ikram
National University of Singopore
Ling Pan
The Hong Kong University of Science and Technology
Dianbo Liu
National University of Singapore
Reviewed on OpenReview: https: // openreview. net/ forun? id= UgZIR67F5
Abstract
Generative Flow Networks (GFlowNets) are a family of probabilistic generative models recently invented that learn to sample compositional objects proportional to their rewards. One big challenge of GFlowNets is training them effectively when dealing with long time horizons and sparse rewards. To address this, we propose Evolution guided generative flow networks (EGFN), a simple and powerful augmentation to the GFlowNets training using Evolutionary algorithms (EA). Our method can work on top of any GFlowNets training objective, by training set of agent parameters using EA, storing the resulting trajectories in the prioritized replay buffer, and training the GelowNets agent using the stored trajectories. We present a thorough investigation over a wide range of toy and real-world benchmark tasks showing the effectiveness of our method in handling long trajectories and sparse rewards. We release the code at http: / / /ggtthth..b... com com/zari zari i kkram/effn
1 Introduction
"Do contradict myself? Very well then I contradict myself, (I am large, I contain multitudes.)"
Wall Whilman, Song of Myself (1855)
Generative Flow Networks (GFlowNett)) (Bengio al., 2021; 2023) are a family of probabilistic amortized samplers that learn to sample from a space proportionally to some reward function R(z), effectively sampling compositional objects over some probability distribution. As generative process, it composes objects by some sequence of actions, terminating by reaching a termination state.
GFlowNets have shown great potential for diverse challenging applications, such as molecule discovery (Jain et al., 2023a), biological sequence design (Jain et al., 2022), combinatorial optimization (Zhang et al., 2023), latent variable sampling (Liu al., 2023) and road generation (Ikram et al., 2023). The key advantage of GFlowNets over other methods such as reinforcement learning (RL) is that GFlowNets's key objective is not reward maximization, allowing them to sample diverse samples proportionally to the reward function. Although entropy- regulaieee RL also encourages randomness when taking actions, it is not general in when the underlying graph is not a tree (i.e., a state can have multiple parent states) (Zhao et al., 2019).
Despite the recent advancements, the real-world adaptation GFlowNets is still limited by a major problem: temporal credit assignment for long trajectories and sparse rewards. For example, real-world problems such as protein design often are often long-horizon problems, necessitating long trajectories for sampling. Since reward is given only when the agent reaches the terminal states, associating actions with rewards over

---

Published in Transactions on Machine Learning Research (06/2025)
Step three: Train-
Crossover Mutation
Agent
semple
New papulation
Filnesses
Training Objective
samgle
Evaluation
GFlowNet
Samples
Heplay Butter
Population
Slep Evolve
lwo: Sample
The proposed EGFN architecture. Step one provides high-quality trajectories the replay buffer by evolving the agents using the trajectory rewards as fitness. Step two gathers training trajectories from both online and offline trajectories. Step three trains the star agent using the training trajectorics.
Figure 1:
lengthy trajectory becomes challenging. Additionally, reward space is sparse in real-world tasks, making temporal credit assignment more difficult. Trajectory balance (TB) objective (Malkin et al., 2022) attempts tackle the problem by matching the flow across the entire trajectory, but in practice, it induces larger variance and is highly sensitive to sparse rewards (Madan al., 2023), making the training unstable.
Evolutionary algorithms (EA) (Bäck & Schwefel, 1993), class of optimization algorithms inspired by natural selection and evolution, can be a promising candidate for tackling the said challenges. Indeed, the shortcomings of GFlowNets are the advantages of EA, which makes it promising to consider incorporating EA into the learning paradigm of GFlowNets to leverage the best from both worlds. First, the selection operation in EA achieved by fitness evaluation throughout the entire trajectory, which makes them robust to long trajectorics and sparse rewards as they naturally bias towards regions with high expected returns. Secondly, mutation makes EA naturally exploratory, which is crucial for GFlowNets training and mode-finding they rely on diverse samples for better training (Pan et al., 2022). Third, FAAs natural selection biases towards parameters that generate high reward samples, which, coupled with replay buffer, provide sample redundancy, resulting in better gradient signal for stable GFlowNets training.
In this work, we introduce Evolution guided generative flow networks (EGFN), a novel training method for GFlowNets combining gradient- based and gradient-free approaches and benefit from the best of both worlds. Our proposed approach is a threc-step training process, as summarized in Figure First, using a fitness metric across sampled trajectories taken over population of GFlowNets agents, we perform selection, crossover, and mutation neural network parameters of GFlowNets agents to generate To reuse the population's experience, store the evaluated trajectories in the prioritized replay buffer (PRB). For the second step, we sample the stored trajectories from PRB and combine them with online samples from a different GFlowNets agent. Finally, using the gathered samples, we train a GFlowNett agent using gradient doscent over some objectives such as Flow matching (FM), Detailed balance (DB), and Trajectory balance (TB), where they optimize at the transition-level (FM, DB), and trajectory level (TB). The neward-maximiaz capability of EA enhances gradient signal through high reward training samples, ensuring stable GelowNets training extensive evaluation in experimental and wide range of real-world setlings, our method proves elfective in addressing weaknesses related to temporal credit assignment in sparse rewards and long trajectories, surpassing GelowNets baselines in terms of both the number of re-discovered modes and top-K rewards. The key contributions of the work are the following.
population.
in conditions with sparse rewards and long trajectories. Through
We effectively use EA to train GFlowNets in long trajectories and sparse rewards;
We understand several design decisions through thorough ablation experiments to help practitioners;
We perform extensive experiments to validate method's capability.
2

---

Published in Transactions on Machine Learning Research (06/2025)
2 Preliminaries
2.1 Generative Flow Networks (GFlowNets)
Gencrative Flow Networks (GFlowNets) are a family of gencrative models that samples compositional objects through a sequence actions. We define the flow nelwork as a single source initial state 80 with in-flow Z, and one sink for each terminal state z X' with out-flow R(x) > O. We denote the Markovian composition trajectory (so - 91 + + x) as T T, where T is the set of all trajectories. Thus, the problem directed acyclic graph (DAG), (S,E), where each node in S denotes state with an
is formulated initial state 80, and each edge in & denotes transition - 81+1 with a special terminal action indicating == x E X. We specifically consider DAGs that are not tree-structured, thus there exist different paths leading to the same state in the DAG, except for the root, which has no parent. Given a terminal state space X, GFlowNets aim to learn a stochastic policy π that can sample terminal states I E X proportionally to a non-negative reward function R(xr), i.c., л(x) x R(x). GFlowNets construct objects x E X by sampling constructive, irreversible actions a E A that transition St to 8141- An important advantage of GFlowNets object is proportional to given positive reward for that object and can train it in both online and offline settings, allowing us to train from replay buffers.
is that GFlowNets's probability of generating
The key objective of GFlowNets training is to train PF such that (()) x R(x), where,
1-|-1
(σx) = Σ II (r(st+1s;;))
(1)
FET WET
where, Pp is a parametric model representing the forward transition probability of St 8 with parameter 0. There are several widely used loss functions to optimize GFlowNets including FM, DB and TB.
Flow matching. Following Bengio et al. (2021), we define the state flow and edge flow functions F(s) ΣeEr (()) and F(s + 8') = Er=(...-e'... F()), respectively, where F(r) the trajectory flow is a nonnegative function F F T + R+ that probability measure of trajectory T E T is P(T) F(r)/ F(r)/ Erer F(r). Then,
the FM criterion matches the in fnow and the out-flow for all states E S, formally
Σ
F(d -6)=
Σ
(PPeeet(()
e'Ehhildss)
To achieve the criterion, using an estimated edge flow Fe : : Rt, we turn equation 2 loss function
LFM(s; log Σ EParents Fols' + +)
(3)
" (CChild(s) Fo(s 8")
Detailed balance. Following Bengio et al. (2023), we parameterize F(s), F(s + s'), and F(s' +) with (o(s,, (s (s']s,0), O), and (s(sss', 0), respectively, where Pr(s') (s 0) x F(s - 8') and Ps(s]s', 0) x F(s' + 8).
Then, the DB loss for all (s + s') in a sampled trajectory 7 is
(s'ss)))) (DB(s, = log (o(ss)
(4)
Trajectory balance. Malkin et al. (2022) extends the detail balance objective to the trajectory level, via a telescoping operation Eq. (4). Specifically, Zo is a learnable parameter that represents the total flow: Enex R(x) - Pr(s|so: so: 0), and the TB loss is defined as:
Zo II. 1---1 PF (185 1²
CTB(T:0) = log R(x)) 11-1 Pp(0s|8411
(5)
This can incur larger variance as demonstratod in Madan et al. (2023).
We train the GelowNets parameter by minimizing the loss L by performing stochastic gradient descent.
3

---

Published in Transactions on Machine Learning Research (06/2025)
2.2 Evolutionary algorithms (EA)
Evolutionary algorithms (EA) (Bäck, 2006; Spears et al., 1993) are class of optimization algorithms that generally rely on three key techniques: mutation, crossover, and selection as in biological evolution. The crossover operation is responsible for generating new samples based on exchange of information among a population of samples. The mutation operation alters the generated samples, usually with some probability Pmesaation Finally, the selection operation evaluates the fitness score of the population and is responsible for generating the next population. In this work, we apply EA in the context of the weights of the neural works,,,, often referred lo neuroevolution (Stanley & Miikkulainen, 2002b; Risi & Togelius, 2014; Floreano et al., 2008; Lüdders et al., 2017).
3 Evolution guided generative flow networks (EGFN)
Algorithm Evolution Guided GFlowNet Training
Input:
Pf: Forward flow of the star agent with weights 6*
popp: Population of k agents with randomly initiated weights
D Prioritized replay buffer
&: Number of episodes in an evaluation
e: percent of greedily selected elites
8: taline-to-oflline sample ratio
y: mutation strength for each episodes do for each PP E popp do L fitness, D = EVALUATE(PF, &, noise = None, D);
// store experience in replay buffer
Sort popF based on fitness a descending order Select the first ck PF from popf as elite
Select (1-e) PF from popş stochastically based fitness S while |S| < do L crossover between PF E elite and Pr E S and append to S
for each Pp S do L Apply mutation ~ (O 7) to Pry with probability Pmautation
Sample a minibatch of 87 ouline trajectories Tontine from PE and store them to D Sample a minibatch of (1 - ö)T offline trajectories Toffine from D Compute loss L using trajectory balance loss from Tontine U TTf
Update parameters 0* using stochastic descent on loss
EGEN is strategy to augment existing training methods of GFlowNets. The evolutionary part in EGFN (Step onc) samples discrete objects, o.g., molccular structure, using population of GFlowNets agents, evaluates the fitness of the agents based on the samples, and generates better samples by manipulating the weights of the agent population. We store the samples obtained from the population a PRB that the GelowNets sampler uses to, alongside on-policy samples, train its weights (step two & three). To differentiate the GFlowNets agent trained by gradient descent in step two and three from the agent population trained using FA in step one, we refer to the agent trained by gradient descent as the slar agent and GFloowNer agents trained using EA as EA GFlowNets agents. Algorithm 1 details the training loop, which can be summarized in the following three steps:
Step One Generate population of EA GFlowNets agents. Evaluate the fitness of the agents' weights by evaluating the samples gathered from the agents'. Apply the necessary selection, mutation, and crossover to the weights to generate the next population. Store the generated trajectories r to the PRB.
{(TI...., T8),, (e)...,7)2.....7777....
Step Two Gather online trajectories from star agent Pe and offline samples from PRB.
4

---

Published in Transactions on Machine Learning Research (06/2025)
Step Three Train Pe using (ri, .....::77 Tr) using gradient descent any GFlowNets loss function such as equations 3, or 5.
3.1
Step one:
Evolve
This step involves optimizing EA GFlowNets agent weights to produce trajectories that accelerate Pr training using the PRB. To this end, before the train begins, initialize pop, population of k EA GFlowNets agents with random weights. We optimize the population weights a standard EA process that contains selection, crossover, and mu- tation. Algorithm in the appendix C details the evaluation process.
Population
Sample agents proportionally to their fitness
Gather unselected agents
Gather Elite agents
Selection. The selection process begins with evaluation of the population by calculating each agent's s fitness scores. We define the fitness score of an agent by the mean reward of & trajectories {TI, ,e...., TE} sampled from the agent. Next, based on fitness scores, we transfer the lop e% elite agents' weights the next population, ummodified. No- tably, store the &y trajoctories sampled from this step to the PRB in this step. Optionally, we restore the star agent parameters to the agent population by replacing the parameters with the worst fitness periodically.
Crossever among selected agents
Pairwise crossover
Population after crossover
The crossover operation in EGFN. Here, we fill proportion of population through the crossover between agents selected proportionally to their fitness. We fill the rest with the crossover between the unselected agents and the elite agents.
Figure 2:
Crossover. The crossover step ensures weight mix- ing between agents' weights, ensuring stochasticity (Spears, 1995). Here, perform the crossover in two steps. First, we perform selection tournament process among the agents to get pop elite agents, sampling proportionally to their fitness value and performing crossover among them. Next, we perform a crossover between the unselected agents and elite. We combine the two sets of agents and pass them on to the mutation process. To perform the crossover, we simply swap randomly chosen the weights uniformly (see details in Algorithm 3).
The mutation process ensures natural exploratory policy in agents. We apply mutation by adding gaussian perturbation (0,,7) 7) to the agent weights. In this work, only apply mulation to the non-elite agents.
Mutation.
3.2 Step two: Sample
In this step, we gather trajectory samples (TI, T2...., ... T to train the star agent. We use both online trajectories sampled by the star agent and offline trajectories stored in the PRB. For online trajectories, construct trajectory 7 by applying PF to get states. It is noteworthy that there are many works (Rector-
I,, E X, where X' is the set of all terminal (Rector-Brooks Brooks et al., 2023; Kim et al., 2023; Pan et al., 2023a; 2022) that augment or perturb the online trajectories by applying stochastic exploration, temperature scaling, etc. In this work, we choose a simple on-policy sampling from Pt to get the online trajectories. For oflline samples, we simply use PRB to sample trajectories collected from step one proportionally to the terminal reward. For this work, we take a simple approach for PRB, uniformly sampling 50% trajectories from the 20 percentile and 50% trajectories from the rest.
3.3 Step three: Train
We train the star agent by calculating loss C using equation 3, 4, or 5 and minimizing the loss by applying stochastic gradient descent. to the parameter 0.
5

---

Published in Transactions on Machine Learning Research (06/2025)
20
x10
20
X10
20
11 error
0.01072
1.0
0.0001
0.3
Number of Modes
1000
2000
1000
2000
1000
2000
10
1000
2000
0
1000
20010
1000)
2000
Number Training Steps
GEN
PPO MARS
SAC
GFN+ Random Ensembles
EGFN
CAFN
IQL
Figure 3: Left: example hypergrid environment for dimension D horizon H 16. Here, the 2 4 yellow tiles refer to the high reward modes. Right: Experimental results comparison for the hypergrid task between EGFN, GFlowNets, RL, and MCMC baseline across increasing dimensions for 2500 training steps. Right top: the & error between the learned distribution density and the true target density. Right bottom: the number of discovered modes across training process. As dimension the grid increases, the trajectory length also increases. The proposed EGFN method achieves better performance than all baselines, with broader performance gap between EGFN and GFlowNets with increasing trajectory length.
4 Experiments
In this section, we validate EGFN for different synthetic and real-world tasks. Section 4.1 presents an investigation of EGFN's performance long trajectory and sparse rewards, generalizability across multiple GFlowNets objectives, and an ablation study on different components. Next, we present five real-world molecule generation experiments. In section 4.2 and 4.4, offer large-scale molecule experiments to confirm EGFN's ability to produce longer sequences. Additionally, we compare our method with experiments relevant to previous literature in section 4.3, D.2, and D.3. For all the following experiments, we use k = 5, E = 4, = 0.2, and = All baselines are equipped with replay bullers that have parameters similar to the EGFN in order to conduct fair comparisons. The exception to this PPO, where we double the sample size to guarantce fairness. All result figures report the mean and variance over three random scods. Appendix E reports additional experiment details.
4.1 Synthetic tasks
We first study the effectiveness of EGFN investigating the well-studied hypergrid task introduced by Bengio et al. (2021). The hypergrid is a D-dimensional environment of H horizons, with a H state-space, D + action-space, and 2 modes. The ith action in the action space corresponds to moving 1 unit the ith dimension, with the Dth action being termination action with which the agent completes the trajectory and gets reward specified by
D
D
R(x) = Ro + R I
Xd H --1
0.5 E (0.25, 0.5]
R₂
II
Xd
0.5
E (0.3, 0.4]
(6)
H
d=1
d=1
where Il is the indicator function and Ro, R1, and R2 are reward control parameters. In this empirical experiment, two questions interest us.
Does EGFN ougmentation provide improvement against the best GFlowNets baseline for longer Irajeclories and sparse rewards?
Is this method generally applicable to other baselines?

---

Published in Transactions on Machine Learning Research (06/2025)
x10 - low sparsity
x10- modest sparsity
x10-7 worst sparsity
error
6.0
5.5
Number of Modes
1000
2000
1000
2000
1000
2000
20
20
20
0
0
1000
2000
1000
2000
0
1000
2000
Number of Training Steps
GFN EGFN
MARS
GAFN
IQL
SAC
GFN + Random Ensembles
Figure 4: Experimental results comparison for the hypergrid task between EGFN, Gets (GFlowNets, GAFN), RL (PPO, SAC, IQL), and MCMC (MARS) baseline across increasing dimensions for 2500 training steps. Top: the & error between the learned distribution densily and the true targel density. Bollom: the number of discovered modes across the training process. Here, H 20, D As the Ro value decreases, the reward sparsity increases. The proposed EGFN method achieves better performance than all baselines, with broader performance gap between RGFN and GFlowNets with increasing sparsity.
Setup. We run all hypergrid experiments for D E (3,4,5), H = 20, and RS E {10-3, 10-1, 110-3,10-4,10-- To determine the best GFlowNets baseline, run three objectives E {FM, DB} (please see Figure 10 in appendix D.1) and decide to use DB with a PRB of size 1000. For a fair comparison, we use DB for implementing EGFN. We use Rg = 10-5 for the long trajectory experiment and D 5 for the reward sparsity experiment, keeping other variables fixed. Finally, we present an ablation study on different components used in our experiments for H = 20, D = 5, and Ro 10-5
For a complete picture, we compare our method with RL baselines such as PPO (Schulman et al., 2017), SAC (Haarnoja et al., 2018; Christodoulou, 2019), and IQL (Kostrikov et al., 2022) and MCMC bascline such MARS (Xie et al., 2020), and recent GFlowNet baseline such as GAFN. Besides, we also compare it against a simpler variation to our method that involves GFlowNets with its offline trajectories sampled by random policy ensembles. We provide an additional overview of the baselines in Appendix G.
Long time horizon result. this experiment, as D increases, |7| increases, showing the performance over increasing |r]. Figure demonstrates that EGFN outperforms GFlowNets baseline both in terms of mode finding efficiency and LI error. Notably, as |7| increases, the performance gap increases, confirming its efficacy in challenging environments. Unexpectedly, MARS prove to be very slow for these challenging environments. Besides, while RI. baseline such PPO competes with GFlowNets and EGFN in the beginning, it fails to discover all modes due to its mode maximization objective.
Reward sparsity result. Next, to understand the effect of sparse rewards, we compare our method against GFlowNets for Ro E (10-3, 10-1, 10-5). With a decreasing Ro, reward sparsity increases. Figure shows that EGFN outperforms GFlowNets. Similar to the previous experiment, we see an increasing performance gap as the reward sparsity increases. Similar to previous experiment, both RL and MCMC baselines are no match for such difficult environments.
A blation study result. To understand the individual effect of cach component of our method, we run the hypergrid experiment by comparing our method against the same without PRB and nutation. Figure 5 details the results of the experiment, underscoring the importance of the mutation operator in EGFN. It shows that PRB individually is not effective for improved results, but when is coupled with mutation, our method delivers better results. Appendix F presents a detailed ablation on different parts of EA used in EGFN, showing larger population (appendix F.1), lower elite population (appendix F.2), larger replay buffer
7

---

Published in Transactions on Machine Learning Research (06/2025)
x10 -8
2.0
30
Relative Mcan Error
1.8
Number of Modes
20
1.4
10
1.2
0
0
500
1000
1500
2000
2500
500
1000
1500
2000
2500
Number of Training Steps
Number of Training Steps
GFN
EGFN Mutation + PRB
EGFN
GFN+RB
EGFN Mutation PRB
EGFN Crossover
Figure 5: Experimental results for the hypergrid task on different components of EGEN, Top: the & error between the learned distribution density and the true target density. Bottom; the number of discovered modes across the training process. Here, H 20, D = 5, and we use DB objective.
(appendix F.3), greater mutation strength (appendix F.4), higher priority percentile (appendix F.5), and high split (appendix F.6) generally improves EGEN.
4.2
Antibody sequence optimization task
Numbet of Modes
500
Top-K Eg 1
Top-K inder +
250
GFlowNet EGFN
34.91
39.04
40.47
35.95
500
1000
1500
2000
2500
GAFN
17.46
38.08
Number of Training Steps
SAC
0.0
40.41
EGFN
GAFN
LED&GEN
SAC
Table 1: Experiment results for antibody sequence optimization
between EGFN, GFlowNets, GAFN, and SAC baseline task after Figure 6: Experimental results comparison for an- training for 2500 steps. EGFN achieves the lowest instability tibody sequence optimization task for 2500 training index while retaining the best diversity. The performances steps. EGFN achieves the best performance in terms measures are taken from 1000 samples after training and K = of mode discovery.
Setup. In this task, our goal is to generate an antibody heavy chain sequence of length 50 that, augmented with a pre-defined suffix sequence, optimizes the instability index (Guruprasad et al., 1990) of the antibody sequence. We represent antibody sequences as x = (æ1,..., za), where x; E 1,..., 21 refers to the 20 amino acid (AA) type, with the gap token ,.> to ensure variable length. We label the training sequences using BioPython (Cock et al., 2009) and transform the instability index to reward by performing the following transformation: R(x) inder-3 10 where index is the instability index æ. Finally, we define sequences with an instability index less than 35 as modes!
Results. For comparison, include GFlowNets, GAFN, and SAC. We train all the baselines for 2500 steps and sample 1000 sequences afterwards. To caleulate the diversity of the samples,
the edit distance
Based the literature, any index greater than unstable, but define modes with inder < 35 make the reward space more sparse.
8

---

Published in Transactions on Machine Learning Research (06/2025)
7.5
1.5
Number of Modes R >
Top- 100 Reward
Top-100 1000 Reward
Number of Modes R
100
0.5
50
0.0
20
10
Number of States Visited x10*)
GFN
BGFN
Figure 7: From left to right. Example binder produced in the Soluable Epexy Hydrolase (sEH) binder generation task. Here the structure corresponds to the molecule with SMILES representation [0P(()-]) (0) c1ccc2ccccc2c1. sEH binder generation experiment over 2.5 x 10 training steps. GFlowNets implementation uses FM objective. The number of modes with reward threshold of 7.5 and 8.0. The average reward across the top-100 and 1000 molecules. The proposed augmentation with EGFN achieves better results both in terms of mode discovery and average reward.
(Ea). Figure 4.2 reports the results from both training and post-training. During training, EGFN discovers a significant number of modes compared the other baselines. We can see that this translates to post-training sample quality, EGFN samples have lower mean instability index and higher diversity.
4.3 Soluable Epoxy Hydrolase (sEH) binder generation task
Setup. In this experiment, we are interested in generating molecules with desired chemical properties that are not too similar to one another. Here, we represent molecules states as graph structures and actions as a vocabulary of blocks specified by junction tree modeling (Bickerton et al., 2012; Shi et al., 2020). In the pharmaceutical industry, drug-likeliness (Bickerton et. al., 2012), synthesizability (Shi et al., 2020), and toxicity are crucial properties. Hence, we are interested in finding diverse candidate molecules for given criteria to increase chances for post-sclection. Here, the criteria is the molccule's binding energy to the 4JNC inhibitor of the soluble epoxide hydrolase (sEH) protein. To this end, train praxy reward function for predicting the negative binding energy that serves as the reward function. We perform the experiment following the experimental details and reward function specifications from Bengio et al. (2021). Since we are interested in both the diversity and efficacy drugs, we define a mode as a molecule with a reward greater than 7.5 and tanimoto similarity among previous modes less than 0.7. We use FM as the GFlowNets bascline and implement EGEN for the same objective.
Results. Since the state space is large, show the result of the number of modes 7.5, the number of modes > 8.0, the top-100, and the top 1000 over the first 2.5 x 10a states visited. Figure 7 confirms that EGFN outperforms GFlowNets bascline for mode discovery. Remarkably, EGFN discovers rare molccule with a very high reward (R > 8) that GFlowNets fails to discover. Besides, EGFN has a better top-100 and top-1000 reward performance than GFlowNets baseline, soliciting its mode diversity.
4.4 hu4D5 CDR H3 mutation generation task
Setup. To further show the robustness, here consider the task of generating CDR mutants on a hu4D5 antibody mutant dataset (Mason et al., 2021). After de-duplication and removal of multi-label samples, this dataset contains 9k binding and 25k non-binding hu4D5 CDR mutants to 10 mutations. To classify the generated samples, we train a binary classifier that achieves 85% accuracy on an IID validation set. For this task, we define samples, which are classified as binders with a high probability (> 96%) as modes. The reward transformation in this task R(x)) = pPoind 1.
Results. We compare EGFN against GFlowNets, GAFN, and SAC and train the baselines for 2500 steps. As shown in Figure 8, the modes discovered by EGFN during training exceed the other baselines.
9

---

Published in Transactions on Machine Learning Research (06/2025)
4.5 Result summary
In both the synthetic and real-world experiments, EGFN performs well for mode discovery using fewer training steps than GelowNets baseline. The per- formance gap increases with increasing trajectory length and reward sparsity. We also discover that the mutation operator is the most important factor for performance improvement.
200
175
150
Number of Modes
125
100
5 Discussion and Limitations
75
Why does EGPN work? To explore this, we com- pare the trajectories stored in the training step for GFlowNets and EGFN across training steps E (500, 1000, 1500, 2000, 2500} in the hypergrid task in Figure 9 for different levels of sparsity (Ro E {10 10. We that when there is low reward sparsity (left), the training trajectory length distri- bution of both methods is similar. However, when the reward sparsity increases (right), GFlowNets training trajectories center around a lower trajectory length than that of EGFN. However, for a reward- symmetrical environment like hypergrid, the sam- pled trajectory length must be uniformly distributed to truly capture the data distribution. EGFN can achieve this through population variation, diversify- ing the sampled trajectories the PRB.
25
500
1000
1500
2000
2500
Number of Training Steps
EGFN
GFN
GAPN
SAC
Figure 8: Experimental comparison of discovered modes across the training process for the hu4D5 CDR H3 mulation generation task between EGFN, GFlowNets, SAC, and GAFN baseline for 2500 training steps. Right top: & error between the learned distribution density and the true target density, EGFN augmentation achieves better mode discovery compared to other baselines.
trajectory
A potential limitation of this work that it is based on GFlowNets, which is an amortized sam- pler with the provable objective of fitting the reward distribution instead of RL's reward maximization. Thus, depending on the perspective, this method may seem slower than RL. Besides, because of the evolution step, the method takes longer than the vanilla GFlowNets. For example, we report the run- time analysis on the QM9 task (appendix D.3) in table 2, which performed with an Intel Xeon Pro- cessor (Skylake, IBRS) with 512 GB of RAM and single A100 NVIDIA GPU. We can see that the runtime has increased by 35%. However, we do note that there is no additional cost for increasing the population size, except for increased memory require- ment, since we perform the population sampling by utilizing threading.
0.02
Dembly
0.00
0.02
Dersitt
0.02
0.00
0.00
100
100
Trajectory length
500
1000
2500
2300
Figure Experimental results for the trajectory lengths of the trajectories stored in the training samples across different training steps for 10 (left) and 10-4 (right). Top: GFlowNet Bottom: EGFN
Another avenue where work falls short is that it does not propose better credit assignment solution but rather utilizes the quality-diversit samples from EA to address the challenge of training EGFN in long time horizons and sparse rewards. Figure 22 motivates this approach, showing that the population achieves high mean reward quicker than the star agent.
Finally, while we keep the total training samples in each step the same across the basclines for our experiments, do strictly enforce the number of reward calls to be same. However, they are also roughly the same, e.g., EGFN uses 36 calls while other baselines use 32, except for PPO which doubles the reward calls since
10

---

Published in Transactions on Machine Learning Research (06/2025)
Table Runtime analysis comparing EGFN and GFlowNets on the QM9 task.
Wall-clock time per step
K
number of eval samples
number of online samples
EGFN
3.033 + 0.004
10
1
24
GFN
2.243 + 0.016
0
0
32
we double the amount of on-policy samples due to not having off-policy samples. This said, we still see significant improvement of EGFN compared of GFlowNets when we plot the results with number trajectory evaluations on the X axis (Figure Appendix H).
6
Related work
6.1
Evolution in learning
There has been many attempts to augment learning, especially RL, with EA. Early works such Whiteson (2006) combine NEAT (Stanley & Mikkuuainen, 2002a) and Q Learning (Watkins & Dayan, 1992) by using evolutionary strategies to better tune the function approximators, uses EA for exploration in policy gradient, generating diverse samples using mutation. Fernando et al. (2017) use EA for allowing parameter reuse without ratast forgetting. Recently, many methods use EA to enhance deep RL architectures such as Proximal Policy Optimization (PPO) (Hämäläinen et al., 2020), Soft-Actor Critic (SAC) (Hou et al., 2020), and Policy Gradient (Khadka & Tumer, 2018). The key idca from these approaches is to use EA to overcome the temporal credit assigument and improve exploration by getting diverse samples (Tee et al., 2020), with some exceptions such as Gangwani & Peng (2018); Fujimoto et al. (2018); Pourchot & Sigaud (2019) where they utilize EA to tune the parameter of the actor itself.
a similar manner, Colas al. (2018)
6.2 GFlowNets
GFlowNets have recently been applied to various problems (Liu et al., 2023; Bengio et al., 2021). There have also been recent efforts extending GFlowNets to continuous (Lahlou et al., 2023) and stochastic worlds (Pan et al., 2023b), and also leveraging the power of pre-trained models (Pan et al., 2024). In GFlowNets training, exploration is an important concept for training convergence, which many works attempt in different ways. For example, Bengio et al. (2021) use e-greedy exploration strategy, Kim el al. (2023) learn the logits conditioned on different annealed temperatures, Pan et al. (2022) introduces angmented flows into the flow network represented by intrinsic rewards, etc. The temporal credit assignment for long trajectories and sparse reward is a more recently studied topic for GFlowNets. Recent works such as Malkin et al. (2022) attempt to tackle this problem by minimizing the loss over an entire trajectory as opposed state- wise FM proposed by Bengio et al. (2021), however, it may incur large variance as demonstrated in Madan et al. (2023).
7 Conclusion
In this work, we presented EGFN, a simple and effective EA based strategy for training GFlowNets, especially for credit assignment in long trajectories and sparse rewards. This strategy mixes the best of both worlds: EA 's nopulation-based approach biases towards regions with long-term returns, and GFlowNet's gradient-based objectives handle the matching of the reward distribution with the sample distribution by leveraging better gradient signals. Besides, EA promotes natural diversity of the explored region, removing the need to use any other exploration strategies for CFlowNets training. Furthermore, by incorporating PRB for offline samples, EA promotes redundancy of high region samples, stabilizing the GFlowNet training with better gradient signals. We validate our mothod on a wide range of challenging toy and real-worid benchmarks with exponentially large combinatorial search spaces, showing that our method outperforms the best GFlowNet baselines on long time horizon and sparse rewards.
11

---

Published in Transactions on Machine Learning Research (06/2025)
In this work, we implement standard evolutionary algorithm for EGFN. Incorporating complex sub-modules of EA for GFlowNets training such as Covariance Matrix Adaptation and Evolution Strategy (CMA-ES) such as the work in Pourchot & Sigand (2019) can be an exciting future work. Another future direction could be integrating the gradient signal from the GFlowNets objectives into the EA strategy, creating leedback loop. Besides, while we use a reward-maxiimzation formulation for the EA this work, there are works such as Parker-Holder et al. (2020) that directly improves diversity by formulation. We leave that for the future work.
Broader Impact Statement
The primary goal of this work is to advance the field of Machine Learning, The authors, thus, do not foresce any specific negative impacts of this work.
Acknowledgments
DL is funded by NUS-UToronto joint grant. Furthermore, ZI thankful his family. The authors are grateful to Emmanuel Bengio and Moksh Jain for their valuable feedback and suggestions in this work.
References
Thomas Bäck. Evolutionary computation: Toward new philosophy of machine intelligence. In IEEE Press Series on Computational Intelligence, 2006. URL https: //ap1. semanticscholar org/CorpusID: 27611267.
Thomas Bäck and Hans-Paul Schwefel. An overview of evolutionary algorithms for parameter optimization. Evolutionary computation, 1(1):1-22) 1993.
Luis A Barrera, Anastasia Vedenko, Jesse V Kurland, Julia M Rogers, Stephen S Gisselbrecht, Elizabeth J Rossin, Jaie Woodard, Luca Mariani, Kian Hong Kock, Sachi Inukai, et al. Survey of variation in human transcription factors reveals prevalent dna binding changes. Science, 351(6600):1450-1455 2016. 20
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow work based generative models for non-iterative diverse candidate generation. Advances in Newral Information Processing Systems, 34:27381-27394, 2021. 1, 6, 9, 11, 18, 19, 20
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet. foundations. Journal of Machine Learning Research, 24(210):1-55 2023. 1,
G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew T. Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90-98, 2012. 9
Petros Christodoulou. Soft actor-critic for discrete action settings. arXiv preprint arXiv:1910. 07207, 2019. 7
Peter JA Cock, Tiago Antao, Jeffrey T Chang, Brad A Chapman, Cymon J Cox, Andrew Dalke, Iddo Friedberg, Thomas Hamelryck, Frank KaufT, Bartek Wilezynski, et al. Biopython: freely available python tools for computational molecular biology and bioinformatics. Bioinformatics, 25(11):1422, 2009. 8
Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. Gep-pg: Decoupling exploration and exploitation in deep reinforcement learning algorithms. In International conference on machine learning, pp. 1039 1048.
PMLR, 2018. 11
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXio preprint xrXi::1701.08873 2017.
Dario Floreano, Peter Dürr, and Claudio Mattiussi. Neuroevolution: from architectures to learning. Evolu- lionary Intelligence, 1:47-62, 2008. URL https: /api. semanticscholats wttellaarrrrr//oaeo arggCorrpuII::2944444 294466334
12

---

Published in Transactions on Machine Learning Research (06/2025)
Nathan C Frey, Daniel Berenberg, Karina Zadorozhny, Joseph Kleinhenz, Julien Lafrance- Vanasse, Isidro Hotzel, Yan Wu, Stephen Ra, Richard Bonneau, Kyunghyun Cho, et al. Protein discovery with discrete walk-jump sampling. arXiv preprint TrXi:22306.1230, 2023. 20
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic
methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018. 11
Tanmay Gangwani and Jian Peng. Policy optimization by genetic distillation. In International Conference on Learning Representations, 2018. 11
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017. 20
Kunchur Guruprasad, BV Bhasker Reddy, and Madhusudan W Pandit. Correlation between stability of a protein and its dipeptide composition: novel approach for predicting in vivo stability of protein from its primary sequence. Protein Engineering, Design and Selection, 4(2): 161, 1990. 8
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861 1870. PMLR, 2018. 7
Perttu Hämäläinen, Amin Babadi, Xiaoxiao Ma, and Jaakko Lehtinen. Ppo-cma: Proximal policy optimization with covariance matrix adaptation. In 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1-6. IEEE, 2020. 11
Zhimin Hou, Kuangen Zhang, Yi Wan, Dongyu Li, Chenglong Fu, and Haoyong Yu. Off-policy maximum entropy reinforcement learning: Soft actor-critic with advantage weighted mixture policy (sac-awmp). arxiv preprint ar Xiv:2000 02829, 2020. 11
Zarif Ikram, Ling Pan, and Dianbo Liu. Probabilistic generative modeling for procedural roundabout generation for developing countrics. In NeurIPS 2023 Workshop Adaptive Experimental Design and 1, 17
Active Learning in the Real World, 2023. URL https://openrerierr wwtp:/eeeetooooddd/aaaaao.
Zarif Ikram, Dianbo Liu, and M Saifur Rahman. Gradient-guided discrete walk-jump sampling for biological sequence generation. Transactions Machine Learning Research, 2024. URL ctttps://openeeiiee
Forumiid=fFhoo4PPP 20
Moksh Jain, Emmanuel Bengio, Alex Harcia, Jarrid Rector- Brooks, Bonaventure FP Dossou, Chanakya Ajit Elbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference Machine Learning, pp. 9786 9801. PMLR, 2022.
Moksh Jain, Tristan Deleu, Jason Hartford, Cheng-Hao Liu, Alex Hernandez-Garde and Yoshua Bengio. Gftownets for ai-driven scientific discovery. Digital Discovery, 2(3):557-577, 2023a.
Moksh Jain, Sharath Chandra Raparthy, Alex Mernández-Garena, Jarrid Rector-Brooks, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. In International Conference on Machine Learning, pp. 14631 14653. PMLR, 2023b. 21
Shauharda Khadka and Kagan Tumer. Evolution-guided policy gradient in reinforcement learning. In Neural Information Processing Systems, 2018. URL bttpss//api... semaaticcchhtt orgg/orrpsID::300 53096951.
11
Minsu Kim, Joohwan Ko, Dinghuai Zhang, Ling Pan, Taeyoung Yun, Woo Chang Kim, Jinkyoo Park, and Yoshua Bengio. Learning scale logits for tenperatuur--onnditoonll gflownets. In NeurIPS 2023 Al for Science Workshop, 2023. 5, 11
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022. URL http://///ppooooooooooooorrrrtt
68n2s92JVVF. 7
13

---

Published in Transactions on Machine Learning Research (06/2025)
Salem Lahlou, Tristan Deleu, Pablo Lemos, Dinghuai Zhang, Alexandra Volokhova, Alex Heraandez-Garca,, Téna Néhale Exzine, Yoshua Bengio, and Nikolay Malkin. A theory of continuous generative flow networks. In International Conference on Machine Learning, pp. 18269 18300. PMLR, 2023. 11
Kyunghyun Lee, Byeong-Us Uk Lee, Ukcheol Shin, and In So Kweon. An efficient asynchronous method for integrating evolutionary and gradient-basee policy search. Advonces in Neurol Information Processing Systems, 33:10124 10135, 2020. 11
Dianbo Liu, Moksh Jain, Bonaventure FP Dossou, Qianli Shen, Salem Lahlou, Anirudh Goyal, Nikolay Malkin, Chris Chinenye Emezue, Dinghuai Zhang, Nadhir Hassen, et al. Gflowout: Dropout with generative flow networks. In International Conference on Machine Learning, pp.
PMLR, 2023. 1, 11
Benno Lüders, Mikkel Schläger, Aleksandra Korach, and Sebastian Risi. Continual and one-shot learning through neural networks with dynamic external memory. In Applpppiittions, 2017. URL attps://api. semanticscholar. wttg/CorpusDD:3413300 4
Kanika Madan, Jarrid Rector-Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, Andrei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gfowwets from partial episodes for improved convergence and stability. In International Conference on Machine Learning, pp. 23467-23488 PMLR,
2023. 2, 3, 11
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio, Trajectory balance: Improved credit assignment gflownets. Advances in Neural Information Processing Systems, 55:5955-5961 2022. 2,
3, 11
Derek M Mason, Simon Friedensohn, Cédric R Weber, Christian Jordi, Bastian Wagner, Simon M Meng, Roy A Ehling, Lucia. Bonati, Jan Dahinden, Pablo Gainza, et al. Optimization of therapeutic antibodies by predicting antigen specificity from antibody sequence via deep learning. Nature Biomedical Engineering, (66))600-612, 2021. 9
Ling Pan, Dinghuai Zhang, Aaron Courville, Longbo Huang, and Yoshua Bengio. Generative augmented flow networks. In The Eleventh International Conference on Learning Representations, 2022. 2, 5, 11, 18, 19
Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of gflownets with local credit PMLR,
and incomplete trajectories. In International Conference on Machine Learning, pp. 2023a. 5
Ling Pan, Dinghuai Zhang, Moksh Jain, Longbo Huang, and Yoshua Bengio. Stochastic generative flow networks. In Uncertainty in Artificial Intelligence, pp. 1628 1638. PMLR, 2023b. 11
Ling Pan, Moksh Jain, Kanika Madan, and Yoshua Bengio. Pre-training and fine-tuning generative flow networks. In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.cr/ htwp://wwww..eetttooridiidddddde httpe//oommidddhiiiiiippleee 11
Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems, 33: 88050-18068 2020. 12
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, bigh-performance deep learning library. Advances in neural information processing systems, 32, 2019. 16
Alois Pourchot and Olivier Sigaud. Cem-rl: Combining cvolutionary and gradient-based methods for policy scarch. In 7th International Conference on Learning Representations, ICLR 2019, 2019. 11, 12
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014. 17, 21
14

---

Published in Transactions on Machine Learning Research (06/2025)
Jarrid Rector-Brooks. Kanika Madan, Moksh Jain, Maksym Korablyov, Cheng- Hao Liu, Sarath Chandar, Nikolay Malkin, and Yoshua Bengio. Thompson sampling for improved exploration in gflownets. In JCML, 2023 Workshop on Structured Probabilistic Inference {18} Generative Modeling, 2023. 5
Sebastian Risi and Julian Togelius. Neuroevolution in games: State of the art and open challenges.
IEEE Transactions Computational Intelligence and AI in Games, 9:25-41, 2014. URL https:
11245845. 4
John Schulman, Filip Wolski, Prafulla Dhariwal, Alee Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint arXio:1707.00347, 2017. 7
Max W Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving gflownet training. arXiv preprint arXiv:2305.071 2023. 17,
Chence Shi, Minkai Xu, Hongyu Guo, Ming Zhang, and Jian Tang. A graph to graphs framework for retrosyuthesis prediction. In International conference on machine learning, pp. 8818 8827. PMLR, 2020. 9
William M Spears. Adapting crossover in evolutionary algorithms. Evolutionary programming, 1995. 5
William M. Spears, Kenneth A. De Jong, Thomas Bäck, David B. Fogel, and Hugo de Garis. An overview of evolutionary computation. In European Conference on Machine Learning, 1993. URL https://api. semanticscholar. org/ CorpusID: 175549,
Kenneth O Stanley and Risto Mikkulaannn. Evolving neural networks through augmenting topologies. Evolutionary computation, 10(2):99-122 2002a. 11
Kenneth O. Stanley and Risto Mikkulainen. Evolving neural networks through augmenting topologies. Evo- lutionary Computation, 10:99-127, 2002b. URL https::/apt semanti cscholar
semanticscholar org/CorpusID:49080 498161.
Teague Sterling and John J Irwin. Zinc 15-ligand discovery for everyone. Journal of chemical information and modeling, 35(11):2324-25) 2337, 2015. 19
Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-beuch: Benchmarks for data-driven offline model-based optimization. In International Conference on Machine Learning, pp. (165--21676. PMTRR 2022. 17
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279 292, 1992. 11
Shimon Whiteson. Evolutionary function approximation for reinforcement learning. Journal of Machine Learning Research, 7, 2006. 11
Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. Mars: Markov molecular sampling for multi-objective drug discovery. In International Conference on Learning Representations, 2020.
Dinghuai Zhang, Hanjun Dai, Nikolay Malkin, Aaron Courville, Yoshua Bengio, and Ling Pan. Let the flows tell: Solving graph combinatorial problems with gllownets. In Thirly-seventh Conference on Neural Information Processing Systems, 2023.
Dinghuai Zhang, Ling Pan, Ricky TQ Chen, Aaron Courville, and Yoshua Bengio. Distributional gflownets with quantile flows. Transactions on Machine Learning Research, 2024. 19
Shuo Zhang, Yang Liu, and Lei Xie. Molecular mechanics-driven graph neural network with multiplex graph for molecular structures. arXiv preprint arXiv:2011.07445 2020. 18, 21
Rui Zhao, Xudong Sun, and Volker Tresp. Maximum entropy-regularized multi-goal reinforcement learning. In International Conference on Machine Learning, pp. 7553-7562, PMLR, 2019. 1
15

---

Published in Transactions on Machine Learning Research (06/2025)
A
Codes
The hypergrid and sEH binder task is based on the code from https://gi httpa///ggtnbb..mm//bbbarrrrr com/zdhNarsil/ Aaszibuttional-GPloooleet. The QM9 and TFBind8 task is based on the code from https: / //ttt//bgbtt...// /tt.ttb.cccom///// maxwshen/gflowmet. All our implementation code uses the PyTorch library (Paszke et al., 2019). We used
MolView https://molviee: org/ visualize the molecule diagrams for our paper.
B Summary of notations
We summarize the notations used in paper in the table 3 below.
Table 3: Notations summary
Symbol
Description
state space
X
terminal state space
A
action space (s - s')
T
trajectory space initial state in S
state in terminal state in X trajectory in T forward flow backward flow population size replay buffer elite population ion ratio mutation strength
x
PF
PB
k
D
C
Fitness evaluation algorithm
Algorithm Evaluation of Forward Flows
Data: Forward flow Pr
Result: Updated replay buffer with trajectories and fitness of Pr Procedure EVALUATE(PP,
fitness <-
for iter to E do
Initialize start state
// Can also be parallelized
Initialize trajectory T to an empty list while not a terminal state do Sample action based on p=(s)Op,) transition(s,a) Append (s', a) to the T
8
x-+ s'
Compute R(s) using reward function using the last state in T fitness +- fitness R(s)
Append T to D
Return
fitness D
16

---

Published in Transactions on Machine Learning Research (06/2025)
Flow Matching
Trajectory Balance
Detailed Balance
2.00
2.0
1.0
1.9
1.6
11
1.8
1.25
1.00
500
1000
1500
2000
1000
2000
2500
500
1000
1500
2000
2500
Number of Nodtes
30
20
10-
1000
1500
Number of Training Steps
GFN
IGFN
Figure 10: Experimental results for the hypergrid task between EGFN and GFN across different GFlowNets objectives. Top: the error between the learned distribution density and the true target density. Bottom: the number of discovered modes across the training process. The proposed augmentation with EGFN achieves better results for three objectives.
D Additional Experiments
D.1 Generalizability experiment
To how well EGFN works with different GelowNets objectives, show the result of angmentation of our method over all three GFlowNets objectives in Figure 10. We see that EGFN offers steady improvement across all three GFlowNets objectives.
D.2 Transcription factor binder generation task
Setup this experiment, we generate nucleotide string of length 8. Although the string could be generated autoregressively, in this experiment setting, we use Prepend-Append Markov decision process (PA-MDP), used in similar settings by Shen et al. (2023); Ikram et al. (2023). Using this MDP, GFlowNets agent actions prepend or append to the nucleotide string. The reward is DNA binding affinity to human transcription factor provided by Trabucco et al. (2022). We attempt three GFlowNets objectives, finally deciding to use TB the best GFlowNets baseline and implement EGEN with the same using a reward exponent
Number Modes
100
0-
Relative Meea Frror
-0.2
-0.4
3.
750
500
750
1000
1250
150a
1750
2000
Results Figure 11 shows the result over 2000 train- ing steps, showing that GFlowNets outperforms GFlowNets baseline both in terms of the number of modes discovered and the mean relative error.
Training Steps
GFN
EGFN
Figure 11: Experiment results for transcription factor binder generation lask over 2000 training steps for 3 = 3. Top: number of discovered modes across the train- ing process. Bottom: the relative mean error. The pro- posed augmentation with EGFN achieves better results. GelowNets implementation uses TB objective.
D.3 Small molecule generation Task
Sctup this experiment, we generate a small molocule graph based on the QM9 data (Ranakrish- et al., 2014) that maximizes the energy gap between its HOMO and LUMO orbitals, thereby increasing
17

---

Published in Transactions on Machine Learning Research (06/2025)
Number of Modes
300
200
100
Delative MeanEror
0.05
0.00
0.05
250
500
750
1000
1250
1500
1750
2000
Training Steps
EGPN
From top to bottom: Example molecule produced in the qm9 task with SMILES representation C1CCCCC1NOCFO. Experiment results for small molecule generation task on the QM9 data over 2000 training steps for 1. The number of discovered modes the training process. The relative mean error. The proposed augmentation with EGFN achieves better results. GFlowNets implementation uses TB objective.
Figure 12:
its stability. The resulting molecule is 5-block molecule, having choice among 12 blocks for its two stems. For the reward function, we use pre-traiiin trained MXMNet proxy by Zhang et al. (2020) with reward exponent B = 1. Similar to D.2, we use TB for this experiment.
Results In Figure 12, we report the mode discovery and L1 error results over 2000 training steps. Similar previous experiments, EGFN maintains a steady improvement over the GFlowNets baseline for mode discovery while decrcasing the L1 error quicker.
E
Additional implementation details
E.1 Hypergrid task
In our experiments, R₁ and R2 stay at fixed value of 0.5 and In our experiments, Ro varies within {10-3, 10 53. A mode is the terminal state x for which R(x) = Rmaz- From the equation evident that there are 2 distinct reward regions, with cach rogion having M number of modes (M 1 for our experiments). Besides, H refers to the horizon of the environment, meaning each dimension of x can be equal to i E 1,, H 1}. For example, Figure 4 uses D = 5 and H = 20. Clearly, while increasing both D and H increases the complexity of the task, effecting the trajectory length |r| and the number of states |, only increasing D increases the number modes. To calculate the empirical probability density, we collect the past visited 200000 states and calculate the probability density.
is
Architecture We model the forward layer with a 3-layer MLP with 256 hidden dimensions, followed by a leaky ReLU. The forward layer takes the one-hot encoding of the states inputs and outputs action logits. For FM, we simply space and train the MLP as both the forward and backward flow. We use learning rate of 10-4 for FM and 10-3 for both TB and DB, including a learning rale of 0.1 for Zo. The replay bufler uses a maximum size of 1000, and we use a worst-reward first policy for replay replacement. For RL and MCMC baselines, we use the implementation provided by Bengio et al. (2021). For GAFN bascline, we use the official code provided by Pan et al. (2022), performing hyper-paramed (0.05, 0.03, 0.01) for experiments with D E (3, 4,5,, respectively.
the forward layer to model the edge flow. For TB and DB, double the action
tuning of intrinsic reward and using intrinsic reward
18

---

Published in Transactions on Machine Learning Research (06/2025)
E.2 Antibody sequence optimization task
In this task, we are interested in generating the mutation of the first 50 characters of a heavy-chain antibody sequence. To adapt to the variable length of the sequence, the total number of actions 21 ?. The heavy and light chain suffixes appended to the generated 21- 21-length sequences are SVSS and SAA.AAAAAACTTTATTTTTTENINEEEEIEC SASSST/TTTTTTTSSSSSAAAAAVVIOAOTT
gap token of '-, making
respectively. We collect the chain pair from the observed antibody space (OAS) database. Finally, we use reward policy to reduce the instability index of the mutated sequences. The main goal of this task is to assess the GFlowNet's ability to perform longer sequences with long trajectories. To the best of our knowledge, no GFlowNets literature has experimented with tasks with such long trajectory length, mainly because GFlowNets struggle with longer trajectories.
Architecture This task's architecture closely resembles the hypergrid setting. Careful architecture choice may improve the observed results of the task, but this outside the scope of this work.
E.3 sEH binder generation Task
action at
agent
state St
next state St+1
Figure 13: Illustratoon of GelowNets policy for sEH binder generation task. Figure adopted from Pan et al. (2022)
For this task, the number of actions is within 100 2000, depending on the state, making | 10 We allow the agent to choose from a library of 72 blocks. Similar to Bengio et al. (2021), include the different symmetry groups of given block, making the action count 105 per stem. We also allow the agent to select up to 8 blocks, choosing them as suggested by Sterling & Trwin (2015) from the ZINC dataset (Sterling & Irwin, 2015). Following Zhang et al. (2024), we use Tanimoto similarity, defined by the ratio between the intersection and the union of two molecules based on their SMILES representation. To maintain diversity, we define a mode to be a terminal state for which the normalized negative binding energy the 4JNC inhibitor of the soluble epoxide hydrolase (sEH) protein is more than 7.5 and the tanimoto similarity of other discovered modes is less than 0.7. Note that this objective is more limiting than simply counting the number of different Muriko scaffolds that reach the reward threshold like Bengio et al. (2021). Since are focusing on both molecule separation and optimization, our approach is more applicable for de novo molecule design, while the scaffold-based metric is suitable for lead optimization.
2.A still need stop token the GelowNets agent.
19

---

Published in Transactions on Machine Learning Research (06/2025)
Architecture Following Bengio et al. (2021), et al., 2017) that receives the atom graph to calculate the proxy reward of the molecules. Similarly, we use another MPNN that receives the block graph for flow estimation. The block graph is a tree of learned node embeddings that represent the blocks and edge embeddings that represent the bonds. To represent the flow, we pass the stems through a 10-layer graph convolution followed by GRU to calculate their embedding and pass the embedding through 3-layer MLP to get a 105-dimension logit. Similarly, action, we pass the global mean pooling to the 3-layer MLP. The MLPs use 256 hidden dimensions, followed by lea We use learning rate of 0.0005 and minibatch size of 4. For EGFN, sample probability of 0.2. Besides, we use reward exponent B 10 and a normalizing constant of 8.
message passing neural network (MPNN) (Gilmer
represent the stop
offline
Figure 14: Molecule blocks for the sEH protein task. Figure adapted from Bengio et al. (2021)
E.4 hu4D5 CDR H3 mutation generation task
Following Frey et al. (2023); Ikram et al. (2024), we consider the task of generating CDR3 10-length mutations on a hu4D5 antibody mulant dataset. The dataset contains both non-binder and binder sequences. We train a discriminator model on the binding likelihood of a sequence and then compare the starting points for making sequences that improve both the binding likelihood and the diversity shown by the pairwise edit distance of the new sequences.
The architecture of different baselines follows the baselines of the previously discussed tasks. We use a 35-layer Bytenet architecture with a hidden layer of 128 for the discriminator model. This is followed by a 3-layer ID-CNN and a 3-layer MLP, with leakyReLU activations added between the layers. We train the model on the hu4D5 antibody mulant dataset, which achieves 85% accuracy on the test set.
Archtecture
E.5 TFBind8 task
For this task, the goal to generate an 8-length DNA sequence that maximizes the binding activity score with a particular transcription factor SIX6REFR1 (Barrera al., 2016). We use a precalculated oracle for the proxy reward calculation. Using PA-MDP, we prepend or append neucleotide in each step. Note that this formulation reduces the trajectory length significantly despite our effort to showcase better performance in long trajectories, but we use it following previous works.
A Architecture Following Shen et al. (2023), the GFlowNets architecture uses a 2-layer MLP with 128- dimension hidden layer parameterizing SSR (S, + R+). For each training step, we train on both online and offline trajectorics for throe steps, using minibatch of 32. Besides, we use a lcarning rate of 10-4 for policy and 0.01 for Zg. Finally, we use a reward exponent do not any exploration for EGFN).
and an exploration probability of 0.01 (we
20

---

Published in Transactions on Machine Learning Research (06/2025)
E.6 QM9 task
The goal here is to generale diverse molecules based the QM9 data (Ramakrishnan et al., 2014) that maximize the HOMO-LUMO, To that end, we use the reward praxy that Jain et al. (2023b) provides based on Zhang ct al. (2020). Similar to the sEH task, we generate molecules with atoms and bonds. The blocks used here are the following: C, O, N, C-F, C=O, C#C, clccocc1, C1Ceccc1, C1CCNC1, cCC.
Architecture Using a PA-MDP, we use a 2-layer MLP with 1024 hidden dimensions for flow estimation. The reward proxy is MXMNet proxy trained on the QM9 data. We use reward exponent 3 = The learning rate and training style follow the ones used for the TFBind8 task, with the exception of exploration probability (0.1 here) and hidden dimension (1024 here).
We detail the summary of the training hyperparameters in table 4.
Table 4: Summary of the hyperparameters for all experiments
Antigrial/Attbood/CDDDRRR sEH Small Molecules
TFBindS
QM9
Learning Rate
10-4 (FM), 10-3
5 x 10-4 N/A 10
10-4
10-4
Zg Learning Rate
0.1
0.01
0.01
B MDP
3
Emumerate
Sequence Insert
PA-MDP
PA-MDP
Explocation (none for EGFN) Replay Buffer Training
0.01
50%
(20% for egfu)
50%
50%
MLP layers
3
2
MLP hiddenn dimensions
256
256
128
1024
F
Additional ablation experiments
F.1
Number of population
Flow Matching
Trajectory Balance
Detailed Balance
6.4
11 ereor
6.0
5.5
6.0
5.0
1000
1500
2000
2500
1000
1500
2000
2500
1000
1500
2000
2500
Numbe of Wotes
7.5
5.0
10-
20-
10-
0.0
500
1000
1500
2000
2500
500
1000
1500
2000
2500
500
1000
1500
2000
2500
Number of Training Steps
Figure 15: Experimental results for the hypergrid task for EGFN among different values k across the three training objectives. Top: the & error between the learned distribution density and the true target density. Bottom: the number of discovered modes across the training process. Here, H = 20, D = 5. The results show that increasing population size leads to diminishing returns.
To investigate the effect of population size, we vary the l: E {5, 10, 15), while keeping c = 0.2, D D= 5, H. 20, R2 = 1 5 We plot the results of the experiment in Figure 15. shows that increasing beyond 10 leads to diminishing returns, motivating our choice of 10 for all the experiments. For DB, however, increasing k leads to considerable improvement. Indeed, this is useful because increased population size leads more evaluation round required. While these evaluation round can be parallelized with threads we do in our work, massive population size requirement is difficult to satisfy.
Takeaway: FM and DB benefit from higher sample diversity provided by larger population.
21

---

Published in Transactions on Machine Learning Research (06/2025)
F.2
Elite population
1e-7
Flow Matching
Trajectory Balance
Detalled Balance
6.4
11 Leeror
5.5
6.0
5.0
500
1000
1500
2000
2500
500
1000
1500
2000
2500
500
1000
1500
2000
2500
Number of Modiss
30
10
20
20
10
500
1000
1500
2000
2500
500
1000
1500
2000
2500
0
500
1000
1500
2000
2500
Number of Training Steps
cps
ops
eps
Figure Experimental results for the hypergrid task EGFN among different values of across the three training objectives. Top: the & error between the learned distribution density and the true target density. Bottom: the number of discovered modes across the training process. Here, = 20, = 5. We observe that lower leads better results for FM and TB.
Following ablation on k we next perform ablation on the elite population ratio, e E {0.2, 0.4.0. For this experiment, we use the same hypergrid settings for == 10. We plot the results of the experiment in Figure 16. It shows that low improves mode discovery, especially for FM and TB objectives. This result is reasonable: apply mutation and crossover only to the non-elite population, so having low number of elite population means we have better chance at exploring using the non-elite population's mutation and crossover.
Takeaway: FM and TB benefit from lower mode-seeking behavior through lower number of elite population.
F.3
Replay buffer size
Buffer Size
1000
Buffer Size 5000
Buffer Size 10000
ererr
6.0
6.0
6.0
5.5
5.5
2000
2000
2000
Number of Modes
30
30
20
10
0-
0-
0
2000
2000
Number of Training Steps
Figure 17: Experimental results for the hypergrid task between GFlowNets EGFN across different values of D). Top: the & error between the learned distribution density and the true target density. Bottom: the number of discovered modes across the training process. Here, we use DB for H = 20, D The results show that while increasing replay buffer size improves the robustness of the result, it has little effect otherwise.
To understand the effect of replay buffer size, we run GFlowNets baseline with PRB and EGFN on a 20x20x20<20x200 environment. with Ro 10-5 for replay buffer size (1000, 5000, 10000). We present.
22

---

Published in Transactions on Machine Learning Research (06/2025)
the findings in Figure 17. From the figure, that increasing buffer size generally has little effect for GFlowNets, but it improves EGFN's robust.ness a little.
Takeaway:
Higher replay buffer capacity preserves more diverse samples, resulting in more consistent EGFN result.
F.4
Mutation strength
1e-7
Flow Matching
1e-7
Trajectory Balance
1e-7
Detailed Belance
Number crrer
6.0
6.0
5.5
5.5
5.0
5.0
500
1000
1500
2000
2500
0
500
1000
1500
2000
2500
500
1000
1500
2000
2500
30-
20
500
1000
1500
2000
500
1000
1500
2000
2500
500
1500
2000
2500
Number of Training Steps
Figure 18: Experimental results for the hypergrid task for EGFN among different values of across the three training objectives. Top: the l error between the learned distribution density and the true target density. Bottom: the number of discovered modes better results for FM and TB, while higher y leads to better results for DB.
the training process. Here, H 20, 5. We observe that lower 7 leads to
We now turn our attention the mutation. To observe the effect of the mutation strength Y we run the hypergrid experiment for the three training objectives using EGFN for 7 E (1,5, 10}. The hypergrid before. We plot the mode discovery and l error between
configurations follow the the same configurations the learned distribution density and the true target density over 2500 training steps in Figure 18. While the results indicate that having higher leads to better result for DB, the improvement is not extraordinary. Besides, in our work, we experience training instability for higher 7. Thus, we restrict to be throughout. in our work.
Takeaway: Higher mutation strength generally leads to diminishing returns.
F.5 Priority percentile
PRB an important component of our EGFN setup. Therefore, an important question is how the definition of priority samples stored in the replay buffer affects the results. To answer this question, we the hypergrid experiment in the hardest setting while changing the priority percentile (50, 70, 90} controlling the sampling split, i.e,, while sampling offline trajectories from the PRB, 50% of them come from the priority samples and the rest come from the non-priority samples. Figure 19 shows the results. It indicates that having high-reward priority samples stored the replay buffer benefits the training terms of mode discovery, but there is little improvement in distribution fitting.
Takeaway: Higher priority percentile improves mode-discovery of TB.
23

---

Published in Transactions on Machine Learning Research (06/2025)
Trajectory Balance
6.00
eeto
5.50
500
1000
1500
2000
2500
of
15
0
500
1000
1500
2000
2500
Training Steps
percentile 90%
percentile 70%
percentile 50%
Figure 19: Experimental results for the hypergrid task for EGFN among different percentiles of priority samples for the TB objective. Throughout the experiment, we keep a 50-50 split between the priority and non-priority samples. Top: the & between the learned distribution density and the true target density. Bottom: the number of discovered modes across the training process. Here, = 20, 5. We observe that higher percentile priority samples lead better results for TB.
F.6
Priority split
Trajectory Balance
6.0
5.5
5.0
500
1000
1500
2000
2500
Number of Modes
10
1500
Number Training Stops
prortty
priority
prioriy sample
Figure 20: Experimental results for the hypergrid task for EGFN among different splits of priority sampling for the TB objective. Throughout the experiment, we used 90 percentile samples as priority samples. Top: the & error between the learned distribution density and the true target density. Bottom: the number of discovered modes the training process. Here, H 20, D = 5. We observe that having high priority-non-poe pooiriii split leads to better results for TB.
Finally, decision just as important as the priority percentile is the sampling ratio between the priority and non-priority offline trajectories. This time, we keep the priority percentile fixed at 90%, while changing
24

---

Published in Transactions on Machine Learning Research (06/2025)
the priority to non-priority ratio splits between (20, 50, 80). We run experiment similar to the previous experiments with the aforementioned setting and report the results in Figure 20. The results clearly demonstrate that, despite their collection from a small subset of states, sampling more from the priority samples is beneficial for both mode discovery and distribution fitting.
Takeaway: Higher priority-non-prioril split improves mode-discovery and distribution-fitiin of TB.
G Baselines
MArkov moleculaR Sampling method (MARS) is multi-objective molecular design method that define a Markov chain over the explicit molecular graph space and design a kernel to navigate high probable candidates with acceptance-rejection sampling.
MARS.
GAFN. Generative Augmented Flow Networks (GAFN) is a GFlowNets variant that aims to address the exploration challenge of GelowNets by enabling intermediate rewards GFlowNets and thus intrinsic rewards.
SAC. Soft Actor-Critic (SAC) is off-policy RL algorithm that maximizes trade-off between expected reward and entropy, encouraging diverse and stable exploration.
PPO. Proximal Policy Optimization (PPO) is a policy-gradient method that simplifies training by con- straining policy updates, ensuring stable and reliable improvement during optimization.
IQL. Implicit Q-Learning (TQL) an offline reinforcement learning approach that effectively extracts value-based policies by decoupling value estimation from policy improvement, achieving stable learning from static datasets.
H Additional analysis
Long time horzion results with reward evalutations. To understand the sample complexity of EGFN in comparison to GFlowNets, plot the results for long time horizon task against the number of R(x) calls for the default hyperparameters in this paper. We see that EGFN captures the R(x) distribution better than GFlowNets with comparable trajectory evaluations.
20
x10
20
x10
20
1.25
11 error
0.0002
6.0
1.00
5.5
0.0001
0.75
Number of Modes
0
20000
40000
60000
20000
40000
60000
20000
40000
60000
10
20
0
20000 10000
60000
0
20000 40000 60000
20000 40000
60000
Number of R(x) calls
GFN
EGFN
Figure 21: Experimental results comparison for the hypergrid task between EGFN and GFlowNets across increasing dimensions for 2500 training steps. 'hee results shows that for similar amount of evaluation calls, EGEN performs better than GelowNets with increasing difficulty.
25

---

Published in Transactions on Machine Learning Research (06/2025)
Population performances against the star agent. To understand the contribution of the population during the training of the star agent, plot the performances of the population and the star agent of EGFN while training in an increasingly complex environment. For fair comparison, we choose mean fitness for the population (K= 4) and top 10% reward for the star agent as a performance metric, as is trained on a GFlowNets loss. The results in Figure show that the population are more resislant to the increasing difficulty, which ultimately drives the PRB samples, improving GFlowNets training.
20
20
20
3.0
2.5
2.5
2.5
2.0
2.0
2.0
[(
1.5
1.5
1.5
1.0
1.0
1.0
0.5
0.5
0.5
0
1000
2000
0
1000
2000
1000
2000
Number Training Steps
Star agent (top 10% reward)
Population (Average roward)
Figure 22: Comparing the star agent and population on increasing dimensions for 2500 training steps. For star agent, report the top 10% reward, and for population, report the mean reward.
Additional algorithms
1.1 Crossover
Algorithm Crossover step
Data: Agent weights Opr, Opez Result: Crossover between agent weights
Procedure CROSSOVER (0pp, Oppz)
for weights W1, in Opr, Opr, do
N BONNUM_RRM (ON_RDW(w1)
num mutations U(N)
for count
num malations do
index ~ U(N)
r() < 0.5 then
Wi [inder] w2 [index]
else
W2a [inder] = w/ [index]]
26


Reviews:
[{"review_id": "n4NoBjkWkC", "reviewer": "Reviewer_mpvD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The proposed approach combines both GFlowNets and AE into a single training framework to get the best of both worlds.\n2. Results suggest the proposed approach is superior to the evaluated baselines (results are reported for both real and synthetic data).\n\n**Weaknesses**\n1. The paper is somewhat difficult to follow. Since I am not a GFlowNets expert, a more comprehensive and motivational background section would greatly help in understanding the core ideas.\n2. While the authors report results across multiple datasets, the paper lacks a thorough ablation or analytical component. For instance, if my understanding is correct, a central contribution of the work is the integration of Evolutionary Algorithms (EA). In that case, it would be helpful to analyze the impact of various EA components:\n* How does the population size influence model performance?\n* What are the specific fitness, mutation, and crossover functions used, and how do they affect the outcomes?\n* What is the effect of selecting the different $\\epsilon$% elite agents?\n* What proportion of samples are drawn from $P^*_F$, and what is their influence on model performance?\n\nCurrently, Algorithm 1 offers the most clarity in explaining the method. I encourage the authors to similarly clarify these aspects in the main text.\n\n3. The abstract states: “One big challenge of GFlowNets is training them effectively when dealing with long time horizons and sparse rewards.” However, it is unclear how the proposed method addresses this challenge. A more explicit explanation would strengthen the contribution.\n4. Can the proposed EA-based method be extended to train models with other reward-driven objectives, such as PPO?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Evolution-guided Generative Flow Networks (EGFN), an augmentation framework that enhances GFlowNets with Evolutionary Algorithms (EA). The approach leverages gradient-free EA to select a set of trajectories from multiple agents, which are then used to train a central ‘star-agent’ through gradient-based optimization. The authors validate the effectiveness of their method on both synthetic and real-world datasets, showing that it outperforms baseline models, as the cost of increased inference time.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My primary concern with this submission is the connection between the proposed method and the experiments presented. I believe the authors should clarify this relationship more clearly and include additional supporting experiments to better emphasize their contribution.\nPlease refer to the questions listed above under the weaknesses section.", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xYKBHACqMU", "reviewer": "Reviewer_9dFT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper conducts a wide range of experiments, and shows strong results compared to quite a few baselines including SAC, original GFlowNets and more recent variants. The paper also has quite extensive ablation study, including effects of prioritized replay, mutation, crossover, and using random ensembles instead of evolution guided population for off-policy sampling.\n\nWeaknesses\n- The evaluation protocol is not that clear to me. For example, in Fig 3 and 4, what samples are used to compute the L1 error and the number of modes discovered? Do you use samples from the star agent only or do you also use samples from the replay buffer / EA population? If the latter, the comparison might be unfair.\n- Regarding the reward sparsity experiment, I think a more natural way to make the reward more sparse is to make the region of R1 and R2 smaller, i.e., making the modes less accessible by a random agent.\n- Presentation could be improved.\n    - Sec 3.1 could provide more details of crossover, i.e., how is crossover implemented between two agents? Is it simply swapping all the weights uniformly at random?\n    - In Fig 3, the colors are not easily distinguishable. GFN and IQL seem to have the same color. MARS and SAC also have quite similar colors. In Fig 5, GFN and EGFN have the same color.\n    - It would be better to put Eq 6 in the main text rather than appendix.\n    - It looks like there is a typo in the ablation study setup (Sec 4.1, Setup). Should H=16 be H=20 instead?\n    - The $R_0$ in Fig 9 caption is different from what is stated in the main text (first line below Table 2).\n- In Fig 9, why can the trajectory length be less than 0?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper aims to improve the effectiveness of GFlowNets in long horizon and sparse reward settings. The key contribution is to use a separate population of GFlowNet agents to provide off-policy samples in addition to the on-policy samples. This population is refined by evolutionary algorithms (including selection, crossover, mutation). Experiments on synthetic and real-world benchmarks show that the proposed method can discover more modes with fewer training steps and can better match the target density.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Clarify what samples are used to compute the L1 error and the number of modes discovered\n- Preferably increase the reward sparsity by shrinking the rewarding region\n- Improve presentation, fix typos and inconsistencies", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sEb8DyFL6H", "reviewer": "Reviewer_VDKA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is a well-documented exploration on evolutionary methods for GFlowNets. I find all of the claims to be backed up and the experimental evidence suitable and interesting enough for the TMLR audience.  On the novelty, it is a standard evolutionary method, but I am not aware of other works investigating it in this way. And, the extensive experimental analysis and ablations throughout the appendix will be useful to practitioners.\n\nOne question I have: is it fair to compare the number of training steps between all of the methods in the experiments? My minor concern is that the methods use significantly different information for each update, e.g., spanning GFN, EGFN, PPO, SAC, and others, and it seems like some of them may be more computationally expensive than others.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper trains investigates training a population of GFlowNets with the evolutionary algorithm in Alg 1, which maintains a population of networks and selects the elite ones to mutate, interspersed with sampling trajectories and evaluating the performance. This is experimentally compared on a grid MDP (Fig 3/4/5), antibody sequence optimization (Fig 6/Table1), sEH binder generation (Fig 7), and hu4D5 CDR H3 mutation generation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)