Evolution guided generative flow networks

## Abstract

Abstract Generative Flow Networks ( GFlowNets )are a family of probabilistic generative models that learn to sample compositional objects proportional to their rewards. One big challenge of GFlowNets is training them effectively when dealing with long time horizons and sparse rewards. To address this, we propose Evolution guided generative flow networks ( EGFN ), a simple but powerful augmentation to the GFlowNets training using Evolutionary algorithms ( EA ). Our method can work on top of any GFlowNets training objective, by training a set of agent parameters using EA , storing the resulting trajectories in the prioritized replay buffer, and training the GFlowNets agent using the stored trajectories. We present a thorough investigation over a wide range of toy and real-world benchmark tasks showing the effectiveness of our method in handling long trajectories and sparse rewards.

## 1 Introduction

Generative Flow Networks (GFlowNets)(Bengio et al., , ) are a family of probabilistic amortized samplers that learns to sample from a space proportionally to some reward function $R(x)$, effectively sampling compositional objects over some probability distribution. As a generative process, it composes objects by some sequence of actions, terminating by reaching a termination state.

GFlowNets have shown great potential for diverse challenging applications, such as molecule discovery (Jain et al., ), biological sequence design (Jain et al., ), combinatorial optimization (Zhang et al., ), latent variable sampling (Liu et al., ) and road generation (Ikram et al., ). The key advantage of GFlowNets over other methods such as reinforcement learning (RL)is that GFlowNets’s key objective is not reward maximization, allowing them to sample diverse samples from different peaks of high rewards. Although entropy-regularized RL also encourages randomness when taking actions, it is not general in when the underlying graph is not a tree (i.e., a state can have multiple parent states) (Zhao et al., ).

Despite the recent advancements, the real-world adaptation of GFlowNets is still limited by a major problem: temporal credit assignment for long trajectories and sparse rewards. For example, real-world problems such as protein design often are often long-horizon problems, necessitating long trajectories for sampling. Since reward is given only when the agent reaches the terminal states, associating intermediate actions with rewards over a lengthy trajectory becomes challenging. Additionally, reward space is sparse in real-world tasks, making temporal credit assignment more difficult. Trajectory balance (TB)objective (Malkin et al., ) attempts to tackle the problem by matching the flow across the entire trajectory, but in practice, it induces larger variance and is highly sensitive to sparse rewards (Madan et al., ), making the training unstable.

Evolutionary algorithms (EA)(Bäck & Schwefel, ), a class of optimization algorithms inspired by natural selection and evolution, can be a promising candidate for tackling the said challenges. Indeed, the shortcomings of GFlowNets are the advantages of EA, which makes it promising to consider incorporating EA into the learning paradigm of GFlowNets to leverage the best from both worlds. First, the selection operation in EA is achieved by fitness evaluation throughout the entire trajectory, which makes them robust to long trajectories and sparse rewards as they naturally bias towards regions with high expected returns. Secondly, mutation makes EA naturally exploratory, which is crucial for GFlowNets training and mode-finding as they rely on diverse samples for better training (Pan et al., ). Third, EA’s natural selection biases towards parameters that generate high reward samples, which, coupled with a replay buffer, can provide sample redundancy, resulting in a better gradient signal for stable GFlowNets training.

Figure: Figure 1: The proposed EGFN architecture. Step one provides high-quality trajectories to the replay buffer. Step two gathers training trajectories from both online and offline trajectories. Step three trains the star agent using the training trajectories.
Refer to caption: /html/2402.02186/assets/imgs/egfn_architecture.png

In this work, we introduce Evolution guided generative flow networks (EGFN), a novel training method for GFlowNets combining gradient-based and gradient-free approaches and benefit from the best of both worlds. Our proposed approach is a three-step training process. First, using a fitness metric across sampled trajectories taken over a population of GFlowNets agents, we perform selection, crossover, and mutation on neural network parameters of GFlowNets agents to generate a new population. To reuse the population’s experience, we store the evaluated trajectories in the prioritized replay buffer (PRB). For the second step, we sample the stored trajectories from a prioritized replay buffer (PRB)and combine them with online samples from a different GFlowNets agent. Finally, using the gathered samples, we train a GFlowNets agent using gradient descent over some objectives such as Flow matching (FM), TB, and Detailed balance (DB). The reward-maximizing capability of EA enhances gradient signal through high reward training samples, ensuring stable GFlowNets training even in conditions with sparse rewards and long trajectories. Through extensive evaluation in experimental and a wide range of real-world settings, our method proves effective in addressing weaknesses related to temporal credit assignment in sparse rewards and long trajectories, surpassing GFlowNets baselines in terms of both the number of re-discovered modes and top-K rewards.

## 2 Preliminaries

### 2.1 Generative Flow Networks (GFlowNets)

Generative Flow Networks (GFlowNets) are a family of generative models that samples compositional objects through a sequence of actions. Given a terminal state space $\mathcal{X}$, they aim to learn a stochastic policy $\pi$ that can sample terminal states $x\in\mathcal{X}$ proportionally to a non-negative reward function $R(x)$, i.e., $\pi(x)\propto R(x)$. GFlowNets construct objects $x\in\mathcal{X}$ by sampling constructive, irreversible actions $a\in\mathcal{A}$ that transition $s_{t}$ to $s_{t+1}$. We denote the Markovian composition trajectory $(s_{0}\rightarrow s_{1}\xrightarrow{}\dots\rightarrow x)$ as $\tau\in\mathcal{T}$, where $\mathcal{T}$ is the set of all trajectories. Thus, the problem is formulated as a directed acyclic graph (DAG), $(\mathcal{S},\mathcal{E})$, where each node in $\mathcal{S}$ denotes a state with an initial state $s_{0}$, and each edge in $\mathcal{E}$ denotes a transition $s_{t}\rightarrow s_{t+1}$ with a special terminal action indicating $s=x\in\mathcal{X}$. There exist different paths leading to the same state in the DAG. An important advantage of GFlowNets is that GFN can sample proportionally to different peaks of reward and we can train it in both online and offline settings, allowing us to train from replay buffers.

The key objective of GFlowNets training is to train $P_{F}$ such that $\pi(x)\propto R(x)$, where,

$$ $\pi(x)=\sum_{\tau\in\mathcal{T}:x\in\tau}\prod_{t=0}^{|\tau-2|}{P_{F}(s_{t+1}|s_{t};\theta)}$ (1) $$

where, $P_{F}$ is a parametric model reprenting the forward transition probability of $s_{t}$ to $s_{t+1}$ with parameter $\theta$. There are several widely used loss functions to optimize GFlowNets including FM, DB and TB.

Flow matching
Following Bengio et al. (), we define the state flow and edge flow functions $F(s)=\sum_{s\in\tau}{F(\tau)}$ and $F(s\rightarrow s^{\prime})=\sum_{\tau=(\dots s\rightarrow s^{\prime}...)}{F(\tau)}$, respectively. Then, the FM criterion matches the in-flow and the out-flow for all states $s\in\mathcal{S}$, formally –

$$ $\sum_{s:(s\rightarrow s^{\prime})\in\mathcal{A}}{F(s\rightarrow s^{\prime})}=\sum_{s^{\prime\prime}:(s^{\prime}\rightarrow s^{\prime\prime})\in\mathcal{A}}{F(s^{\prime}\rightarrow s^{\prime\prime})}$ (2) $$

where $F(s^{\prime}\rightarrow s^{\prime\prime})=R(s)$ if $s\in\mathcal{X}$. Using $P_{F}(s^{\prime}|s,\theta)=F(s^{\prime}\rightarrow s^{\prime\prime})$, we turn equation [2](#S2.E2) to a loss function –

$$ $\mathcal{L}_{\text{FM}}(s^{\prime},\theta)=\left[\text{log}\frac{\sum_{(s\rightarrow s^{\prime})\in\mathcal{A}}{P_{F}(s^{\prime}|s,\theta)}}{\sum_{(s^{\prime}\rightarrow s^{\prime\prime})\in\mathcal{A}}{P_{F}(s^{\prime}|s^{\prime\prime},\theta)}}\right]^{2}$ (3) $$

Detailed balance Following Bengio et al. (), we parameterize $F(s)$, $F(s\rightarrow s^{\prime})$, and $F(s^{\prime}\rightarrow s)$ with $F_{\theta}(s)$, $P_{F}(s^{\prime}|s,\theta)$, and $P_{B}(s|s^{\prime},\theta)$, respectively. Then, the DB loss for a sampled trajectory $\tau\in\mathcal{T}$ is –

$$ $\mathcal{L}_{\text{DB}}(s,s^{\prime},\theta)=\left[\text{log}\frac{F_{\theta}(s)P_{F}(s^{\prime}|s,\theta)}{F_{\theta}(s^{\prime})P_{B}(s|s^{\prime},\theta)}\right]^{2}$ (4) $$

for all $(s\rightarrow s^{\prime})\in\tau$.

Trajectory balance Malkin et al. () extends the detail balance objective to the trajectory level, via a telescoping operation of Eq. ([4](#S2.E4)). Specifically, $Z_{\theta}$ is a learnable parameter that represents the total flow: $\sum_{x\in\mathcal{X}}{R(x)}=\sum_{s:s_{0}\rightarrow s\in\tau\forall\tau\in\mathcal{T}}{P_{F}(s|s_{0};\theta)}$, and the TB loss is defined as:

$$ $\mathcal{L}_{\text{TB}}(\tau,\theta)=\left[\text{log}\frac{Z_{\theta}\prod_{t=0}^{|\tau|-1}{P_{F}(s_{t+1}|s_{t},\theta)}}{R(x)\prod_{t=0}^{|\tau|-1}{P_{B}(s_{t}|s_{t+1},\theta)}}\right]^{2}.$ (5) $$

This can incur larger variance as demonstrated in Madan et al. ().

We train the GFlowNets parameter $\theta$ by minimizing the loss $\mathcal{L}$ by performing stochastic gradient descent.

Figure: Algorithm 1 Evolution Guided GFlowNet Training

### 2.2 Evolutionary algorithms ( EA )

Evolutionary algorithms (EA)(Bäck, ; Spears et al., ) are a class of combinatorial optimization algorithms that generally rely on three key techniques: mutation, crossover, and selection as in biological evolution. The crossover operation is responsible for generating new samples based on exchange of segments among a population of samples. The mutation operation alters the generated samples, usually with some probability $p_{mutation}$. Finally, the selection operation evaluates the fitness score of the population and is responsible for generating the next population. In this work, we apply EA in the context of the weights of the neural networks, often referred to as neuroevolution (Stanley & Miikkulainen, ; Risi & Togelius, ; Floreano et al., ; Lüders et al., ).

## 3 Evolution guided generative flow networks ( EGFN )

EGFN is an augmentation of existing training methods of GFlowNets. The evolutionary part in EGFN (Step one) samples discrete objects, e.g., a molecular structure, using a population of GFlowNets agents, evaluates the fitness of the agents based on the samples, and generates better samples by manipulating the weights of the agent population. We store the samples obtained from the population in a PRB that the GFlowNets sampler uses to, alongside on-policy samples, train its weights (Step two & three). To differentiate the GFlowNets agent trained by gradient descent in step two and three from the agent population trained by EA in step one, we refer to the agent trained by gradient descent as the star agent and GFlowNets agents trained by EA as EA GFlowNets agents. The training loop can be summarized in the following three steps:

Generate a population of EA GFlowNets agents. Evaluate the fitness of the agents’ weights by evaluating the samples gathered from the agents’. Apply the necessary selection, mutation, and crossover to the weights to generate the next population. Store the generated trajectories $\{(\tau_{1},\dots,\tau_{\mathcal{E}})_{1},\tau_{1},\dots,\tau_{\mathcal{E}})_{2},\dots,(\tau_{1},\dots,\tau_{\mathcal{E}})_{k}\}$ to the PRB.

Gather online trajectories from star agent $P_{F}^{*}$ and offline samples from PRB.

Train $P_{F}^{*}$ using $\{\tau_{1},\tau_{2},\dots,\tau_{T}\}$ using gradient descent on any GFlowNets loss function such as equations [3](#S2.E3), [4](#S2.E4), or [5](#S2.E5).

### 3.1 Step one: Evolve

This step involves optimizing EA GFlowNets agent weights to produce trajectories that accelerate $P_{F}^{*}$ training using the PRB. To this end, before the train begins, we initialize $pop$, a population of $k$ EA GFlowNets agents with random weights. We optimize the population weights in a standard EA process that contains selection, crossover, and mutation. Algorithm [2](#alg2) in the appendix [C](#A3) details the evaluation process.

Selection
The selection process begins with an evaluation of the population by calculating each agent’s fitness scores. We define the fitness score of an agent by the mean reward of $\mathcal{E}$ trajectories $\{\tau_{1},\tau_{2},\dots,\tau_{\mathcal{E}}\}$ sampled from the agent. Next, based on fitness scores, we transfer the top $\epsilon$% $elite$ agents’ weights to the next population, unmodified. Notably, we store the $k\mathcal{E}$ trajectories sampled from this step to the PRB in this step.

Crossover
The crossover step ensures weight mixing between agents’, ensuring stochasticity. Here, we perform the crossover in two steps. First, we perform a selection tournament process among the agents to get $pop$ - $elite$ agents, sampling proportionally to their fitness value and performing crossover among them. Next, we perform a crossover between the unselected agents and $elite$. We combine the two sets of agents and pass them on to the mutation process.

Figure: Figure 2: The crossover operation in EGFN. Here, we fill a proportion of population through the crossover between agents selected proportionally to their fitness. We fill the rest with the crossover between the unselected agents and the elite agents.
Refer to caption: /html/2402.02186/assets/imgs/crossover.png

Mutation
The mutation process ensures natural exploration among agents. We apply mutation by adding a gaussian perturbation $\mathcal{N}(0,\gamma)$ to the agent weights. In this work, we only apply mutation to the non-$elite$ agents.

### 3.2 Step two: Sample

In this step, we gather trajectory samples $\{\tau_{1},\tau_{2},\dots,\tau_{T}\}$ to train the star agent. We use both online trajectories sampled by the star agent and offline trajectories stored in the PRB. For online trajectories, we construct a trajectory $\tau$ by applying $P_{F}^{*}$ to get $s_{0}\rightarrow s_{1}\rightarrow\dots\rightarrow x,x\in\mathcal{X}$, where $\mathcal{X}$ is the set of all terminal states. It is noteworthy that there are many works (Rector-Brooks et al., ; Kim et al., ; Pan et al., , ) that augment or perturb the online trajectories by applying stochastic exploration, temperature scaling, etc. In this work, we choose a simple on-policy sampling from $P_{F}^{*}$ to get the online trajectories. For offline samples, we simply use PRB to sample trajectories collected from step one proportionally to the terminal reward. For this work, we take a simple approach for PRB, uniformly sampling 50% trajectories from the 20 percentile and 50% trajectories from the rest.

### 3.3 Step three: Train

We train the star agent by calculating loss $\mathcal{L}$ using equation [3](#S2.E3), [4](#S2.E4), or [5](#S2.E5) and minimizing the loss by applying stochastic gradient descent to the parameter $\theta$.

## 4 Experiments

In this section, we validate EGFN for different synthetic and real-world tasks. [4.1](#S4.SS1) presents an investigation of EGFN’s performance in long trajectory and sparse rewards, generalizability across multiple GFlowNets objectives, and an ablation study on different components. Next, we present three real-world molecule generation experiments in [4.2](#S4.SS2), [4.3](#S4.SS3), and [4.4](#S4.SS4). Finally, we present an experiment summary. For all the following experiments, we use $k=10$, $\mathcal{E}=4$, $\epsilon=0.2$, and $\gamma=1$. For fair comparison, we equip all baselines with a replay buffer of comparable settings to the EGFN. All result figures report the mean and variance over three random seeds.

### 4.1 Synthetic tasks

Figure: Figure 3: Left: An example hypergrid environment for $\text{dimension }D=2,\text{horizon }H=16$. Here, the $2^{2}=4$ yellow tiles refer to the high reward modes. Right: Experimental results comparison for the hypergrid task between EGFN, GFlowNets, RL, and MCMC baseline across increasing dimensions for 2500 training steps. Right top: the $\ell_{1}$ error between the learned distribution density and the true target density. Right bottom: the number of discovered modes across the training process. As the dimension of the grid increases, the trajectory length also increases. The proposed EGFN method achieves better performance than all baselines, with broader performance gap between EGFN and GFlowNets with increasing trajectory length.
Refer to caption: /html/2402.02186/assets/imgs/hypergrid_sample.png

We first study the effectiveness of EGFN investigating the well-studied hypergrid task introduced by Bengio et al. (). The hypergrid is a $D$-dimensional environment of $H$ horizons, with a $H^{D}$ state-space, $D+1$ action-space, and $2^{D}$ modes. The $i$th action in the action space corresponds to moving 1 step in the $i$th dimension, with the $D$th action being a termination action with which the agent completes the trajectory and gets a reward specified by equation [6](#A4.E6).

In this empirical experiment, two questions interest us.

- •
Does EGFN augmentation provide improvement against the best GFlowNets baseline for longer trajectories and sparse rewards?
- •
Is this method generally applicable to other baselines?

Setup
We run all hypergrid experiments for $D\in\{3,4,5\}$, $H=20$, and $R_{0}\in\{10^{-3},10^{-4},10^{-5}\}$. To determine the best GFlowNets baseline, we run three objectives $\in\{\text{\lx@glossaries@gls@link{acronym}{fm}{\leavevmode FM}},\text{\lx@glossaries@gls@link{acronym}{tb}{\leavevmode TB}},\text{\lx@glossaries@gls@link{acronym}{db}{\leavevmode DB}}\}$ (please see figure [7](#S4.F7) below) and decide to use DB with a PRB of size 1000. For a fair comparison, we use DB for implementing EGFN. We use $R_{0}=10^{-5}$ for the long trajectory experiment and $D=5$ for the reward sparsity experiment, keeping other variables fixed. Finally, we present an ablation study on different components used in our experiments for $H=16,D=5,\text{and }R_{0}=10^{-5}$.

For a complete picture, we compare our method with RL baselines such as PPO (Schulman et al., ) and SAC (Haarnoja et al., ; Christodoulou, ) and MCMC baseline such as MARS (Xie et al., ).

Long time horizon result In this experiment, as $D$ increases, $|\tau|$ increases, showing the performance over increasing $|\tau|$. Figure [3](#S4.F3) demonstrates that EGFN outperforms GFlowNets baseline both in terms of mode finding efficiency and L1 error. Notably, as $|\tau|$ increases, the performance gap increases, confirming its efficacy in challenging environments. Unexpectedly, MARS prove to be very slow for these challenging environments. Besides, while RL baseline such PPO competes with GFlowNets and EGFN in the beginning, it fails to discover all modes due to its mode maximization objective.

Figure: Figure 4: Experimental results for the hypergrid task on different components of EGFN. Top: the $\ell_{1}$ error between the learned distribution density and the true target density. Bottom: the number of discovered modes across the training process. Here, $H=20,D=5$, and we use DB objective. We observe that PRB itself does not improve the results, but when it is coupled with mutation, the results improve.
Refer to caption: /html/2402.02186/assets/imgs/ablation_exp_db.png

Reward sparsity result Next, to understand the effect of sparse rewards, we compare our method against GFlowNets. With a decreasing $R_{0}$, reward sparsity increases. Figure [5](#S4.F5) shows that EGFN outperforms traditional GFlowNets. Similar to the previous experiment, we see an increasing performance gap as the reward sparsity increases. Similar to previous experiment, both RL and MCMC baselines are no match for such difficult environments.

Different training objectives To see how well EGFN works with different GFlowNets objectives, we show the result of augmentation of our method over all three GFlowNets objectives in figure [7](#S4.F7). We see that EGFN offers a steady improvement across all three GFlowNets objectives.

Figure: Figure 5: Experimental results comparison for the hypergrid task between EGFN, GFlowNets, RL, and MCMC baseline across increasing dimensions for 2500 training steps. Top: the $\ell_{1}$ error between the learned distribution density and the true target density. Bottom: the number of discovered modes across the training process. Here, $H=20,D=5$. As the $R_{0}$ value decreases, the reward sparsity increases. The proposed EGFN method achieves better performance than all baselines, with broader performance gap between EGFN and GFlowNets with increasing sparsity.
Refer to caption: /html/2402.02186/assets/imgs/sparsity_exp.png

Ablation study result To understand the individual effect of each component of our method, we run the hypergrid experiment by comparing our method against the same without PRB and mutation. Figure [4](#S4.F4) details the results of the experiment, underscoring the importance of the mutation operator in EGFN. It shows that PRB individually is not effective for improved results, but when it is coupled with mutation, our method delivers better results.

### 4.2 Soluable Epoxy Hydrolase ( sEH )binder generation task ( | 𝒳 | ≈ 10 16 ) 𝒳 superscript 10 16 \left(|\mathcal{X}|\approx 10^{16}\right)

Setup
In this experiment, we are interested in generating molecules with desired chemical properties that are not too similar to one another. Here, we represent molecules states as graph structures and actions as a vocabulary of blocks specified by junction tree modeling (Bickerton et al., ; Shi et al., ). In the pharmaceutical industry, drug-likeliness (Bickerton et al., ), synthesizability (Shi et al., ), and toxicity are crucial properties. Hence, we are interested in finding diverse candidate molecules for a given criteria to increase chances for post-selection. Here, the criteria is the molecule’s binding energy to the 4JNC inhibitor of the soluble epoxide hydrolase (sEH) protein. To this end, we train a proxy reward function for predicting the negative binding energy that serves as the reward function. We perform the experiment following the experimental details and reward function specifications from Bengio et al. (); Zhang et al. (). Since we are interested in both the diversity and efficacy of drugs, we define a mode as a molecule with a reward greater than 7.5 and a Tanimoto similarity among previous modes less than 0.7. We use FM as the GFlowNets baseline and implement EGFN for the same objective.

Results Since the state space is large, we show the result of the number of modes with reward threshold of 7.5 and 8.0, the top-100, and the top-1000 over the first $2.5\times 10^{4}$ states visited. Figure [10](#S4.F10) confirms that EGFN outperforms GFlowNets baseline for mode discovery. Remarkably, EGFN discovers rare molecules with very high reward ($R>8$) that GFlowNets fails to discover. Besides, EGFN has a better top-100 and top-1000 reward performance than GFlowNets baseline, soliciting its mode diversity, which is essential in a molecular discovery setting.

### 4.3 Transcription factor binder generation task ( | 𝒳 | ≈ 70000 ) 𝒳 70000 \left(|\mathcal{X}|\approx 70000\right)

Setup In this experiment, we generate a nucleotide sequence as a string of length 8. Although the string could be generated autoregressively, in this experiment setting, we use a Prepend-Append Markov decision process (PA-MDP), used in similar settings by Shen et al. (); Ikram et al. (). Using this MDP, GFlowNets agent actions prepend or append to the nucleotide string. The reward is a DNA binding affinity to a human transcription factor provided by Trabucco et al. (). We attempt three GFlowNets objectives, finally deciding to use TB as the best GFlowNets baseline and implement EGFN with the same using a reward exponent $\beta=3$.

Figure: Figure 6: Experiment results for transcription factor binder generation task over 2000 training steps for $\beta=3$. Top: the number of discovered modes across the training process. Bottom: the relative mean error. The proposed augmentation with EGFN achieves better results. GFlowNets implementation uses TB objective.
Refer to caption: /html/2402.02186/assets/imgs/tfbind_exp.png

Results Figure [6](#S4.F6) shows the result over 2000 training steps, showing that GFlowNets outperforms GFlowNets baseline both in terms of the number of modes discovered and the mean relative error.

Figure: Figure 7: Experimental results for the hypergrid task between EGFN and GFN across different GFlowNets objectives. Top: the $\ell_{1}$ error between the learned distribution density and the true target density. Bottom: the number of discovered modes across the training process. The proposed augmentation with EGFN achieves better results for all three objectives.
Refer to caption: /html/2402.02186/assets/imgs/generalizability_exp.png

Figure: Figure 8: From top to bottom: Example molecule produced in the qm9 task with SMILES representation C1CCCCC1NOCFO. Experiment results for small molecule generation task on the QM9 data over 2000 training steps for $\beta=1$. The number of discovered modes across the training process. The relative mean error. The proposed augmentation with EGFN achieves better results. GFlowNets implementation uses TB objective.
Refer to caption: /html/2402.02186/assets/imgs/qm9_sample.png

### 4.4 Small molecule generation Task ( | 𝒳 | ≈ 60000 ) 𝒳 60000 \left(|\mathcal{X}|\approx 60000\right)

Setup In this experiment, we generate a small molecule graph based on the QM9 data (Ramakrishnan et al., ) that maximizes the energy gap between its HOMO and LUMO orbitals, thereby increasing its stability. The resulting molecule is a 5-block molecule, having a choice among 12 blocks for its two stems. For the reward function, we use a pre-trained MXMNet proxy by Zhang et al. () with a reward exponent $\beta=1$. Similar to [4.3](#S4.SS3), we use TB for this experiment.

Results
In figure [8](#S4.F8), we report the mode discovery and L1 error results over 2000 training steps. Similar to previous experiments, EGFN maintains a steady improvement over the GFlowNets baseline for mode discovery while decreasing the L1 error quicker.

### 4.5 Result summary

In both the synthetic and real-world experiments, EGFN performs well for mode discovery using fewer training steps than GFlowNets baseline. The performance gap increases with increasing trajectory length and reward sparsity. We also discover that the mutation operator is the most important factor for performance improvement.

Figure: Figure 9: Experimental results for the trajectory lengths of the trajectories stored in the training samples across different training steps for $R_{0}=10^{-2}$ (left) and $10^{-4}$ (right). Top: GFlowNets Bottom: EGFN
Refer to caption: /html/2402.02186/assets/imgs/combined_trajectory.png

Figure: Figure 10: From left to right. (Example binder produced in the Soluable Epoxy Hydrolase (sEH)binder generation task. Here the structure corresponds to the molecule with SMILES representation O=P([O-])(O)c1ccc2ccccc2c1. sEH binder generation experiment over $2.5\times 10^{4}$ training steps. GFlowNets implementation uses FM objective. The number of modes with a reward threshold of 7.5 and 8.0. The average reward across the top-100 and 1000 molecules. The proposed augmentation with EGFN achieves better results both in terms of mode discovery and average reward.
Refer to caption: /html/2402.02186/assets/imgs/seh_sample.png

## 5 Discussion

Why does EGFN work? To explore this, we compare the trajectories stored in the training step for GFlowNets and EGFN across training steps $\in\{500,1000,1500,2000,2500\}$ in the hypergrid task in figure [9](#S4.F9) for different levels of sparsity ($R_{0}\in\{10^{-2},10^{-5}\}$). We see that when reward sparsity is low (left), training trajectory length distribution of both methods is similar. However, when the reward sparsity increases (right), GFlowNets training trajectories center around a lower trajectory length than that of EGFN. However, for a reward-symmetrical environment like hypergrid, the trajectory length’s distribution must tend towards the mean to truly capture the data distribution. EGFN can achieve this by its reward-maximizing evolutionary process which supplies diverse modes to the PRB, ensuring a better training for the star agent.

## 6 Related work

### 6.1 Evolution in learning

There has been many attempts to augment learning, especially RL, with EA. Early works such Whiteson () combine NEAT (Stanley & Miikkulainen, ) and Q Learning (Watkins & Dayan, ) by using evolutionary strategies to better tune the function approximators. In a similar manner, Colas et al. () uses EA for exploration in policy gradient, generating diverse samples using mutation. Fernando et al. () use EA for allowing parameter reuse without catastrophic forgetting. Recently, many methods use EA to enhance deep RL architectures such as Proximal Policy Optimization (PPO)(Hämäläinen et al., ), Soft-Actor Critic (SAC)(Hou et al., ), and Policy Gradient (Khadka & Tumer, ). The key idea from these approaches is to use EA to overcome the temporal credit assignment and improve exploration by getting diverse samples (Lee et al., ), with some exceptions such as Gangwani & Peng (); Fujimoto et al. (); Pourchot & Sigaud () where they utilize EA to tune the parameter of the actor itself.

### 6.2 GFlowNets

GFlowNets have recently been applied to various problems (Liu et al., ; Bengio et al., ). There have also been recent efforts in extending GFlowNets to continuous (Lahlou et al., ) and stochastic worlds (Pan et al., ), and also leveraging the power of pre-trained models (Pan et al., ). In GFlowNets training, exploration is an important concept for training convergence, which many works attempt in different ways. For example, Bengio et al. () use $\epsilon$-greedy exploration strategy, Kim et al. () learn the logits conditioned on different annealed temperatures, Pan et al. () introduces augmented flows into the flow network represented by intrinsic rewards, etc. The temporal credit assignment for long trajectories and sparse reward is a more recently studied topic for GFlowNets. Recent works such as Malkin et al. () attempt to tackle this problem by minimizing the loss over an entire trajectory as opposed to state-wise FM proposed by Bengio et al. (), however, it may incur large variance as demonstrated in Madan et al. ().

## 7 Conclusion

In this work, we presented EGFN, a simple but effective augmentation EA strategy for training GFlowNets, especially for credit assignment in long trajectories and sparse rewards. This strategy mixes the best of both worlds: EA’s population-based approach biases towards regions with long-term returns, and GFlowNet’s gradient-based objectives handle the matching of the reward distribution with the sample distribution by leveraging better gradient signals. Besides, EA promotes natural diversity of the explored region, removing the need to use any other exploration strategies for GFlowNets training. Furthermore, by incorporating PRB for offline samples, EA promotes redundancy of high region samples, stabilizing the GFlowNet training with better gradient signals. We validate our method on a wide range of challenging toy and real-world benchmarks with exponentially large combinatorial search spaces, showing that our method outperforms the best GFlowNet baselines on long time horizon and sparse rewards.

In this work, we implement a standard evolutionary algorithm for EGFN. Incorporating more complex sub-modules of EA for GFlowNets training such as Covariance Matrix Adaptation and Evolution Strategy (CMA-ES) such as the work in Pourchot & Sigaud () can be an exciting future work. Another future direction could be integrating the gradient signal from the GFlowNets objectives into the EA strategy, creating a feedback loop. Besides, while we use a reward-maximization formulation for the EA in this work, there are works such as Parker-Holder et al. () that directly improves diversity by formulation. We leave that for the future work.

## Acknowledgements

The authors are grateful to Emmanuel Bengio and Moksh Jain for their valuable feedback and suggestions in this work.

## Impact statement

The primary goal of this work is to advance the field of Machine Learning. The authors, thus, do not foresee any specfic negative impacts of this work.

## Appendix A Reproducibility

Our code is available at [https://anonymous.4open.science/r/E-GFN/](https://anonymous.4open.science/r/E-GFN/). The hypergrid and sEH binder task is based on the code from [https://github.com/zdhNarsil/Distributional-GFlowNets](https://github.com/zdhNarsil/Distributional-GFlowNets). The QM9 and TFBind8 task is based on the code from [https://github.com/maxwshen/gflownet](https://github.com/maxwshen/gflownet). All our implementation code uses the PyTorch library (Paszke et al., ). We used MolView [https://molview.org/](https://molview.org/) to visualize the molecule diagrams for our paper.

## Appendix B Summary of notations

We summarize the notations used in our paper in the table [1](#A2.T1) below.

**Table 1: Notations summary**
| Symbol | Description |
| --- | --- |
| $\mathcal{S}$ | state space |
| $\mathcal{X}$ | terminal state space |
| $\mathcal{A}$ | action space ($s\rightarrow s^{\prime}$) |
| $\mathcal{T}$ | trajectory space |
| $s_{0}$ | initial state in $\mathcal{S}$ |
| $s$ | state in $\mathcal{S}$ |
| x | terminal state in $\mathcal{X}$ |
| $\tau$ | trajectory in $\mathcal{T}$ |
| $P_{F}$ | forward flow |
| $P_{B}$ | backward flow |
| $k$ | population size |
| $\mathcal{D}$ | replay buffer |
| $\epsilon$ | elite population ratio |
| $\gamma$ | mutation strength |

## Appendix C Fitness evaluation algorithm

Figure: Algorithm 2 Evaluation of Forward Flows

## Appendix D Additional implementation details

### D.1 Hypergrid task

The hypergrid reward function is defined by -

$$ $R(\texttt{x})=R_{0}+R_{1}\prod_{d=1}^{D}{\mathbb{I}\left[\left|\frac{\texttt{x}_{d}}{H-1}-0.5\right|\in\left(0.25,0.5\right]\right]}+R_{2}\prod_{d=1}^{D}{\mathbb{I}\left[\left|\frac{\texttt{x}_{d}}{H-1}-0.5\right|\in\left(0.3,0.4\right]\right]}$ (6) $$

where $\mathbb{I}$ is the indicator function and $R_{0},R_{1},$ and $R_{2}$ are reward control parameters. In our experiments, $R_{1}$ and $R_{2}$ stay at a fixed value of 0.5 and 2. In our experiments, $R_{0}$ varies within $\{10^{-3},10^{-4},10^{-5}\}$. A mode is the terminal state x for which $R(\texttt{x})=R_{max}$. From the equation [6](#A4.E6), it is evident that there are $2^{D}$ distinct modes. Besides, $H$ refers to the horizon of the environment, meaning each dimension of x can be equal to $i\in\{0,1,2,\dots,H-1\}$. For example, figure [5](#S4.F5) uses $D=5$ and $H=20$. Clearly, while increasing both $D$ and $H$ increases the complexity of the task, effecting the trajectory length $|\tau|$ and the number of states $|\mathcal{X}|$, only increasing $D$ increases the number of modes. To calculate the empirical probability density, we collect the past visited 200000 states and calculate the probability density.

Architecture We model the forward layer with a 3-layer MLP with 256 hidden dimensions, followed by a leaky ReLU. The forward layer takes the one-hot encoding of the states as inputs and outputs action logits. For FM, we simply use the forward layer to model the edge flow. For TB and DB, we double the action space and train the MLP as both the forward and backward flow. We use a learning rate of $10^{-4}$ for FM and $10^{-3}$ for both TB and DB, including a learning rate of 0.1 for $Z_{\theta}$. The replay buffer uses a maximum size of 1000, and we use a worst-reward first policy for replay replacement.

### D.2 sEH binder generation Task

Figure: Figure 11: Illustration of GFlowNets policy for sEH binder generation task. Figure adopted from Pan et al. ()
Refer to caption: /html/2402.02186/assets/imgs/seh_process.png

For this task, the number of actions is within 100 to 2000, depending on the state, making $|\mathcal{X}|\approx 10^{16}$. We allow the agent to choose from a library of 72 blocks. Similar to Bengio et al. (), we include the different symmetry groups of a given block, making the action count 105 per stem. We also allow the agent to select up to 8 blocks, choosing them as suggested by (Sterling & Irwin, ) from the ZINC dataset (Sterling & Irwin, ). Following Zhang et al. (), we use Tanimoto similarity, defined by the ratio between the intersection and the union of two molecules based on their SMILES representation. To maintain diversity, we define a mode to be a terminal state for which the normalized negative binding energy to the 4JNC inhibitor of the soluble epoxide hydrolase (sEH) protein is more than 7.5 and the tanimoto similarity of other discovered modes is less than 0.7. Note that this objective is more limiting than simply counting the number of different Bemis-Murcko scaffolds that reach the reward threshold like Bengio et al. (). Since we are focusing on both molecule separation and optimization, our approach is more applicable for de novo molecule design, while the scaffold-based metric is suitable for lead optimization.

Architecture Following Bengio et al. (), we use a message passing neural network (MPNN)(Gilmer et al., ) that receives the atom graph to calculate the proxy reward of the molecules. Similarly, we use another MPNN that receives the block graph for flow estimation. The block graph is a tree of learned node embeddings that represent the blocks and edge embeddings that represent the bonds. To represent the flow, we pass the stems through a 10-layer graph convolution followed by GRU to calculate their embedding and pass the embedding through a 3-layer MLP to get a 105-dimension logit. Similarly, to represent the stop action, we pass the global mean pooling to the 3-layer MLP. The MLPs use 256 hidden dimensions, followed by a leakyReLU. We use a learning rate of 0.0005 and a minibatch size of 4. For EGFN, we use an offline sample probability of 0.2. Besides, we use a reward exponent $\beta$ = 10 and a normalizing constant of 8. For RL and MCMC baselines, we use the implementation provided by Bengio et al. ().

Figure: Figure 12: Molecule blocks for the sEH protein task. Figure adapted from Bengio et al. ()
Refer to caption: /html/2402.02186/assets/imgs/seh_blocks.png

### D.3 TFBind8 task

For this task, the goal is to generate an 8-length DNA sequence that maximizes the binding activity score with a particular transcription factor SIX6REFR1 (Barrera et al., ). We use a precalculated oracle for the proxy reward calculation. Using a PA-MDP, we prepend or append a neucleotide in each step. Note that this formulation reduces the trajectory length significantly despite our effort to showcase better performance in long trajectories, but we use it following previous works.

Architecture Following (Shen et al., ), the GFlowNets architecture uses a 2-layer MLP with 128-dimension hidden layer parameterizing SSR ($\mathcal{S},\mathcal{S^{\prime}}\rightarrow\mathbb{R}^{+}$). For each training step, we train on both online and offline trajectories for three steps, using a minibatch of 32. Besides, we use a learning rate of $10^{-4}$ for policy and $0.01$ for $Z_{\theta}$. Finally, we use a reward exponent $\beta$ = 3 and an exploration probability of 0.01 (we do not use any exploration for EGFN).

### D.4 QM9 task

The goal here is to generate diverse molecules based on the QM9 data (Ramakrishnan et al., ) that maximize the HOMO-LUMO. To that end, we use the reward proxy that Jain et al. () provides based on Zhang et al. (). Similar to the sEH task, we generate molecules with atoms and bonds. The blocks used here are the following: C, 0, N, C-F, C=0, C#C, c1ccccc1, C1CCCCC1, C1CCNC1, CCC.

Architecture Using a PA-MDP, we use a 2-layer MLP with 1024 hidden dimensions for flow estimation. The reward proxy is a MXMNet proxy trained on the QM9 data. We use a reward exponent $\beta=1$. The learning rate and training style follow the ones used for the TFBind8 task, with the exception of exploration probability (0.1 here) and hidden dimension (1024 here).

We detail the summary of the training hyperparameters in table [2](#A4.T2).

**Table 2: Summary of the hyperparameters for all experiments**
|  | Hypergrid | sEH Small Molecules | TFBind8 | QM9 |
| --- | --- | --- | --- | --- |
| Learning Rate | $10^{-4}$ (FM), $10^{-3}$ | $5\times 10^{-4}$ | $10^{-4}$ | $10^{-4}$ |
| $Z_{\theta}$ Learning Rate | 0.1 | N/A | 0.01 | 0.01 |
| $\beta$ | 1 | 10 | 3 | 1 |
| MDP | Enumerate | Sequence Insert | PA-MDP | PA-MDP |
| Exploration $\epsilon$ (none for EGFN) | 0 | 0 | 0.01 | 0.1 |
| Replay Buffer Training | 50% | 0 (20% for egfn) | 50% | 50% |
| MLP layers | 3 | 3 | 2 | 2 |
| MLP hidden dimensions | 256 | 256 | 128 | 1024 |

## Appendix E Additional ablation experiments

### E.1 Number of population

To investigate the effect of population size, we vary the $k\in$ $\{5,10,15\}$, while keeping $\epsilon=0.2,D=5,H=20,R_{0}=10^{-5}$. We plot the results of the experiment in figure [13](#A5.F13). It shows that increasing $k$ beyond 10 leads to diminishing returns, motivating our choice of $k=10$ for all the experiments. For DB, however, increasing $k$ leads to considerable improvement. Indeed, this is useful because increased population size leads to more evaluation round required. While these evaluation round can be parallelized with threads as we do in our work, massive population size requirement is difficult to satisfy.

Figure: Figure 13: Experimental results for the hypergrid task for EGFN among different values of $k$ across the three training objectives. Top: the $\ell_{1}$ error between the learned distribution density and the true target density. Bottom: the number of discovered modes across the training process. Here, $H=20,D=5$. The results show that increasing population size leads to diminishing returns.
Refer to caption: /html/2402.02186/assets/imgs/ablation_population_size.png

### E.2 Elite population

Following ablation on $k$, we next perform ablation on the elite population ratio, $\epsilon\in\{0.2,0.4,0.6\}$. For this experiment, we use the same hypergrid settings for $k=10$. We plot the results of the experiment in figure [14](#A5.F14). It shows that low $\epsilon$ improves mode discovery, especially for FM and TB objectives. This result is reasonable: we apply mutation and crossover only to the non-elite population, so having a low number of elite population means we have a better chance at exploring using the non-elite population’s mutation and crossover.

Figure: Figure 14: Experimental results for the hypergrid task for EGFN among different values of $\epsilon$ across the three training objectives. Top: the $\ell_{1}$ error between the learned distribution density and the true target density. Bottom: the number of discovered modes across the training process. Here, $H=20,D=5$. We observe that lower $\epsilon$ leads to better results for FM and TB.
Refer to caption: /html/2402.02186/assets/imgs/ablation_elite_rate.png

### E.3 Replay buffer size

To understand the effect of replay buffer size, we run GFlowNets baseline with PRB and EGFN on a 20x20x20x20x20 environment with $R_{0}=10^{-5}$ for replay buffer size $\in\{1000,5000,10000\}$. We present the findings in figure [15](#A5.F15). From the figure, we see that increasing buffer size generally has little effect for GFlowNets, but it improves EGFN’s robustness a little.

Figure: Figure 15: Experimental results for the hypergrid task between GFlowNets and EGFN across different values of $|\mathcal{D}|$. Top: the $\ell_{1}$ error between the learned distribution density and the true target density. Bottom: the number of discovered modes across the training process. Here, we use DB for $H=20,D=5$. The results show that while increasing replay buffer size improves the robustness of the result, it has little effect otherwise.
Refer to caption: /html/2402.02186/assets/imgs/ablation_buffer_size_db.png

### E.4 Mutation strength

We now turn our attention to the mutation. To observe the effect of the mutation strength $\gamma$, we run the hypergrid experiment for the three training objectives using EGFN for $\gamma\in{1,5,10}$. The hypergrid configurations follow the the same configurations as before. We plot the mode discovery and $\ell_{1}$ error between the learned distribution density and the true target density over 2500 training steps in figure [16](#A5.F16). While the results indicate that having a higher $\gamma$ leads to better result for DB, the improvement is not extraordinary. Besides, in our work, we experience training instability for higher $\gamma$. Thus, we restrict $\gamma$ to be 1 throughout in our work.

Figure: Figure 16: Experimental results for the hypergrid task for EGFN among different values of $\gamma$ across the three training objectives. Top: the $\ell_{1}$ error between the learned distribution density and the true target density. Bottom: the number of discovered modes across the training process. Here, $H=20,D=5$. We observe that lower $\gamma$ leads to better results for FM and TB, while higher $\gamma$ leads to better results for DB.
Refer to caption: /html/2402.02186/assets/imgs/ablation_mutation_strength.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: What is the computational overhead of maintaining a whole population of GFN agents that must be evaluated in addition? 

Why not train all agents in the population or train the best agent(s) in the population instead of maintaining a separate star agent?

How do the authors ensure a fair comparison to the provided baseline regarding the number of evaluated trajectories?  (And how is the PRB filled for the baselines?)

Regarding the ablation EGFN-PRB-mutation, how is the EA connected with the training of the star agent?
soundness: 3
strengths: Overall, the proposed approach is well-explained and illustrated. The paper provides various empirical results in a simple exemplary task, as well as molecule generation tasks, to demonstrate real-world applicability.  For a fair comparison, it provides various baselines and ablations. The empirical results show improved performance, particularly in sparse reward scenarios and large state spaces. Furthermore, the authors provide a discussion on potential limitations and provide reasoning for the advantages of the proposed approach based on an intuitive empirical analysis
summary: The paper proposes using an Evolutionary Algorithm to fill a Prioritized Replay Buffer with (more) diverse trajectories to enhance the training process of Generative Flow Networks.
weaknesses: Regarding the proposed method, the EA does not seem to influence the actual training process beyond providing diverse experiences to be sampled. In that regard, an evaluation comparing the performance of the population to the star agent to validate the assumptions would have been helpful. Also, in addition to the provided baseline comparison, I am missing a comparison based on the number of evaluated trajectories to assess the sample complexity advantages of the proposed approach. While improving in sparse scenarios, the proposed approach seems to perform slightly worse in the more generic tasks despite being computationally more intense. Regarding the baselines used, especially GAFN and MARS, I am missing a short introduction, explanation, or comparison. Also, the sparsity levels shown in Fig. 4 should be elaborated more concretely. Regarding the presentation, the overall writing might be slightly improved, e.g., regarding grammar. 

Minor comments:

- p.3 l.109f.: abbreviations FM, DB, and TB should be introduced first. 
- Alg. 1, l.181: P^*_F should be P_F? Alternatively, the reason the star agent is used to evaluate the population should be elaborated on.
- Alg. 1 l.182f.: vars for online and offline trajectories should differ

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. How many reward calls are used per training step for EGFN and each baseline?
2. The biochemical tasks appear to share many similarities. Is there a specific reason for dividing them into three sections (4.2, 4.3, and 4.4)?
3. In lines 254-259, two prioritization methods are introduced: proportional sampling and percentile-based heuristics. Which one is actually used in the experiments?
4. I suspect that memory consumption increases linearly in $K$ (the population size). Is this true?
soundness: 1
strengths: 1. The idea of using Evolutionary algorithm that evolves a population of GFlowNets is new, though similar approaches have already been introduced in reinforcement learning to enhance exploration [1, 2].
2. The proposed algorithm is validated through various experiments, including real-world biochemical tasks. I also enjoyed their analysis of why the proposed algorithm works (section 5).

[1] Salimans, Tim, et al. "Evolution strategies as a scalable alternative to reinforcement learning." arXiv:1703.03864 (2017).  
[2] Khadka, Shauharda, and Kagan Tumer. "Evolution-guided policy gradient in reinforcement learning." NeurIPS (2018).
summary: This paper introduces Evolution guided generative flow networks (EGFN), a new algorithm equipped with an evolutionary algorithm (EA) for better generative flow network (GFN) training. EGFN collects diverse and high-reward samples using a population of GFNs that evolves throughout the training procedure. The collected samples are then utilized to train a target *star* GFN agent in an off-policy manner. EGFN showed faster learning capability in one synthetic and three biochemical tasks, especially when the reward signal is sparse.
weaknesses: 1. In its current form, the paper contains several ambiguous or incorrect claims in Section 2.1. Here are some key issues I noticed:  
1-1. **Lines 93 - 102**: The DAG structure should be defined first to clearly specify the action space according to the DAG’s edges. Additionally, the phrase “sample proportionally to different peaks of reward” in line 101 (and line 34) is misleading.  
1-2. **Line 112**: The notation $F(\tau)$ has never been defined but is used to define $F(s)$.  
1-3. **Line 118**: There’s an incorrect use of the prime ( ` ) symbol. Moreover, the equation $P_F(s' | s, \theta) = F(s \to s')$ is inaccurate. The RHS should be divided by $F(s)$. A similar issue appears in **lines 124-125**.  
1-4. **Line 133**: The expression $\sum_x R(x) = \sum_{s:s_0 \to s\in \tau \forall \tau \in \mathcal{T}} P_F(s|s_0;\theta)$ needs more explanation. At first glance, it doesn’t seem to hold generally.  
I believe these points could be clarified easily with careful revisions.

2. I’m unclear on why EGFN improves credit assignments. The star agent in the EGFN framework uses conventional learning objectives like DB or TB, and I couldn’t find any specific design element that enhances credit assignment. From what I understand, EGFN’s main advantage is its evolving population of GFNs, which provides more diverse experiences for the star agent to learn from. This should enhance exploration, which is especially beneficial in sparse environments.

3. I have some concerns about the experiments:  
3-1. Experiment Setup (Reward Calls): Were all algorithms given the same number of reward calls? All learning progress figures use training steps as the x-axis, but I suspect EGFN might use additional reward calls per training step due to the rewards needed for fitness calculation (line 173). However, in real-world applications where reward evaluation is costly (e.g., in vitro experiments), sample efficiency is often more critical than learning efficiency [3, 4]. Therefore, I recommend including results with a fixed number of reward calls, especially for biochemical sequence generation tasks.  
3-2. **(minor) Line 304 and 898**: The paper states the number of modes for the hypergrid task is $2^D$, but this doesn’t seem correct. There are indeed $2^D$ reward “regions” if a region is defined as a collection of adjacent modes. However, the actual number of modes could be $2^D \times M$, where $M$ represents the number of modes in each region, potentially increasing with $H$.

4. (minor) The reference is outdated and not well organized. Some of them, but not limited to, are: in line 663, Pan et al. 2023a was accepted by ICML 2023, and in line 728, Zhang et al. 2023b was accepted by TMLR. Also, there are two references for "Generative augmented flow networks."

[3] Gao, Wenhao, et al. "Sample efficiency matters: a benchmark for practical molecular optimization." NeurIPS (2022).  
[4] Kim, Hyeonah, et al. "Genetic-guided GFlowNets: Advancing in Practical Molecular Optimization Benchmark." arXiv:2402.05961 (2024).

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: - Can GFlowNets be applied to more traditional generative modelling tasks? (e.g. images, etc). 
- In Figure 4, it would help to clear up which of the labelled methods are RL, MCMC, or GFlowNet variants.
- In page 2 paragraph 2, it would be good to re-clarify what TB stands for, and introduce these prior objectives together.
- How are neural network weights mixed in the crossover step? Is this an important detail, or is a naive strategy good enough?
soundness: 2
strengths: The idea to apply evolutionary algorithms to GFlowNets is a new direction. Bootstrap-based methods such as GFlowNets often suffer from collapse or poor gradient flow, thus motivation evolutionary algorithms as a potential solution. The proposed method is simple, and outperforms previous methods on synthetic tasks. The proposed method consistently outperforms other GFLowNet settings, and discovers more modes of a solution.
summary: This paper introduces a variant of GFlowNets which is trained via evolutionary optimization. The key argument is that GFlowNets are a generative process which are learned via a reward signal, however the propagation of this reward throughout the time horizon is tricky. The paper proposes the use of classical evolutionary techniques to alleviate these issues. A population of networks are maintained, then selection + crossover + mutation are performed over the parameters during the evolutionary step. The evolutionary step is interleaved with traditional gradient descent.
weaknesses: The method introduces additional complexity in the form of the evolutionary optimization, but does not analyze why such a decision would improve performance. The evolutionary algorithms applied are well-known, and the combined algorithm boils down to an ad-hoc fitting of EA and gradient descent sequentially. The added complexity results in slower training speed, as mentioned in the paper. This paper would strongly benefit from a more principled look at the training dynamics of GFlowNets, and a stronger opinion on *why* evolutionary algorithms help learning. Given the smaller-scale nature of the tasks considered, this is a reasonable desire.