Genetic-guided GFlowNets for Sample Efficient Molecular Optimization
Authors: Hyeonah Kim
Minsu Kim
Sanghyeok Choi
Jinkyoo Park
Venue: NeurIPS 2024 poster (2024)
Keywords: Molecular Optimization
Sample Efficiency
Generative Flow Networks
Genetic Algorithm
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We propose a genetic-guided GFlowNet, which integrates a genetic search to guide GFlowNets to explore high-reward regions efficiently, achieving SOTA performance in an official molecular optimization benchmark.

Abstract: The challenge of discovering new molecules with desired properties is crucial in domains like drug discovery and material design. Recent advances in deep learning-based generative methods have shown promise but face the issue of sample efficiency due to the computational expense of evaluating the reward function. This paper proposes a novel algorithm for sample-efficient molecular optimization by distilling a powerful genetic algorithm into deep generative policy using GFlowNets training, the off-policy method for amortized inference. This approach enables the deep generative policy to learn from domain knowledge, which has been explicitly integrated into the genetic algorithm. Our method achieves state-of-the-art performance in the official molecular optimization benchmark, significantly outperforming previous methods. It also demonstrates effectiveness in designing inhibitors against SARS-CoV-2 with substantially fewer reward calls.

Genetic-guided GFlowNets for Sample Efficient Molecular Optimization
Hyeonah Kim
Minsu Kim
Sanghyeok Choi' Jinkyoo Park 1.2
Korea Advanced Institute of Science and Technology (KAIST), *OMELET
Abstract
The challenge of discovering new molecules with desired properties is crucial in domains like drug discovery and material design. Recent advances in deep learning-basse generative methods have shown promise but face the issue of sample efficiency due to the computational expense of evaluating the reward function. This paper proposes novel algorithm for cample-efficien molecular optimization by distilling powerful genetic algorithm into deep generative policy using GFlowNets training, the off- off-policy method for amortized inference. This approach enables the deep generative policy to learn from domain knowledge, which has been explicitly integrated into the genetic algorithm. Our method achieves state-of-the- art performance in the official molecular optimization benchmark, significantly outperforming previous methods. It also demonstrates effectiveness in designing inhibitors against SARS-CoV-2 with substantially fewer reward calls.
Introduction
Discovering new molecules is one of the fundamental tasks in the chemical domain, with applications in drug discovery [1] and material design [2]. Particularly, de novo molecular design focuses on generating novel molecules with desired properties from scratch. In this context, deep learning- based
generative methods have emerged, showing promising results (e.g., [3-7J). However, these methods still face a key challenge: the reward function is computationally expensive (e.g.. assessing binding affinity through docking simulations), while the molecule space is combinatorially large.
Sample-efficient molecular optimization is thus crucial for discovering high-reward molecular struc- tures with limited reward calls, especially for real-world applicability. The recently proposed bench- mark, Practical Molecular Optimization (PMO) [8], has extensively assessed the sample efficiency of various algorithms, including reinforcement learning [3, 9], active learning [10], variational au- toencoders [4, 5]. generative flow networks (GFlowNets) [6, 11]. and classical optimization methods like Bayesian optimization [12] and genetic algorithms [13, 14]. Interestingly, the PMO benchmark has revealed shift in algorithm rankings, with classical algorithms often outperforming recently proposed methods such as GFlowNets when the sample efficiency is considered.
Recent investigations [15, 16), including those highlighted by the PMO benchmark [8), indicate that the classical frameworks, especially genetic algorithms (GA), still exhibit competitive performances compared to recently proposed deep learning methods. These studies underscore that GAs effectively navigate the chemical space using domain-specific genetic operators. In contrast, deep learning methods usually do not leverage domain-specine knowledge, relying instead deep networks to autonomously learn these insights. It can lead to inefficient training processes due to the lack of expert guidance [17]. To address this limitation, Genetic Expert Guided Learning (GEGL) [17] has been introduced, which enhances deep learning by distilling GA-generated samples into a deep generative policy using maximum likelihood estimation. This approach enables the deep generative policy to
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

a. SMILES generation with policy
b. Graph-based genetic search
CC-Ccloo(C([We)- (C
DHCOSECOMM)
SMILES
Population
Crossover Mutation
Offspring
Pollcy g
(GFlowNet Training)
Buffer with reward evaluation
Overview of Genetic GFN. Our generative policy is trained to sample molecules proportional to rewards, and the genetic search refines them higher-reward samples.
Figure 1:
implicitly utilize domain-specific knowledge from the GA as form of guidance. However, despite its successes, GEGL, may face challenges in generalizing to unexplored regions and in learning peaky distribution from samples, as it maximizes likelihood equally across all high-reward samples without adequately configuring the reward landscape.
To address this, we propose a novel molecular optimization algorithm that integrates domain- specialized genetic algorithms into GFlowNets, an off-policy training method that trains policy sample proportional their rewards. As illustrated in Fig. 1, we first generate diverse candidates using the current policy and refine the candidates into higher-reward samples using GAs. Subse- quently, we fine-tune the policy with a GFlowNet using the collected samples. Unlike the MIE training, which can be stuck in local modes when the reward landscape is peaky, GFlowNet trains the policy to sample molecules proportional to rewards. To enhance sample efficiency, we perform unsu- pervised pretraining chemical datasets and regularize the GFlowNets policy with KL-divergence to align generative probability with the dataset distribution, focusing on the compact valid space.
Our contributions can be interpreted from both perspectives of GFlowNets and GAs:
GA increases the exploitation power of GFlowNets. The proposed algorithm incorporates an effective off- policy exploration into GFlowNets based on domain-speent specialized GA. This approach aligns with recent studies that utilize off-policy explorations to guide toward the high-reward regions [18]. Our key contribution lies in explicitly leveraging domain knowledge about chemical structures and effectively distilling into GFlowNets, which enable exceptional performance real-world tasks beyond small-scale molecular generations. This contribution is crucial for the field of GFlowNets, which have struggled with samplc-efficient molecular optimization tasks, even when using proxy reward models for active learning [10, 8]; see Section 4.2.
GelowNets increases population diversity of GA. The proposed algorithm generates diversified samples using GFlowNets, enabling GA to effectively improve samples; our method can be regarded as new genetic algorithm with deep generative policy-based population resetting. Our GFlowNet policy, parameterized over the whole space, periodically resets the population by diversely sampling individuals proportional to rewards, mitigating premature convergence of GA [19, 20]. Experiments show that ours outperforms recent GAs in the PMO benchmark; see Section 5.1.
Our extensive experiments demonstrate the effectiveness and practical applicability of the proposed method. First, our method achieves the highest total score of 16.213 across 23 oracles in the Practical Molecular Optimization benchmark [8], outperforming all other baselines. Second, we conduct in in silico experiments for designing AAV--CoV-2 inhibitors. The proposed method successfully generates inhibitors with ten times fewer reward calls. Moreover, our method effectively balances optimization and diversity, achieving higher scores with increased diversity compared to previously top-ranked methods.
Background
2.1 Sample efficient de nove molecular optimization
Molecular design the process of proposing new molecules likely to exhibit desirable outcomes. Compared to traditional virtual screening approaches, which identify suitable molecules from virtual

---

libraries with a large number of molecules known priori, de novo approaches seek to generate molecule structures anew. The desired properties can be measured using score functions O, called oracle. Formally, molecular design can be formulated as arg max (zz), where r is molecule, and X' denotes the chemical space which comprises all possible molecules.
The publication of standard benchmarks and datasets has facilitated the assessment of de novo design methods (e.g., GuacaMol [21]. Therapeutics Data Commons (TDC) [22]). The score functions are designed to consider various properties, such as the presence and absence of substructures, similarity, isomers, structural features, physicochemical properties, biological activity, and binding affinity (i.e., docking score). Notably, the PMO benchmark [8] offers unified framework that comprehensively evaluates the sample efficiency of range of molecular design methods.
2.2 Generative flow networks
Generative flow networks (GFlowNetss) [23] are introduced as a new class of probabilistic models to sample discrete compositional object X' from the target distribution, i.e., P(z) e e-((x)) In general, direct sampling from the target distribution is challenging since the partition function Z = Ezxx e -E(x) is intractable when the sample space is combinatorially large. Hence, GFlowNets sample object from an unnormalized distribution as constructive generative process, where discrete actions iteratively modify a state a partially constructed object. We define a trajectory as
(So. (Su...., .......................... sT), where 8y is terminal state corresponding to fully constructed object æ.
GelowNet models flow F of particles along directed acyclic graph (DAG). The source and sink nodes of the DAG correspond to the initial state 80 and terminal states ST, respectively. The trajectory flow (TT) is defined as flow through the trajectory т, and the state flow F(s) is defined as the sum of trajectory flows that include the state i.e., F(s) Σxer F(T). The edge flow F(s + 8') the sum of trajectory flows through the edge from state s to i.e., F(s + Eis,'Jer F(r).
From the flow function F, we derive two policy distributions. The forward policy Pr (s(s'(s) s) is the probability of transitioning from state to its child state 8', defined as the edge flow F(s - 8') normalized by the state flow F(s), i.e., Pr(s'(s) = F(s - s')/Fil (s). Similarly, the backward policy PB(s]s') is the probability of moving from state s' to parent state 8, defined as (s(ss'') = F(s s')/F(s'). Utilizing these forward and backward policies, GelowNets derive optimal sampler P(sT) = PI = R((T)( 3/6 balance conditions (e.g.. [6, 24-27J) are satisfied.
Trajectory balance loss [24]. One of the most popular conditions is trajectory balance (TB), which directly parameterizes Pr. PB. and flow of initial state (i.e., partition function) Z to satisfy the following trajectory balance condition:
z II
(F( = R(ST) II t=1
Pa (s(ss
t=1
Then, this equation is converted into loss function to be minimized along sampled trajectories, i.c.,
(log Zo IT (r(s) R(x) 11m PB (=(0:_1ss;;0)
LTB(T:0)
In GFlowNet training, employing exploratory behavior policies replay training is allowed since GFlowNet can trained in an off-policy manner, which is a key advantage [23, 24, 28, 18).
Genetic-guided GFlowNets
This section describes how the desired molecules are discovered with Genetic GFN. We model the generation process of molecules as a string-based constructive process. First, we pretrain the policy to learn the distribution of valid molecules. During the optimization phase, we iteratively generate molecules and update the policy with GFlowNet training using high-reward molecules. Particularly, we introduce graph-based genetic search to refine generated samples.
3

---

Algorithm Genetic GFN training with limited reward calls
1: Set &- Tpre. D f- 0
2: while [D] numOracle
3: D < DU U {ar, (x)) where x ~ TT(())
SMILES generation with policy Graph- genetic search
4: Initialize population D pop from
5: for
to numGen do
6:
a + Croasover x' -- Mutate(2)
where (æ1, Pop
7:
8:
D + U ( (z))), Dorr t Dou U f O(x))}
9:
D pop + Select(Dpoy Doar
10:
end for
11:
for k == numReplay do
Updating the policy with GFlowNel training Get B from with rank-based sampling (Eq. (4))
12:
13:
Update minimize
[ड| EeeB LTB ( (T (z) pre (æ))
14:
end for
15:
end while
3.1 Factorized string-based generative policy and unsupervised pretraining
Building on insights from previous works [3, 17], we employ a string-based representation strategy, simplifying the molecular generation process by reducing it to a one-directional sequence generation. We adopt sequence generative policy using string- -ased assembly strategy, especially the simplified molecular-input line-entry system (SMILES) [29]. Motivated by REINVENT [3], we parameterize the policy using a recurrent neural network architecture [30]. Then, the probability ne (a:) of generating molecule, can be factorized to I no (2 -t-1).; 1), where ; En are characters of SMILES representation of ac.
As demonstrated in previous studies, including [3, 17, 8], pretraining is inevitable since training the generative policy from scratch is excessively sample-ineffccient Therefore, our policy pre-traines to maximize the likelihood of valid molecules on existing chemical datasets 2 Prre note that pretraining does not require oracle information. Precisely, the policy is pretrained to minimize the following:
Cere
log no [1, ... It-1).
3.2
GFlowNet training of the generative policy with graph-based genetic search
To generate desirable molecules with limited reward calls, we iteratively generate samples using two distinct strategies (Section 3.2.1) and fine-tune the policy, initialized with Tpe. using GFlowNet by replaying collected samples (Section 3.2.2). The overall procedure is described in Algorithm 1.
3.2.1 Molecule generation strategies in Genetic GFN
We employ two distinct molecule generation strategies, SMILES generation with our training policy and graph-based genetic search. These two strategies are synergized to generate diversified and high-reward samples, efficiently searching the vast chemical space.
SMILES generation with policy. The training policy π& generates SMILES sequences. Since the policy is trained using the trajectory balance loss (see the following subsection), it is the same as sampling from I (r collected SMILES token, and log R(x) = (30(x) with an inverse temperature B.
R(s = xx), where 5;-1 is represented by previously
Graph-based genetic search. To effectively search the higher-reward region, we employ genetic algorithm that iteratively evolves populations through crossover, mutation, and selection. We adopt the operations of the graph-based genetic algorithm, Graph GA [13], which has proven to effectively search the molecule space with finely designed genetic operations; please refer to the original paper for details. The genetic search is performed follows:

---

1. Initialize a population D pop: The initial population is selected from the whole buffer D, consisting of samples from the policy and previous genetic search.
2. Generate offspring D A child is generated from randomly chosen two parent molecules by combining the fragments (crossover). Then, the child is randomly modified (mutation).
3. Select a new population Doop: Sample from Poop Dott. and go back to 2.
One key advantage is that offspring have large distance from the parents in the ID string space, even if the molecule distances are small, which is beneficial to avoid being stuck in local optima; see the experimental results in Table 3b.
3.2.2 Updating the generative policy with GFlowNets training
Using the generated samples, the policy is fine-tuned using the trajectory balance loss. The off-policy property of GFlowNet losses enables the utilization of refined samples from the genetic search. In particular, for better sample efficiency, we employ replay training with a rank-based reweighed buffer.
TB loss with KL-divergence penalty. The generative policy is trained using the trajectory balance loss in Eq. (1). Note that we set Pn to for simplicity since SMILES generations are conducted in one direction. To ensure that the policy does not deviate excessively from the pretrained policy during training, we introduce a Kullback-Leible (KI.) divergence penalty inspired by the works in language model fine-tuning [31, 32]. Thus, our model is updated minimize the following loss function:
L = LTB 0) + aKL (Ta (Tee
where Tpr denotes the the pre-trained policy. As result, is trained to generate desired (by LTB) and valid (by pre) molecules. Note that trajectories on which the proposed loss minimized are sampled from the experience buffer.
Rank-based reweighed experience buffer. wards high-rewars candidates by assigning greater weight to trajectories with higher ranks, thereby enhancing the focus on more promising solutions [33, 34]. The weight is computed as follows:
The rank-based reweighting biases the samples to-
(k|D) ranko,
EMED (k|D) + ranko ranko,,(x)))
Here, k is weight-shifing factor, and ranko, D (x) is relative rank of value of ((x) in the dataset D. Note that also utilize rank-based sampling the genetic search (steps and 3).
4
Related works
4.1 Genetic algorithms for molecular optimization
Genetic algorithm (GA) is representative hetaiheuristic inspired by the evolutionary process. This subsection focuses on discussing the application of GA in molecular optimization. As one of the seminal works, graph-basee based GA (Graph GA) was proposed with sophisticatedly designed operations based on chemical knowledge [13]. Note that our method also adopts Graph GA operations in the genetic search. Various strategies for molecular assembly, not limited to graphs, have been utilized in GA [35, 14, 36). A recent contribution by [16] introduces enhanced version of Graph GA. They introduce quantile-uniform sampling to bias population towards containing higher reward samples while maintaining diversity. Experimental results from Mol GA demonstrate the effectiienesss of GAs as strong baselines, achieving -fateoof-the-art (SOTA) performance the PMO benchmark.
4.2 GelowNets for molecular optimization
Generative Flow Networks (GFlowNets or GFN) [23, 6] have drawn significant attention in scientific
discovery [37], particularly molecular optimization and biological sequence design [6, 10, 38, 18, 39, 40, 11, 41]. GFlowNets, which are off-policy variational inference methods [42], are closely related to value-based reinforcement learning within soft Markov Decision Processes (soft MDPs) [43, 44], focusing learning maximum entropy agents [45, 46]. This allows GelowNets to generate
5

---

Table I: Mean and standard deviation of AUC top-10 (1) from five independent runs. We oracle ID (in lexicographical order) instead of a name for better readability, and the best mean scores are denoted in bold for each task. The results of further baselines provided in Appendix G.2.
Genetic GFN (Ours)
SMILES
Fragment
Pragment
ID
Mol GA [16]
REINVENT [3]
GEGL [17]
GP BO [12]
GFN [6]
GFN-AL [10]
#1
0.949
0.010
0.928 + 0.015
0,881 0,016
0.842
0,902 0.011
0.382 # 0.010
0.459 0.028
#2
0.761
0.019
0.740 + 0.055
0,644 = 0.019
0.626 1 0.088
0.579 = 0.055
0.428 +: 0.002
0.437 0.007
#3
0.802
0.029
0.629 + 0.062
0,717 = 0.027
0.044
0.746 = 0.025
0.263 + 0.009
0.326 + 0.008
#4
0.733 0.109
0.656 + 0.013
0.662 = 0.044
0.656
0.615 = 0.009
0.582 0.001
0.587 0.002
#s
0.974 0.006
0,950 + 0.004
0.957 0.007
0.898
0,941 0.017
0.480+0000
0.601 0.055
#6
0,856
0.039
0,835 + 0.012
0.781 = 0.013
0.769
0.009
0.726 0.004
0.689 + 0.003
0.700 + 0.008
#7
0.881
0.042
0.894 + 0.025
0.885 +: 0.031
0.816 0.027
0.861 + 0.027
0.589 +: 0.009
0.666 0.006
W8
0.969 0.003
0.926 + 0.014
0,942 + 0.012
0.883 0.040
0.791 0.024
0.468 0.211
#9
0.897
0.007
0.894 1. 0.005
0,838 0,030
0.808 0.007
0,805 0.007
0.576 1 0.021
0.199 0.199
#10
0.764
0.069
0.835 + 0.040
0.782 + 0.029
0.580 0.086
0.611 + 0.080
0.359 + 0.009
0.442 + 0.017
#11
0.379
0.010
0.329 1 0.006
0,363 1 0.011
0.338 1 0.016
0.298 0.016
0.192 1 0.003
0,207 0.003
0,294
0.007
0.284 1:
0,281 0,002
0.274 0.007
0.296 0.011
0.181 0.002
#13
0.708
0.U57
0.762 + 0,048
0.634 = 0.042
0.599 0.035
0.631
0.093
0.291 + 0.005
0.332 0.012
#14
0.860
0.008
0.853 1 0.005
0.834 1 0.010
0.832 1 0.005
0.788 1 0.005
0.787 1 0.002
0.785 1 0.003
#15
0.595
0.014
0.610 + 0.038
0.535 = 0.015
0.537 = 0.015
0.494 = 0.006
0.423 + 0.006
0,434 0.006
#16
0,942
0.000
1:
0,941 0,000
0.941 0.000
0,937 0.0UZ
0.904
0.917
#17
0,819 0.018
0.830 + 0,010
0.770 = 0.005
0.001
0.741 = 0.010
0.626 + 0.005
0.660 + 0.004
#18
0.615
0.100
0.568 + 0.017
0,551 = 0.024
0.010
0.535 = 0.007
0.461 + 0.002
0.464 + 0.003
0.634- 0.039
0.677 + 0.055
0,470 + 0.041
0.002
0,461 + 0.057
0.180
0.217 0.022
#20
0.583
0.034
0,544 + 0.067
0.544 0.026
0.028
0,544 0.038
0.261 + 0.004
0.292 0.009
#21
0,511
0.054
0.487 + 0.024
0.4588 + 0.018
0.420 + 0.031
0.404 + 0.025
0.183 + 0.001
0.190 + 0.002
#22
0.135
0.271
0.000 + 0.000
0.182 + 0.363
0.119 = 0.238
0.000 = 0.000
0.000
+: 0.000
0.000 + 0.000
#23
0.552
0.033
1: 0.033
0.533
0.009
0.492
0.466 0.025
0.308
0.027
0.353
Sum
16.213
15.686
15.185
14.354
14,264
9.929
9.917
Genetic GPN (Ours)
Mo G4
SMILES REWWENT
GEGL
GP 80
lommers 2h110222212d
relecsub redecovery
smindiate mon
scatfold hop
02
03.0
1600
$0.00
Mumr of www.eac..c.oo.......o Calls
Mari cell
Mari of 0030 kc call
Mam. of cradie calls
Figure 2: The optimization curve of the average scores of Top- 10 over the score function calls. All optimization curves for 23 oracles are provided in Appendix G.2.
diverse candidates for chemical and biological structures. Advances in GFlowNets have included new objective functions [6, 24, 25], improved credit assignments [26, 27], and enhanced off-policy exploration strategies [18, 47-49]. Our approach, an improved off-policy exploration technique for GFlowNets, enhances sample efficiency and scalability for moderate-seus scale chemical discovery. We present extensive experiments targeting the discovery of kurger-scale molecules, with SMILES strings approximately 100 characters long. surpassing the scope of smaller molecule generation tasks commonly addressed in GFlow literature [6, 38].
Experiments
This section provides extensive experimental results, including experiments the official sample- efficient molecular optimization benchmark and in silico design for SARS-CoV-2 inhibitors. The gfn.
codes are available at http://hhhbbboomm/yyyoooohm/////
5.1 Sample efficient molecular optimization
In this experiment section, we compare Genetic GFN with various molecular optimization methods from the perspective of sample efficiency. The performance is primarily quantified by the area under the curve (AUC), with the number of score function calls limited to 10K. Note that we rigorously follow the Practical Molecular Optimization (PMO) benchmark [8].

---

Table 2: Ablation studies. In the GS ablation study (-(GS)), the generative policy solely generates samples, while e-greedy samples from Pp mixed with uniform distribution. The bold text indicates the best value.
Genetic Search
-i-divergence penalty
Genetic GFN
- {GS}
{GS} {e-greedy)
AUC Top-1
16.530 + 0.198
16,070 + 0.290
15,966 + 0.085
16.251 + 0.440
AUC Top-10
16.213 + 0.173
15.738 + 0.274
15.626 + 0.082
15.928 + 0.426
AUC Top--10
15.516 + 0.127
15,030 + 0,322
14.939 + 0.147
15.188 + 0.297
5.1.1 Main results in the official benchmark of PMO
As baselines. we employ Top-8 methods from the PMO benchmark since they recorded the best AUC Top-10 10 in at least one oracle. The baseline methods include various ranges of algorithms and representation strategies. First, REINVENT [3] is an RL method that tunes the policy with adjusted likelihood. Graph GA [13], STONED [14), SMILES GA [21], and SynNet [36] are genetic algorithms that utilize different assembly strategies; they use fragment-based graphs, SELFIES, SMILES, and synthesis, respectively. Additionally, a hill climbing method (SMILES-LSTM-A
[21]) and Bayesian iteratively generates samples and
optimization (GP BO [12]) are included. SMILES-LSTM--EE imitates high-reward samples, while GP BO uses a surrogate model with the Gaussian process (GP) and Graph GA to optimize the GP acquisition functions in the inner loop.
Moreover, we adopt additional methods. Mol GA [16] and GEGL [17]. Mol GA is an advanced version of Graph GA and outperforms other baselines in the PMO benchmark. On the other hand, GEGL is an ablated version of our approach that utilizes imitation learning with a neward-priority queue instead of GelowNet training with rank-based sampling. For both, we adopt the original implementations 23 with hyperparameters searches following the guidelines; see Appendix C.
The main results in Table report the AUC score of Top- 10 candidates with independent five runs with different seeds. In addition, Fig. 2 visually presents the Top- 10 average score across the computational budget, i.e., the number of oracle calls, providing concise overview of the results. Due to the lack of space, the best five results are provided: please check Appendix G.2 for the rest of the results. As shown in Table 1, Genetic GFN outperforms the other baselines with a total of 16.213 and attains the highest AUC Top-10 10 values in 14 out of 23 score functions. The results of diversity and SA score for each oracle are presented in Appendix G.5.
5.1.2 Controllability of the scores-diversity trade-off and ablation studies
Controllability of the scores-diversity trade-off. In the benchmark, we found a pronounced trade- off between attaining high evaluation scores within a limited budget and generating diverse molecular candidates. This section demonstrates the controlla- bility of the score-diversity trade-off through adjust- ments in the inverse temperature B3. Decreasing the inverse temperature gives more diverse candidates. The results in Fig. 3 demonstrate adjustments of / can control the trade-off between score and diver- sity, achieving Pareto-frontier to other baselines in the benchmark. Notably, Genetic GFN with 3 = 30 achieves higher AUC Top-10 with a greater diver- sity compared to the SOTA GA method (Mol GA: 15.686 with diversity of 0.465) and RL method (REINVENT: 15. 18.185 with a diversity of 0.468). Similarly, the weight-shifting factor ls in based sampling can control the trade-off; see Appendix G.
Genetic CPM (8 50)
17.5
Ganetic CFN (8-
17.0
Mui
26.5
RERWENT
15.0
0.45
0.50
2.55
0.50
3.55
Diversity
A=1 1
9= 5
B 10
9
30
0 = 30
AUC-10
11.083
14.597
14.735
15.815
16.213
Div.
0.812
0.670
0.663
0.528
0.432
Figure 3: Average of Top- 10 score and diver- sity. Note that the fragment-based GelowNet achieves 10.957 with a diversity of 0.816.
Ablation studies. The ablation studies investigate the essential components of our framework: the genetic search (GS) and the KL-divergence penalty. To assess the effectiveness of the genetic search, we also compare its performance against the exploration strategy used in previous GFlowNet studies
thtrps://githubbbo
httppp//ggihah.coo/Auuttnn///// Aosti T/mol hhtp:///gihibb.oom//sooooaaaa/ee
7

---

Table 3: Comparison with GFlowNet variants. Notably, samples from the GS have larger SMILES distances than LS, leading to better sample efficiency. The bold text indicates the best value.
(a) Average and standard deviation of AUC scores (1)
(b) Search distances (t)
SMILES
Fragment-based
(GSK33)
SMILES
Molecale
Genetic GFN
L.SGFN [18]
GFN [6]
GFN-AL [10]
Generic GPN
LS-GFN
0.740
0.374
0,528
AUC Top-1
16.530 0.198
15,514 1: 0.269
10.957 i: 0.033
11.032 1: 0.016
(NK3)
SMILES
Molccule
AUC Top-10
16.213 0.173
15.230
0.026
9,918 0.027
9,928 0.027
Genetic
0.706
0.536
AUC Top-100
15.516 0.127
14.619 + 0.027
8.416 0.034
8.064 0.005
LS-GFN
0:403
0.512
[6, 10]. It samples actions from a GFlowNet sampler mixed with uniform distribution, similar to
<-greedy RL. The results, shown in Table 2, reveal that the removal of either component results in decline in performance, underscoring the importance of employing suitable exploration strategy. Detailed results, including statistical analysis, are provided in Appendix G.4.
5.1.3 Comparisons with GFlowNets variants
We compare Genetic GFN with the graph-based GFlowNet [6], GFlowNet-AL [10], and the local search GFlowNet (LS-GFN) [18] using SMILES representations, LS-GFN utilizes Monte Carlo Markov Chain (MCMC) techniques, incorporating partial backtracking and reconstructing solution trajectories with the training policy as the proposal distribution [18]. The experiments conducted on the PMO benchmark, and we implement LS-GFN with SMILES by replacing our genetic search with local search. Note that while the original LS-GFN employs the prepend-append MDP, which does not directly apply to SMILES, we use the same one-directional SMILES generation as ours.
As shown in Table 3a, Genetic GFN outperforms other GFlowNet variants. Notably, generating SMILES is significantly more advantageous than generating graph-based fragments. The performance gap between Genetic GFN and LS GFN highlights the importance of proper exploratory policy. To further analyze, we measure the distance between samples before and after searches in GSK33 and JNK3. The normalized Levenshtein distances for SMILES and Tanimoto similarity for molecules reported in Table 3b. The results show that the local search may be inefficient in effectively searching in moderate-scale chemical spaces because its capabilities heavily depend on the current policy, leading to suboptimal search performance. In contrast, our approach leverages a domain domain-specialized genetic search within the molecule graph space, working an effective off- off-policy exploration the samples with SMILES representation are used to train the string-based generative policy.
5.1.4 Sample efficient multi-objective molecular optimization
According to Zhu et al. [11], we apply our method multi-objectivve tasks: GSK3 and +SK33++NK3+QED+S.
Table 4: Average and standard deviation of hy- pervolumes (1) for each task. The baseline re- sults are directly from the HN-GFN paper [1]]. The bold text indicates the best value.
Notably, GSK38 and JNK3 are potential targets of Alzheimer's Dis- ease treatments [50]. We use linear combination of each objective with given coefficients, and the performance is measured by hypervolumes with IK evaluations. We obtained the results from five independent trials using different seeds. Even though Genetic GFN is not designed for multi- objective molecular optimizattin, it demonstrates notable performance using proper scalar-valued score functions; please Appendix E for details.
GSK3,8 + JNK3 QED SA
GSK33 JNK3
HierVAE+/Part:G0
0.205 + 0.015
0.186 4: 0.009
HierVAE=/EHVI
0.341 0.072
0.211
LaMOO
0.279 = 0.090
0,190 : 0.069
Graph GA
0.368 + 0.020
0.335 4: 0.021
MARS
0.418 0.095
0.273
HN GFN
0.669 = 0.061
0.416 0.023
Genetic GFN
0.718 = 0.138
0.642 0.053
5.1.5 Further analysis
Active learning with Genetic GFN. Similar to Gel, ours work as a generative model in active learning. We compare Genetic GFN-AL with other model-bur -adel- and active learning methods; please refer Appendix D.
Genetic GFN with SELFIES representation. Genetic GFN with SELFIES generation achieves the improved sample efficiency to other SELLFFEEbbbse methods; see Appendix G.6.
8

---

Score: 0.983 (docking: -12-7, OED: 0.383, SA: 2-720)
0.989 (docking: -12.5, CE0: 0.412, SA: 2.251)
0.917
(dockling: -12.0, OED: 8.859, S4: 3.376)
Figure 4: The final candidates for the PI.Pr_JJP 7JIR target with 100 steps.
Score: 0.892
Score: 0.900 (dacking: -13.5, QED: 0.378, SA: 3.35
Score: 0.894 (docking: -13.7, QED: 8.197, SA: 2.951)
(docking: -13.8, QED: 8,268,
SA: 2.2)
Figure 5: The final candidates for the RdRp RdRp_6YYT 6YYT target with 100 steps.
Sensitivity analysis. We provide the experimental results by varying the hyperparameters, such as the offspring size and the number of training loops; please refer to Appendix G.8.
5.2 Designing inhibitors against SARS-CoV-2 targets
In this subsection, we conduct drug discovery experiments for Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-Cov-2) known as the novel coronavirus. One desired property is maximizing the binding affinity to the target protein. The binding affinity measured with a docking score, which is calculated based on the energies of the interaction between the ligand and the receptor. Typically, the computation of docking scores is expensive since it involves predicting the spatial orientation and binding affinity of the molecule in the active site of the target protein. We employ Quick Vina 2 [51] docking software to assess generated molecules.
Additionally, QED (Quantitative Estimate of Drug-likeness) and SA (Synthetic Accessibility) are considered to quantify the drug-likeness and difficulties of synthesizing. The higher QED, which ranges [O, 1]. and the lower SA, which ranges [0, 10], are desired. Therefore, we define the score linear combination of normalized scores according to the previous work [52]. Following [53] and [52], the target proteins are selected:
function s(x) for designing SARS-Cov-2 inhibitors
PLPro 7JIR, critical enzyme in the life cycle of C-V-C and RdRp RdRp_6YYT, which essential for the replication and the transcription of genes.
The experiments are conducted with up 1000 update steps with 128 batch size [52]. As shown in Table 5, ours achieves the highest Top-100 average scores only with 100 steps, which is 10 times fewer than others. Note that the score is recalculated based on the normalized score function in Eq. (5) using average values in the MoIRL-MGPT paper [52]; the full results and a more detailed experimental setup are provided in Appendix F, We also report the best candidates of 100 steps in Fig. and in Fig. 5. The final molecules correspond to the Top-1 score molecules from 3 independent runs.
Table Average Top- 100 scores (t). Ours outperforms baselines with 10 times fewer steps. The bold denotes the best scores.
PLPro
RdRp
JT-VAE GFlowNet Graph GA
0.272
0.216
0.326
0.280
0.723
0.786
REINVENT
0.717
0.799
MoIRL-MGPT
0.772
0.854
Genetic GFN (100)
0.891
0.873
Genetic GFN (1000)
0.925
0,902
Discussion
This paper introduces a Genetic-guided GFlowNet (Genetic GFN), which integrates a domain-specitn genetic algorithm to guide the GFlowNet policy toward higher-reward samples. The method employs off-policy training with rank-based reweighted buffer, enhancing the policy as a powerful amortized inference sampler for chemical discovery. Extensive experiments demonstrate that Genetic GFN effectively generates desirable molecules within the high-dimensional chemical space, including long chemical structure sequences (c.g., 2 100). On the other hand, approach can be considered as

---

novel population reinitialization strategy for genetic algorithms using GFlowNets, which sample diverse objects proportional rewards.
Limitations and future works. Our method assumes the existence of effective genetic algorithms, which is valid for the molecular design domain. However, designing domain-specifie operators for genetic algorithms can be challenging in other fields. One possible future work is to enhance genetic algorithms using the neural policy similar to recent studies [54, 55J. Another direction our approach to other domains, such as combinatorial optimization. For instance, we could utilize powerful GA, hybrid genetic search [56], to design GFlow solver for routing problems.
extend
This paper introduces a new generative model, significantly enhancing sample efficiency in molecular optimization. This advance is likely to hold substantial promise for this field, potentially accelerating the development of new therapies and advanced materials. Our research is currently focused on in-silico experiments. The potential safety concerns of discovered molecules further examined the subsequent processes, such as in-vitro experiments and pre-clinical tests.
Broader Impact.
Acknowledgments and Disclosure of Funding
This work was supported by the National Research Foundation Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024-00410082) The authors grateful to Austin Tripp and Xiuyuan Hu for their help with baselines.
References
[1] James P Hughes, Stephen Rees, S Barrett Kalindjian, and Karen L Philpott. Principles of early drug discovery. British Journal of Pharmacology, 162(6):1233-1222 1249, 2011.
[2] Alexander W Hains, Ziqi Liang, Michael A Woodhouse, and Brian A Gregg. Molecular semiconductors in organic photovoltaic cells. Chemical Reviews, 110(1 1)668996635, 2010.
[3] Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of Cheminformatics, 9(1::1-14, 2017.
[4] Rafael Bombarllo Jennifer N Wei, David Duvenaud, José Miguel
Benjamín Sánchez-Lengeling Dennis Sheberla, Jorge Aguilera- Iparraguirre Timothy D Hirzel, Ryan Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven contimuous representation of molecules. ACS Central Science, 4(2):268-22 2018.
[5] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International Conference Machine Learning, pages 2323-2332. PMLR, 2018
[6] Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non non-iterative diverse candidate generation. In Advances in Neural Information Processing Systems, volume 34, pages 27381 -2334, 2021.
[7] Seul Lee, Jaehyeong Jo, and Sung Ju Hwang. Exploring chemical space with score-based out-of- distribution generation. In International Conference Machine Learning, pages 18872-18892. PMLR, 2023.
[8] Wenhao Gao, Tianfan Fu, Jimeng Sun, and Connor Coley. Sample efficiency matters: benchmark for practical molecular optimization. Advances in Neural Information Processing Systems, 35:21342-2133 2022.
[9] Zhenpeng Zhou, Steven Kearnes, Li, Richard N Zare, and Patrick Riley. Optimization of molecules via deep reinforcement learning. Scientific Reports, 91): 2019.
[10] Moksh Jain, Emmanuel Bengio, Alex Hernández-Garaa Jarrid Brotor-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with GFlowNets. International Conference on Machine Learning, pages 9786-9801. PMLR, 2022.
10

---

[11] Yiheng Zhu, Jialu Wu, Chaowen Hu, Jiahuan Yan, Tingjun Hou, Jian Wu, et al. Sample-efficient multi-objective molecular optimization with GFlowNets. Advances in Neural Information Processing Systems, 36, 2024.
[12] Austin Tripp. Gregor N. C. Simm, and José Miguel Hernández-Lobato. A fresh look at de novo molecular design benchmarks. In NeurIPS 2021 AI for Science Workshop, 2021.
[13] Jan H Jensen. A graph-based genetic algorithm and generative model/Monte Carlo tree search for the exploration of chemical space. Chemical Science, 10(12):3567-3522 2019.
[14] AkshatKumar Nigam, Robert Pollice, Mario Krenn, Gabriel dos Passos Gomes, and Alan Aspuru-Guzik. Beyond generative models: superfast traversal, optimization, novelty, explo- ration and discovery (STONED) algorithm for molecules using SELFIES. Chemical Science, 12(20)-779--7099, 2021.
[15] AkshatKumar Nigam, Pascal Friederich, Mario Krenn, and Alan Aspuru-Guzia Augmenting genetic algorithms with deep neural networks for exploring the chemical space. In International Conference on Learning Representations, 2020.
[16] Austin Tripp and José Miguel Hernández-Lobate Genetic algorithms are strong baselines for molecule generation. arXiv preprint arXiv:2310.09267. 2023.
[17] Sungsoo Ahn, Junsu Kim, Hankook Lee, and Jinwoo Shin. Guiding deep molecular optimization with genetic exploration. In Advances in Neural Information Processing Systems, volume 33, pages 12008-12021, 2020.
[18] Minsu Kim, Taeyoung Yun, Emmanuel Bengio, Dinghuai Zhang, Yoshua Bengio, Sungsoo Ahn, and Jinkyoo Park. Local search GFlowNets. In International Conference Learning Representations, 2024.
[19] David B Fogel. An introduction to simulated evolutionary optimization. IEEE Transactions on Neural Networks, 5(1):3 14, 1994.
[20] Hari Mohan Pandey, Ankit Chaudhary, and Deepti Mehrotra. A comparative review of ap- proaches to prevent premature convergence in GA. Applied Soft Computing, 24:1047-1077, 2014.
[21] Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. GuacaMol: bench- marking models for de novo molecular design. Journal of Chemical Information and Modeling. 1993): 1108, 2019.
[22] Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf H Roohani, Jure Leskovee, Connor W. Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. Thirty-ffith Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[23] Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. Journal of Machine Learning Research, 24(210):1-55, 2023.
[24] Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in GFlowNets. In Advances in Neural Information Processing Systems, volume 35, pages 5955-5967, 2022.
[25] Kanika Madan, Jarrid Rector-Brooks Maksym Korablyov, Emmanuel Bengio, Moksh Jain, Andrei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gllownets from partial episodes for improved convergence and stability. In International Conference on Machine Learning, pages 23467-23483. PMLR, 2023.
[26] Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of GFlowNets with local credit and incomplete trajectories. In International Conference on Machine Learning. pages 26878 26890. PMLR, 2023.
[27] Hyosoon Jang, Minsu Kim, and Sungsoo Ahn. Learning energy decompositions for partial inference of GFlowNets. In International Conference on Learning Representations, 2024.
11

---

[28] Heiko Zimmermann, Fredrik Lindsten, Mon-Willem van de Meent, and Christian A Naesseth. variational perspective on generative flow networks. Transactions on Machine Learning Research, 2023.
[29] David Weininger. SMILES, a chemical language and information system. 1. introduction to methodology and encoding rules, Journal of Chemical Information and Computer Sciences, 28(1):31-36, 1988.
[30] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 1412.3555, 2014.
[31] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.0883 08593, 2019.
[32] Rafael Rafailov, Archit Sharma, Eric Mitchell. Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728-537411 2023.
[33] Austin Tripp, Erik Daxberger, and José Miguel Hernández- Lobato. Sample-efficient optimiza- tion in the latent space of deep generative models via weighted retraining. In Advances in Neural Information Processing Systems, volume 33, pages 1229--12777 2020.
[34] Minsu Kim, Federico Berto, Sungsoo Ahn, and Jinkyoo Park. Bootstrapped training of score- conditioned generator for offline design of biological sequences. In Advances in Neural Information Processing Systems, volume 36. pages 67643-67661, 2023.
[35] Naruki Yoshikawa, Kei Terayama, Masato Sumita, Teruki Homma, Kenta Oono, and Koji Tsuda. Population-based de novo molecule generation, using grammatical evolution. Chemistry Letters, 47(11):1431-1444, 2018
[36] Wenhao Gao, Rocío Mercado, and Connor W. Coley. Amortized tree generation for bottom- up synthesis planning and synthesizable molecular design. In International Conference on Learning Representations, 2022.
[37] Moksh Jain, Tristan Deleu, Jason Hartford, Cheng-Hao Liu, Alex Veeranndez-Gariia. and Yoshua Bengio. GetowNets for Al-driven scientific discovery. Digital Discovery, 23)55577-577, 2023.
[38] Max W Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving GFlowNet training. In International Conference on Machine Learning. pages 30956-30975. PMLR, 2023.
[39] Pouya M Ghari, Alex Tseng, Gökcen Eraslan, Romain Lopez, Tommaso Biancalani, Gabriele Scalia, and Ehsan Hajiramezanali. Generative flow networks assisted biological sequence editing. In NeurlPS 2023 Generative Al and Biology (GenBio) Workshop. 2023.
[40] Miruna Cretu, Charles Harris, Julien Roy, Emmanuel Bengio, and Pietro Liò. SynFlowNet: Towards molecule design with guaranteed synthesis pathways. In ICLR 2024 Workshop on Generative and Experimental Perspectives for Biomolecular Design, 2024.
[41] Hyconah Kim, Minsu Kim, Sungsoo Ahn, and Jinkyoo Park. Symmetric replay training: Enhancing sample efficiency in deep reinforcement learning for combinatorial optimization. In Proceedings of the 4/st International Conference on Machine Learning, pages 24110-24436, PMLR, 2024.
[42] Nikolay Malkin, Salem Lahlou, Tristan Deleu, Xu Ji, Edward Hu, Katie Everett, Dinghuai Zhang, and Yoshua Bengio. GelowNets and variational inference. In International Conference on Learning Representations, 2023.
[43] Sobhan Mohammadpour, Emmanuel Bengio, Emma Frejinger, and Pierre-Luc Bacon. Maximum entropy GFlowNets with soft Q-learning. In International Conference on Artificial Intelligence and Statistics, pages 2593-2601. PMLR, 2024.
12

---

[44] Tristan Deleu, Padideh Nouri, Nikolay Malkin, Doina Precup, and Yoshua Bengio. Discrete probabillistic inference as control in multi-path environments. In The 40th Conference on Uncertainty in Artificial Intelligence, 2024.
[45] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In Advances in Neural Information Processing Systems, volume 30, 2017.
[46] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pages 1352-1361. PMLR, 2017.
[47] Jarrid Rector-Brooks, Kanika Madan, Moksh Jain, Maksym Korablyov, Cheng- Hao Liu, Sarath Chandar, Nikolay Malkin, and Yoshua Bengio. Thompson sampling for improved exploration in GFlowNets. In JCML 2023 Structured Probabilistic Inference Generative Modeling (SPIGM) Workshop, 2023.
[48] Minsu Kim, Joohwan Ko, Taeyoung Yun, Dinghuai Zhang, Ling Pan, Woochang Kim, Jinkyoo Park, Emmanuel Bengio, and Yoshua Bengio. Learning to scale logits for temperature- conditional GFlowNets. In International Conference on Machine Learning, 2024.
[49] Shuai Guo, Jielei Chu, Lei Zhu, and Tianrui Li. Dynamic backtracking in GFlowNet: Enhancing decision steps with reward-dependent adjustment mechanisms. arXiv preprint arXiv:2404.05577 2024.
[50] Yibo Li, Liangren Zhang, and Zhenming Liu. Multi-objective de novo drug design with conditional graph generative model. Journal of Cheminformatiis, 10: 10:1-24, 2018.
[51] Amr Alhossary, Stephanus Daniel Handoko, Yuguang Mu, and Chee-Keong Kwoh. Fast, accurate, and reliable molecular docking with QuickVina Bioinformatics, 31(33)2214-22216 2015.
[52] Xiuyuan Hu, Guoqing Liu, Yang Zhao, and Hao Zhang. De novo drug design using rein- forcement learning with multiple GPT agents. In Advances in Neural Information Processing Systems, volume 36, pages 7405-7418. 2023.
[53] David M Rogers, Rupesh Agarwal, Josh Vermaas, Micholas Dean Smith, Rajitha T Rajeshwar, Connor Cooper, Ada Sedova, Swen Boehm, Matthew Baker, Jens Glaser, et al. SARS-CoV2 billionccomouund docking. Scientific Duta, 10(1):173, 2023.
[54] Tianfan Fu, Wenhao Gao, Connor Coley, and Jimeng Sun. Reinforeed genetic algorithm for structure- based drug design. In Advances in Neural Information Processing Systems, volume 35, pages 12325-12338. 2022.
[55] AkshatKumar Nigam, Robert Pollice, and Alán Aspuru-Guzii Parallel tempered genetic algorithm guided by deep neural networks for inverse molecular design. Digital Discovery, 1(4):390-404, 2022.
[56] Thibaut Vidal, Teodor Gabriel Crainic, Michel Gendreau, Nadia Lahrichi, and Walter Rei. A hybrid genetic algorithm for multidepot and periodic vehicle routing problems. Operations Research, 60(3):611 1-644, 2012.
[57] Joshua Meyers, Benedek Fabian, and Nathan Brown. De novo molecular design and generative models, Drug discovery today, 26(11))270772715,,000 2021.
13

---

Unsupervised Pretraining
b. Genetic GFN Fine-tuning with Oracle
Maximum Likelihood
Lya+ (KL(m)())) xgre (x))
oa moy
Buffer
Reward evaluation
ZINC Oracle]
Policy pine
Policy
Graph-based genetic search
Figure 6: Pretraining and Genetic GFN fine-tuning framework
A
Implementation details of Genetic GFN
Genetic GFN is implemented on top of the PMO benchmark source code (MIT license). Mostly, we adopt the REINVENT implementation including the RNN models and experience buffer; the code is included in the benchmark, and the original implementation is also accessible with Apache-2.0 license. See the following subsections for details.
A.1 Network architecture and pretraining
Network architecture. Our policy network is parameterized using a recurrent neural network containing multiple GRU cells [30]. In molecular optimization, RNN-based models with string molecular representations have proven to be successful [3, 17, 52]. In experiments, we employ the same hyperparameters directly compare with REINVENT, whose input embedding dimension is 128 and hidden dimension is 512 with three layers.
Pretraining. According the PMO benchmark guidelines [8], the pre-training is conducted on ZINC 250K. The overall framework is illustrated in Fig. 6. Since the network architecture is the same as REINVENT, we adopt the provided pretrained model for REINVENT the PMO benchmark. This allows the direct comparison of fine-tuning approaches.
A.2 Hyperparameters
We mostly follow the hyperparmeter setup of REINVENT and GEGL. For instance, the batch size and learning rate are set as 64 and 0.0005 according to REINVENT in the PMO benchmark. On the other hand, the mutation rate and the number of training loops are set 0.01 and 8 following GEGL. We use 64 samples for the replay training and population size, the same as the batch size without tuning. Lastly, the learning rate of Z, the partition function, is set 0.1, also without tuning.
In contrast, we have searched several hyperparameters, offspring size, the number of GA generations, and KL-divergence coefficient We provide the sensitivity analysis for the offspring size and the number of GA generations Appendix G.8. Furthermore, we use the inverse temperature 3 = 10 and the weight-shifting factor A 0.01, but they can be differently used to control score-diversity trade-off, as explained in Section 5.1.2.
A.3 Computing resource
Throughout the experiments, we utilize 48-core CPU, Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz, and single GPU. In the PMO benchmark, runtime varies from less than 10 minutes to several hours for 10K evaluations, depending on tasks and algorithms. However, most of the runtime is consumed in evaluating score functions the motivation for why the sample efficiency matters. On the other hand, in the SARS-CoV-2 inhibitor design tasks, 1000 training steps with batch size of 128 require more than 1 day for PIPro_7JIM and 2 days for RdRp_6YYTE more than 95% of the time is used to evaluation [52].
hhtps://gthhh.com//ennao wwwt::thwwwwwcwwww.wt....ea/o..e www.aaooooooooooooaaoooooooooloo _oportooopppppo_ooo
14

---

Ring Non-ring
Delete stom
Delete cyclic bond
HyC CHty
Apuend atom Hg = D12
Add ring
Mil
Change stom
Change bond order
HO
CH2 HS
HyC
(a) Crossover
(b) Mutations
Figure 7: Examples Graph GA operations. These operations are conducted according to predefined SMARTS patterns to ensure molecule validity, such as adherence valence rules.
B
Genetic operations
This section details each operation in our genetic search; Fig. for illustration. Note that adopt Graph GA of [13], which has demonstrated its powerful performances and has been adopted by GA-related works like Mol GA [16] and GEGL [17].
B.1
Crossover
crossover operation is conducted generate new candidate (called offssring) by exchanging the genetic information of pair of selected individuals (parents). This process mimics the crossover of genetic material in biological reproduction. the context of molecular optimization with graphs, the crossover operation is conducted in two types: 'ring crossover' and 'non-ring crossover with a 50% probability.
These two main crossover operations perform crossover between two parent molecules by cutting and recombining ring substructures. Ring crossover performs ring specifically designed to target ring structures within the molecule. The ring-cut operation cuts the molecule along two different ring patterns, selected randomly. One of the ring patterns checks for specific arrangement of four consecutive ring atoms, and the other pattern checks for ring atom connected to two other ring atoms with single bond. If suitable ring pattern is found, it cuts the molecule along that pattern, resulting in two fragments. On the other hand, non-ring crossover cuts single bond, meaning it is not part of cyclic (ring) structure within the molecule. The obtained fragments from both parents are recombined to create new molecules by applying predefined reaction SMARTS patterns. These operations are repeated for validity ensure that the resulting molecules meet structural and size constraints.
B.2 Mutation
The mutation is random change that is introduced to the genetic information of some individuals. This step adds diversity to the population and helps explore new regions of the solution space. In this work, employ seven different mutation processes and randomly select one of these mutations to modify the offspring molecules slightly. The operations consist atom and bond deletions, appending new atoms, inserting atoms between existing ones, changing bond orders, deleting eyclic bonds, adding cyclic rings, and altering atom types.
1. Deletion of atom: it selects one of five deletion SMARTS patterns, each representing the removal of a specific number of atoms or bonds. These patterns include the removal of single atom, single bond, bond with two attached atoms, and bonds with multiple attached atoms. The selected pattern is applied to the molecule, deleting the specified atom(s) or bond(s).
2. Appending atom:
introduces a new atom to the molecule. The type atom (e.g., C. N, O) and the type of bond (single, double, or triple) are chosen based on predefined probabilities. The function then generates a reaction SMARTS pattern to append the selected atom the molecule, forming bond.
15

---

3. Inserting atom: it inserts new atom between two existing atoms in the molecule. Similar to the appending atom, it selects the type of atom and bond based on predefined probabilities and generates a reaction SMARTS pattern to insert the atom.
4. Changing bond order: it randomly selects one of four SMARTS patterns, each representing change in the bond order between two atoms. These patterns include changing single bond to double bond, a double bond to triple bond, and vice versa.
5. Deletion of cyclic bond: it deletes a bond that is a part of a cyclic structure within the molecule. The SMARTS pattern represents the breaking of a cyclic bond while retaining the atoms connected by the bond.
6. Adding ring:
introduces a new cyelic ring into the molecule by selecting one of four SMARTS patterns, each representing the formation of specific ring type. These patterns create different types of cyclic structures within the molecule.
7. Changing atom: it randomly selects types of atoms from predefined list and generates a SMARTS pattern to change one atom into another. This operation modifies the atom type within the molecule.
16

---

C Hyperparameter setup for baseline methods
In this subsection, we present detailed descriptions for hyperparameter tuning of baselines. Except for Mol GA and GEGL, we adopt all the hyperparameters, initial datasets, and pre-trained models provided in the PMO benchmark. For hyperparameters that affect the sample efficiency, such as population size, we have searched for the proper hyperparameters following the guidelines suggested by [8]. In detail, we tune Mol GA and GEGL on the average AUC Top- 10, the main performance metric. from 3 independent runs of two oracles, zaleplon seppoo__ppp mmo and perindopril _mp. _mp... The best configurations are used in the main experiments.
Mol GA. As mentioned in Section 5.1, we set the offspring size as 5, the most crucial hyperparam- eter, according to the original paper [16]. Then, we searched the starting population size in [100, 200, 500, 1000] and the population size [100, 200, 500]. As shown in Fig. 8, we found the best configuration to be 500 and 100 for the starting population size and the population size, respectively.
papettion_10 Aize
statingg wwtp:/ooop/oooo.... 59ze
PMg.
662
1.10
182
350
330
250
462
200
302
130
1,000
000
Figure 8: Hyperparameter tuning results for Mol GA
GEGL. In GEGL, the policy sampling size is set as the expert sampling size, and both priority queue sizes (denoted as num keep) are the same as the original implementation. Thus, we searched the expert sampling size in [64, 128, 512] and the priority queue size in [128, 512, 1024]. Originally, they were set as 8192 and 1024, which are improper to the sample efficient setting. For the training batch size, we use 64, which is the same as ours. We use the pretrained policy provided in the original code and adapt the setup of the rest of the hyperparameters, including mutation rate, learning rate, and the number of training loops. Based on the results in Fig. 9, we the expert sampling size and priority queue size as 128.
250
555
1.0
240
102-
200
412
230
442
200
356
130
352
257-
156-
362
8.00
2.00
100
233
230
385
1.00
Figure 9: Hyperparameter tuning results for GEGL
17

---

D Comparison Genetic GFN-AL with active learning and model-based algorithms
D.I Implementation of Genetic GEN-AL
We mostly adopt the implementation of GFlowNet-AL, including the training method and the utilization of acquisition function [10]. The source code is accessible online with MIT license." " Additionally, the number of training proxy and generative models are aligned with the PMO benchmark standards, while the replay training and genetic search hyperparameters are set those
used our method in Genetic GFN. The pscudo-code is as follows.
Algorithm 2 Multi-round active learning with Genetic GFN
Input: Pretrained policy Trre
Output: Top-K elements of discovered molecule dataset D Set me f Trr D
2: Initialize the proxy model fø
3: while [D] - numOracle do
4:
5: for PROXY TRAINING
numTrainProxy do
6: Sample æ from with GeneticSearch
7:
( + (z)
8:
DU
DU ((x 11
9:
Update @ minimize Σ (f) y)²
end for
11:
GENERATIVE POLICY TRAINING
Dinnr -
13:
for
| 1 to numTrainPolicy do
14:
Get m' = [m(1 - >)1 samples from Ty with GeneticSearch
15:
Get m' samples from D
16:
Dinner + Dinner U (((((()) o(x
> Acquisition function from [10]
for r = to numReplay do
18:
Get B from Dinner with rank-based sampling
Update to minimize
[ Σ (a)) LTB (x)
20:
end for
21:
end for end while
22:
Hyperparmeters. We use the same hyperparameters for the generative model (i.e., Genetic GFN). Since active learning approaches introduce various hyperparameters, such as the training iterations of the proxy and generative models, not only introduce the proxy model, we tried keep the setup of GFlowNet-AL in the PMO benchmark (e.g., proxy learning rate). We provide hyperparameters in Table note that our proxy model predicts the score using SMILES rather than fragments, unlike the original GFlowNet-AL. Additionally, we adopt y, the ratio of offline inner loop training, and the acquisition function (related with k); see the original GFl paper [10].
data in
D.2 Experimental results
compare Genetic GFN-AL with various model-based and active learning methods. Here, BO regarded model-based method of Graph GA because GP BO uses Graph when optimizing the GP acquisition function. As shown Table 7, SMILES Genetic GFN-AL achieves significantly improved performance compared to fragment GelowNet-AL in the PMO benchmark. It is noteworthy that further utilize the acquisition function, not directly use the proxy prediction as reward, and mix the cracle-touched data, described in the previous section. Therefore, verify the effectiveness of genetic search the active learning setup, we implement another baseline, SMILES GEN-AL using
18

---

Table 6: Hyperparameters of Genetic GFN-AL
Fragment
SMILES Genetic GFN-AL
GFlowNet-AL
(PMO benchmark)
proxy init sample size proxy sample size proxy training iterations proxy training batch size proxy hidden dimension proxy layers generative model training iterations proxy learning rate
480
500
480
500
25
25
64
512
64
8 10
100
0.00025
0.00025
proxy weight decay
0.000001
0.000001
proxy dropout
0.0
0.0
kappa
0.1
0.0
gamma
0.5
0.0
random action prob
0.0
0.05
e-greedy exploration, similar our self-ablation studies in Section 5.1.2. The results demonstrate that Genetic GFN is beneficial to enhancing sample efficiency in the active learning setting. As pointed out the PMO benchmark, though the model-based methods (or active learning methods) are known as more sample efficient, they require careful design to achieve superior performances.
Table 7: Comparing AL and model-based algorithms. The bold text indicates the best value.
SMILES GFN-AL with e-greedy
Fragments
Genetic GFN-AI.
GP BO [12]
GFlowNet-AL (10)
AUC Top-1
13,997 + 0.337
11.954 = 0.190
13.718 + 0.080
11.032 : 0.016
AUC Top-10
13.462 + 0.303
11.257 = 0.171
13.115 :: 0.074
9.928 0.027
AUC Top-100
12.263 0.218
9.901 + 0.237
12,050 + 0.082
8.064 0.005
19

---

E Multi-objective sample efficient molecular optimization tasks
This section provides details on multi-objective sample efficient molecular optimization experiments. Even though Genetic GFN targets single (i.e., scalar-valued) objective optimization tasks, ours directly applies to multi-objective tasks using well-defined coefficients. Inspired by the work of Zhu et al. [11], we conduct experiments on two multi-objective tasks: GSK33 + JNK3 and GSK33 + JNK3 QED SA. Genetic GFN is implemented on top of the original code (MIT license)."
Interestingly. GSK3B and JNK3 are machine-learning-bene based oracles that estimate inhibiting scores against the Glycogen synthase kinase beta target and the c-Jun N-terminal kinase 3 target. Designing dual inhibitors for both targets can be beneficial to designing treatments for Alzheimer's 's Disease [50]. We define scalar-valued score functions using the linear combination of oracle functions with given coefficients. The cost coefficients are set following the HN-GFN paper, i.c., a = (1, 1) for GSK333 + JNK3 and a (3, 4, 2, 1) for GSK33 JNK3 + QED SA. Since the reward scales become larger, we reduce the inverse temperature from 50 to 25.
According to the experiment setup of hypernetwork-bassee based GFlowNets (HN-GFN) [11], we limit the reward calls to 1000. Consequently, the batch size, replay training loops, population, and offspring size adjusted to half. The performance measured using hypervolumes with five independent runs. Note that we use the same pretrained model in the main experiment for PMO.
20

---

F Designing of SAS-Cov-2 inhibitors
According to the previous work [52], we define the score function s(x) for designing SARS-Cov-2 inhibitors as a linear combination of normalized scores, i.e.,
+ 100.625( Kaocking (c)) + 0.1 SQED (x) + 0.1 10- (sA
(x) 0.8
(5)
9
The target proteins are follows:
PLPro PI::o_JJJR: PLPro (papain-like protease) is a critical enzyme in the life cycle of SARS- CoV-2, which can help in studying the enzyme's function and designing inhibitors that could potentially disrupt the virus's ability to replicate and evade the immune system. The 7JIR represents C111S mutant version of PLPro.
SdRp_6YYTI RdRp IRNA-dependent RNA polymerase) is essential for the replication of the genome and the transcription of genes in SARS-CoV-2. The protein structure of RdRp is cataloged in the Protein Data Bank (PDB) under the identification code 6YYT."
Genetic GFN is implemented on top of the implementation of MoIRL-MGPT (Molecular design using Reinforcement Learning with Multiple GPT agents). 10 We employ the same hyperparameters with the experiments on the PMO benchmark, except for the weight-shifting factor (we use k = 0.05).
In Table 5, the scores of baselines computed according the Eq. (5) using the average values in the MoIR--MGPT [52]. We also provide average scores and standard deviation of docking, QED, SA, and diversity in Table and Table 9.
Table 8: The results of Top- 100 molecules for PLPr_7JIK target. The docking, QED, and SA of baselines are directly from MoIRL-MGPT, and the total score is recalculated according to Eq. (5) using average values.
Score (1)
Ducking (L)
QED (t)
SA (L)
Diversity (t)
JT-VAE
0.272
-8.76 +
0.795 +:2 0.038
2,994
0.140
0.836 0.032
GHlowNet
0.326
-9.11 + 0.21
0.726 + 0.015
2,823 + 0.076
0.825 0.010
Graph GA
0.723
-10.83 0.08
0.380 + 0.013
3.638
0.162
0.740 + 0.017
Reinvent
0.717
-10,75 = 0.05
0.392
2.649
0.035
0.619 = 0.023
MoIRL-MGPT
0.772
-11.02 0.06
0.386 = 0.006
2.550 0.047
0.745 = 0.008
Genetic GFN (Ours)
0.908
-12.86 0.17
0.425 + 0.092
2.819 0.105
0.592 + 0.010
Table The results Top- 100 molecules for RdRp_6 target. The docking, QED, and SA of baselines are directly from MoIRL-MGPT, and the total score is recalculated according to Eq. (5) using average values.
Score (t)
Docking ())
QED (1)
SA (L)
Diversity (t)
JT-VAE
0.216
-8.33 + 0.25
0.719 + 0.019
2,959
0.094
0.828 0.018
GFlowNet
0.280
-8.89 0.16
0.656 + 0.033
2,854
0.061
0.770 = 0.015
GraphGA
0.786
-11.26 = 0.12
0.262 + 0.010
3.520
+ 0.049
0.658 +: 0.009
Reinvent
0.799
-11.30 0.04
0.275 1: 0.006
2,917 0.035
0.616 + 0.021
MGPTL-MGPT
0.854
-11.84 0.07
0.278 1 0.005
2,894 1 0.072
0.670 + 0.013
Genetic GFN (Ours)
0.890
-13.26
0.13
0.277 +: 0.076
3.624 0.060 0.708 + 0.010
We provide additional visual results Fig. 10. complexity and functional diversity over iterations.
There seems be trend of increasing molecular
http:://www..rccoom//mmmuuu////
9 http:://ww...ssooorppcnnnnnttttt Available at http://ghhhbcccn///iiiiiiMetaltt
21

---

popo00 anpu BorD Boog 8 Sant Bard Soarte thats
@dRp_6YY* Figure 10: Examples of Top3 inhibitors for SARS-CoV. over steps (seed 1)
(a) PLPro_7JIR
22

---

G
Additional results
G.1
Controllability of the score-diversity using the weight-shifting factor k
As mentioned in Section 5.1.2, the weight-shiffing factor k in Eq. (4) also can control the score- diversity trade-off. Inereasing & gives more diverse candidates by increasing the probability of high-ranked samples, Similar to adjusting the inverse temperature, the results in Table 10 and Fig. 11 demonstrate that adjusting k with fixed also can effectively control the score-diversity trade-off.
Genetic
Genetic GPN (K (=0.05)
Genatic
avg. Top
Genetic GPN (K 0.5)
SMILES REINVENT
GEGL
GP DO
16.0
0.70
Diversity
Figure 11: The average score and diversity with adjustments of
Table 10: The score-diversity trade-off by varying with fixed 3.
Oracle
k = 0.1
k 0.05
& 0.01
k: 0.005
AUC Top-1
15.246
15.801
16.527
16.323
AUC Top-10
14.652
15.330
16.213
16.040
AUC Top-100
13.597
14,453
15.516
15,418
Diversity
0.581
0.515
0,432
0.444
G.2 Additional results for the PMO benchmark
G.2.1 Statistical analysis
We provided the results of the t-tests of AUC Top-10 with Mol GA (2nd place) and SMIELS REINVENT (3rd place). As shown in Table 11, both p-values are less than 0.05, so Genetic GFN outperforms baselines with statistical significance.
Table 11: The results of t-tests with Mol GA and REINVENT.
2nd place
3rd place
SMILES REINVENT
Genetic GFN
Mol GA
Genetic GEN
Mean
16.213
15.685
16.213
15.185
Variance
0.038
0.033 5
0.038
0.194
Observations
5
5
Hypothesized Mean Diff.
Stat
4.426
4,780
PCT t) one-tail
0.001
0,002
Critical one-tail
1.860
2.015
P(T t) two-tail
0.002
0,005
Critical two-tail
2.306
2.571
23

---

G.2.2
Full results of the PMO benchmark
Table 12: Full results of Table 1.
Genetic GFN (Ours)
SMILES REINVENT
SELFIES REINVENT
Mol GA
GEGL.
GP BO
Graph GA
#1
0.949 =: 0.010
0.928 + 0.015
0.881 0.016
0.842
0.019
0.902 0.011
0.867 0.025
0.859
0.013
U2
0.761 + 0.019
0.740 + 0.055
0.644 0.019
0.626
0.018
0.579 0.35
0.621 0.015
0.657 0.022
#3
0.802 = 0.029
0.629 + 0.062
0,717 0.027
0.699
0.041
0.746 + 0.025
0.588 + 0.062
0.593 0.092
#4
0.733 = 0.109
0.656 + 0.013
0.662 = 0.044
0.656
0.039
0.615 + 0.009
0.638 + 0.016
0.602 0.012
#5
0.974 +: 0.006
0,950 1 0.004
0.957 0.007
0.898 1: 0.015
0,941 0.017
0.953 0.009
0.973 0.001
0.856 0.039
0,835 0.012
0.781 0.013
0.769
0.009
0.726
0.740
0.762
#7
0.881 + 0.042
0.894 +: 0.025
0.885 0.031
0.816 0.027
0.861 0.027
0.821 0.041
0.817 0.057
#8
0.969 + 0.003
0,926 + 0.014
0.942 = 0.012
0.930 = 0.011
0.883 + 0.040
0.873 + 0.041
0.949 0.020
#9
0.897 = 0.007
0.894 0.005
0.838 = 0.030
0.808 = 0.007
0.805 + 0.007
0.844 0.016
0.839 0.042
#10
0.764 + 0.069
0.835 + 0.040
0.782 = 0.029
0.580 = 0.086
0,611 0.000
0.624 0.048
0.652 0.106
#11
0.379
0.329 0.006
0.363 0.011
0.338 0.016
0.298 0.06
0.353
0.285
#12
0,294 0.007
0,284 + 0.035
0,281 0.002
0.274 0.007
0.296 0.11
0.252 + 0.010
0.255 0.019
#13
0.708 + 0.057
0.762 + 0,048
0,634 0.042
0.599 = 0.035
0.631 + 0.093
0.589 - 0.040
0.571 0.036
#14
0.860 + 0.008
0.853 1 0.005
0.834 = 0.010
0.832 = 0.005
0.788 + 0.005
0.819 + 0.005
0.813 0.006
#15
0.595 1. 0.014
0.610 1. 0.038
0.535 0.015
0.537 1 0.015
0.494
0.006
0.533 0.024
0,514 0.025
#16
0.942 0.000
0.001
0.000
0,941 0.001
0.937
0.940
0.937
H17
0.819 + 0.018
0,830 + 0.010
0.770 0.005
0.730 0.011
0.741 0.010
0.736 0.008
0.718 0.017
#18
0.615 = 0.100
0.568 + 0.017
0,551
0.024
0,531 = 0.010
0.535 0.007
0.521 + 0.014
0.513 0.026
#19
0.634 = 0.039
0.677 + 0.055
0.470 = 0.041
0.402 = 0.024
0.461 = 0.057
0.492 - 0.055
0.498 0.048
#20
0.583 + 0.034
0.544 + 0.067
0.544 0.026
0,515 1 0.028
0,544 0.038
0.497 0.043
0,483 0.034
#21
0.511
0.487 0.024
0.458
0.420
0.031
0.404
0.342
0.373
#22
0.135
0.271
0,000 + 0.000
0.182 0.363
0.119
0.238
0,000
0.000
0.000
0.000
0.000
#23
0.552 + 0,033
0,514 + 0.033
0,533
0.009
0,492 + 0.021
0,466 0.025
0.509
0.009
0.468
0.025
Sum
16.213
15.686
15.185
14.354
14,264
14.152
14.131
Div.
0.432
0.465
0.468
0.540
0.617
0.555
0.661
Table 13: Full results of Table I (continued).
SMILES LSTM-HC
Fragment GFN
Fragment GFN-AL
STONED
SynNet
SMEILS GA
#1
0.731
0.008
0.765 + 0.048
0.568 + 0.033
0.649 0.079
0.382 0.010
0.459 + 0.028
#2
0.598 0.021
0.608 + 0.020
0,566 1 0.006
0.520 1: 0.017
0.428 4: 0.002
0.437 : 0.007
#3
0.552 = 0.014
0.378 = 0.043
0.439 + 0.035
0.361 + 0.038
0.263 0.009
0.326 = 0.008
#4
0.837 + 0.018
0.612 + 0.007
0.635 +: 0.043
0.612 + 0.005
0.582 +: 0.001
0.587 + 0.002
#5
0.941 1 0.005
0.935 1 0.014
0.970 1: 0.006
0,958 0.015
0.480 0.075
0.601 1: 0.055
46
0.733 0.002
0.791 = 0.014
+ 0.750
0.705
0.689 0.000
0.700 = 0.005
47
0.846 0.019
0.666 + 0.022
0.713 + 0.057
0.714 0.038
0.589 + 0.009
0.666 + 0.006
48
0.830 0.019
0.930 + 0.012
0,862 + 0.004
0.821 + 0.070
0.791 +
0.468 + 0.211
H9
0.693 0.018
0.897 1:200000000 0.032
0.657 0.000
0,853 0.049
0.576 0.021
0.199 +::00 0.199
#10
0.670 - 0.014
0.509 0.065
0.574
0.353 0.061
0.359 0.009
0.442 = 0.017
#11
0.263
0.007
0.264 1 0.032
0.236 1: 0.015
0.187 0.029
0.192 0.003
0.207 1:
#12
0.249
0.003
0.254 1 0.024
0.241 1 0.00
0.178 0.009
0.174 0.002
0.181 1 0.002
#13
0.553
0.040
0.620 + 0.098
0.402 + 0.017
0.419 + 0.028
0.291 0.005
0.332 + 0.012
#14
0.801
0.002
0.829 + 0.012
0.793 + 0.008
0.820 + 0.021
0.787 0.002
0.785 + 0.003
#15
0.491
0.006
0.484 + 0.016
0.541 + 0.021
0.442 0.016
0.423 0.006
0.434 + 0.006
#16
0.939
0.001
0.942 = 0,000
0,941 + 0.001
0.941 0.002
0.904 0.002
0.917 + 0.002
#17
0.728 + 0.005
0.764 + 0.023
0.749 + 0.009
0.723 + 0.023
0.626 + 0.005
0.660 + 0.004
#18
0.529 1 0.004
0.515 1 0.025
0,506 1 0.012
0,507 0.008
0.461 0.002
0,464 1 0.003
#19
0.309 1 0.015
0.600 1 0.103
0.297 0.033
0.449 1.066
0.180 1.002
0.217 1 0.022
#20
0.440 + 0.014
0.375 + 0.029
0.397 + 0.012
0.310 0.019
0.261 0.004
0.292 + 0.009
#21
0.369
0.015
0.309 + 0.080
0.280 + 0.006
0.262 + 0.019
0.183 + 0.001
0.190 + 0.002
#22
0.011
0.021
0.000 + 0.000
0.000 + 0.000
0.000 + 0.000
0.000) +: 0.000
0.000 + 0.000
423
0.470 0.004
0.484 = 0.015
0.493 + 0.014
0.029 0.470
0.308 + 0.027
0.353 + 0.024
Sum
13.583
13,531
12.610
12.254
9,929
9.917
Div.
0.686
0.498
0.728
0.596
0.816
0.846
24

---

Graph
SMLESHI
Syntet
albuterci similarity
amiodiphne mpo
celecorbb reciscoverr
deco hop
0.0
0.9
0.6
0.4
0.4
0.6
0.5
2200
4:00
6000
8000
10000
2020
4000
5000
1000
19003
2000
4300
6:00
8000
10000
2003
4030
2000
1000
12000
1.0
0.0
0.9
0.0
0.6
0.6
0.6
0.4
0.6
0.2
2000 4:00 6000
10000
2030
1050
1000
1000
10003
2000
4200
6:00
8o0n
10000
2002 40:00 ED00
5000 12000
1comeer 20 plcc
JN43
median
1.0
1.0
0.35
0.8
0.6
0.5
0.4
0.2
0.2
0.0
6200 mestranol similntty
2000
13000
osimeronio mpo
Derindopri, mpo
0.0
0.0
0.7
0.6
0.6
0.5
0.90
0.75
0.4
0.8
0.70
2200
4:00
5:00 8100 10000
2036
4050
5000
8000
1000)
2000
4200
6:00
8:00
10000
2000
4000
50:0
5000 1200
rediscovery
0.8
0.6
0.6
0.6
0.4
8.4
0.4
0.5
0.3
0.0
0.2
trogitazone tediscovery
Valsarfan smaarts
zaleplon mpp
0.6
1.0
0.6
0.A
0.5
0.6
0.2
0.
0.3
02
2200
4200
6000
8000
10000
2000
4000
5000
5000
10003
2000
4200
5200
8000
20000
0.5
0.2
0.4
0.8
10
Figure 12: The optimization curves for 23 oracle score functions.
25

---

G.2.3 Ranks with various metrics
According to the PMO benchmark, we also provide the rank of each method with various metrics. The results Table 14 show that Genetic GFN achieves the first place in total, not only in the AUC Top-10.
Table 14: The ranks of 10 methods based various performance metrics
AUC
Average
Mean
Method
Category
Top-1
Top-10
Top-100
Top-1
Top-10
Top-100
Genetic GFN (Ours)
Off-policy
Mal [16]
GA
2.67
On-policy
4
3.00
GEGL [17]
Off Off-policy
4
4
4
4.17
SELFIES-REINNE REINVENI [3]
On-policy
5.67
[12]
Active Leaming
[21]
Off-policy
6.17
Graph GA [13]
GA
7
7.67
STONED [14]
GA
9
8
8.83
SynNet [36]
GA
10
10
II
10
10
11
10.33
SMILES GA [21]
10.50
GelowNet
Off-policy
12
GAlowNetAL [10]
Active Learning
13
12.67
G.3 Further studies on valsartan_smen smarts (#22)
valsartan seartann_smates (#22) in Table 1. The valsartan SMARTS targets molecules containing SMARTS pattern related to valsartan while being characterized by physicochemical properties corresponding the sitagliptin molecule [57]. It measures the arithmetic means of several scores, including (1) binary score about whether contains certain SMARTS structure, (2) LogP. (3) TPSA, and (4) Bertz score. Since utilize a TDC oracle function for evaluations, we provide our empirical observations here.
Notably, have observed that only few methods achieve non-zero
Due to the binary score (I if the certain SMARTS pattern is included), many tries terminate with Especially with a limited number oracle calls, generating molecules containing a certain sub- structure is notoriously hard. Other literature shows that other methods achieve high scores with oracle calls [52]. With 10K calls, even REINVENT and Genetic only succeed finding non-zero out of five independent runs. Another observation is that methods (REINVENT, Genetic GFN, and GEGL) achieving non-zero scores all generate SMILES with RNN- models. Thus, have a conjecture that SMILES generation is effective generating a certain SMARTS pattern. We provide examples of generated molecules with non-zero valsartan_ serrssmmtttttee scores. Note that the other four seeds failed. Each run generates similar molecules (see Top1, 100 samples in Fig. 13 in the additional material), but the samples between the two runs (REINVENT and Genetic GFN) have different structures (the molecule distance between Topl samples 0.854).
molecules
Topl:
Tople:
Top108:
Topl: 0.987
Topl8: 0.981
Topl8e:
(a) REINVENT (seed 0)
(b) Genetic GFN (seed 4)
Figure 13: Examples of molecules with non-zero scores on valsartan__mart (#22)
26

---

G.4
Additional results for ablation studies
G.4.1 Statistical analysis
We provided the results of the t-tests of ablation studies. As shown in Table 15, the p-values for ablating genetic search are less than 0.05, our genetic search is statistically significant component.
Table 15: The results of t-tests of ablation studies.
Genetic Search
KL--divergecce penalty
Genetic GFN
{GS)
Genetic GFN
(GS) (e-greedy)
Genetic GFN
(KL)
Mean
16.213
15.738
16.213
15.627
16.213
15.928
Variance
0.038
0.094
0.038
0.008
0.038
0.227
Observations
5
5
5
5
Hypothesized Mean Diff.
0
0
Stat
2.931
6.109
1.236
P(T t) one-tail
0.011
0.000
0.136
Critical one-tail
1.895
1.943
2.015
P(T t) two-tail
0.021
0.001
0.271
Critical two-tail
2.365
2,447
2.571
G.5 Results for diversity and synthesizability
We report the diversity and synthetic accessibility (SA) score of 1op-100 molecules on each oracle.
Diversity of Top 100
0.9
0.8
SynNet
0.6
SHILES HC
0.5
Grapn
SEIVES-RRIVENT
Gr60
0.3
GA
02
Baatter di dn coue corvenr hop
drdt get secoorre
Auneerr Sommer Dh gereer
ane sesiine
seconn
Figure 14: The average diversity of Top- 1op-100 molecules (t)
Avg. SA Score
AL
SMIES 54
SynNet-
STONED
SMILES HC
SMLES-RERVENT
Mai
greenn ao potteer hop
diit
Atr mocus
ao patipptan
onder seddeee
rithu secrrontos
jose peeuus
Figure 15: The average SA score of Top-100 molecules ())
27

---

G.6 Genetic GEN with SELFIES generation
As our method employs string string-based sequence generation model, applicable SELFIES representation. Following the same procedure and setup described in Section 3 and Section 5.1. The results in Table 16 demonstrate that Genetic GFN significantly outperforms by achieving 14.986 compared to 14.152, not only the other high-ranked method, GP BO (14.264). Notably, our AUC Top- is higher than that of SELFFIES-REIVENNE in oracles.
Table 16: Performance of Genetic GFN with SELFIES generations.
SELFIES REINVENT
SELFIES
Oracle
Genetic GFN
albuterol sitiiarr_ssmmllaitty
0.867 + 0.025
0.918 + 0.031
amlodipine mpo
0.621 + 0.015
0.711 + 0.029
pelecoxibrreiscoov
0.588 = 0.062
0
deco_ deco_hop hhp drd2
0.638 0.016
0.631 + 0.020
0.953 0.009
0.971 0.005
fexofenadine_myr mpo gsk3b
0.740 0.012
0.797 + 0.012
0.821 0.041
0.900 + 0.042
isomers noomers_chh82222
0.873 +: 0.041
0.952 + 0.017
coomers_c911022222p1
0.844 + 0.016
0.879 + 0.031
jnk3
0.624 0.048
0.675 1 0.140
medianl
0.353 0.006
0.351 + 0.034
median2
0.252 0.010
0.263 0.014
mestranol mestranol_similartty
0.589 0.040
0.680 0.076
osimertinih mpo
0.819 0.005
0.849 + 0.008
perindopril_mpo
0.533 0.024
0.551 + 0.015
qed
0,940 0.000
0.942 + 0.000
ranolazine_mpo scaffold hop sitagliptin_mp
0.736 + 0.008
0.785 + 0.013
0.521 1: 0.014
0.531 + 0.020
0.492 = 0.055
0,590
thiothixene_redisee
rediscovery
0.497 0.043
0.527 + 0.036 0.036
trogltaaooe_rrddisovvryy
0.342 0.022
0.087
valsartan smarts
0.000 0.000
zaleplon_mpo
0,509 0.009
0.518 + 0.016
Sum Diversity
14.152
14.986
0.555
0.528
G.7 Genetic GFN with string-based genetic search
We also additionally provide experiments that incorporate STONED (GA with SELFIES) as an exploration strategy guide GFN training instead of Graph GA. Note that STONED only utilizes mutations since designing valid crossover with string representation challenging.
Table 17: Results with different genetic search algorithms
Genetic GFN with STONED
Genetic GFN
AUC Top-1
16.527 + 0.043
15.806 + 0.037
AUC Top-10
16.213 + 0.042
15.439 + 0.037
AUC Top-100
15.516 + 0.041
14.870 + 0.036
28

---

G.8 Experiments with varying hyperparameterss
In this subsection, we verify the robustness Genetic GFN for differing hyperparameter setups. We conduct experiments by varying the number of GA generation (refining loops), offspring size, and the number of training inner loops. The results show that our results are robust each hyperparameter setup by achieving similar or better performance compared to Mol GA in all tested configurations.
Table 18: Ablation studies for the number of GA generation. 'x0' stands for the results of TB loss training without GA explorations.
Oracle
x0
x
x2
x3
AUC Top-1
16.070
15.968
16.527
16,040
AUC Top-10
15.738
15.615
16.213
15.735
AUC Top-100
15,030
14,909
15.516
15.074
Diversity
0,479
0.470
0.432
0.440
Table 19: Results by varying the offspring size.
Oracle
16
32
AUC Top-1
16.174
16.527
15.984
15.963
AUC Top-10
15.846
16.213
15.669
15.621
Top-100
15.175
15.516
14.977
14858
Diversity
0,458
0.432
0.452
0.437
Table 20: Results by varying the number of training inner loops.
Oracle
x16
AUC Top-1
16.125
16.527
16.049
AUC Top-10
15.768
16.213
15.717
AUC Top-100
15.175
15.516
14.977
Diversity
0.432
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state our main research claim, which consistent with the experimental results.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include our limitations and future works in the discussion section.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide implementation details, including pseudo-codes.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the anonymized link of our implementation. Also, the paper uses the public dataset and benchmark; we provide accessible links in the supplemental material.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all details regarding experiments in the appendix and the anonymized codes.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
The experimental results include standard deviation with independent multiple runs. addition, statistical tests are conducted for the main results and self-ablation studies.
Justification:
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
30

---

Justification: We provide the computer resources in the appendix (due to lack of spaces),
9. Code Of Ethies
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpa:
Answer: [Yes]
Justification: We have checked the paper according to the NeurIPS Code of Ethics; our study does not include human participation and privacy-related data.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader impact of this study at the end of the paper.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our study is conducted using the public dataset, and all experimental tasks from previous publications.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper clearly states the source and license of the original code, data, and models in the appedix.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our experiments are conducted using the already published datasets and benchmarks.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our study does not involve crowdsourcing nor human subjects.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our study does not involve crowdsourcing nor human subjects.
31


Reviews:
[{"review_id": "tzmEx3OMcs", "reviewer": "Reviewer_FCer", "summary": "This work proposes a variant of GFlowNet, called genetic GFN, for molecular property optimization. Specifically, the authors use genetic search to guide the GFlowNet to explore high-reward regions, addressing the over-exploration issue in GFlowNet. Besides, the authors incorporate some effective training strategies to improve the performance further. The proposed genetic GFN achieves achieves state-of-the-art performance in practical molecular optimization (PMO) benchmark.", "questions": "See weaknesses", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper studies an important scientific problem, i.e., molecular optimization.\n- The paper is well-organized and easy to follow.\n- The experimental results are comprehensive and strong.", "weaknesses": "- The novelty is somewhat limited, as the proposed Genetic GFN simply combines some existing techniques, that have been studied independently.\n- In Table 4, Genetic GFN is worse than Graph GA and MARS on GSK3 + JNK3. The reason should be discussed.\n- I am more curious about the difference between increasing KL term in loss function and directly using the logP of the reference distribution as part of the reward (Please see equation (4) in DPO [1]). In this situation, what's the different between GFlowNet and PPO.\n\n[1] Direct preference optimization: Your language model is secretly a reward model", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Thanks for the valuable comments.*** We address concerns as follows.\n\n> The novelty is somewhat limited, as the proposed Genetic GFN simply combines some existing techniques, that have been studied independently.\n\nThis method is novel because it is **the first to combine 1D sequence generation using GFlowNets with 2D molecular graph search via genetic algorithms**. This approach allows the policy to generate 1D sequences, which are easier to train, while utilizing a genetic algorithm to explore 2D molecular graphs, reaching regions that might be inaccessible to the 1D policy alone. Thanks to the off-policy nature of GFlowNets, we can leverage the insights from the genetic algorithm's 2D molecular graph search to enhance the training of the 1D sequence policy. Our experimental results empirically support these; please see Tables 2 and 3.\n\nCombining existing methods in innovative ways is both important and novel, as it creates synergies that significantly enhance performance and efficiency, achieving breakthroughs that neither method could accomplish independently.\n\n> In Table 4, Genetic GFN is worse than Graph GA and MARS on GSK3 + JNK3. The reason should be discussed.\n\n|  | GSK3b + JNK3 | GSK3b + JNK3 + QED + SA |\n| -------- | -------- | -------- |\n| Graph GA | **0.368 ± 0.020** |  0.335 ± 0.021 |\n| MARS | **0.418 ± 0.095** |  0.273 ± 0.020 |\n| HN-GFN | 0.669 ± 0.061 | 0.416 ± 0.023 |\n| Genetic GFN | 0.718 ± 0.138 | 0.642 ± 0.053 |\n\nThanks for pointing out. The results are directly brought from the work of Zhu et al. (2023) [1], and there were some mistakes in the number of Graph GA and MARS on the GSK3b + JNK3 task. The provided table here shows the correct numbers. Please also refer to Fig. 3 in the HN-GFN paper (https://arxiv.org/pdf/2302.04040).\n\n[1] Zhu, Yiheng, et al. \"Sample-efficient multi-objective molecular optimization with gflownets.\" Advances in Neural Information Processing Systems 36 (2024).\n(The numbers are from https://openreview.net/forum?id=ztgT8Iok130)\n\n> I am more curious about the difference between increasing KL term in loss function and directly using the logP of the reference distribution as part of the reward (Please see equation (4) in DPO [1]). In this situation, what's the different between GFlowNet and PPO.\n\n\nThanks for the engaging discussion on this important topic, which has broad relevance across many domains: KL constraint optimization (e.g., RLHF).\n\nThere are indeed several methods to incorporate KL constraints during optimization. One effective approach is to include a logP prior within the trajectory balance loss, treating it as a soft reward as you suggested. The following table shows both achieving outperforming performance (AUC10). Note that the results are obtained from three independent runs and our hyperparameters have been searched with Genetic GFN (KL).\n\n| Genetic GFN (KL) | Genetic GFN (logP prior) | \n|  --- | --- |\n| 16.088 ± 0.025 | 15.777 ± 0.018 |\n\nWhen using explicit KL loss terms, it is important to note the differences between PPO and GFlowNets. PPO aims for reward maximization, whereas GFlowNets aim for reward matching, generating samples proportional to the reward. Even with KL constraints, PPO will seek a unimodal maximum reward within the KL constraint region, while GFlowNets will sample diverse, high-reward modes within the KL constraint region.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R8nHEKVhUv", "reviewer": "Reviewer_kTKB", "summary": "This paper proposed a combination of genetic algorithm (GA) and GFlowNets for molecular optimization, with an emphasis on sample efficiency (achieving high property value by few number of reward calls). The key motivation is that GA can incorporate domain specific knowledge by designing the mutation operations, which is the key to improve sample efficiency, while GFlowNets model the overall distribution of the molecular space in a data-driven manner. The pipeline thus consists of two components: a SMILES-based generative model using GFlowNets and a graph-based genetic search. Several techniques for training this pipeline, including unsupervised pretraining, experience buffer for off-policy training, and TB + KL loss functions, are proposed. Experimental results on PMO benchmark and in silico design for SARS-CoV-2 inhibitors are conducted to demonstrate the effectiveness of the proposed method.", "questions": "- line 55, please remove \"expectional\"\n- line 118, no references to \"previous works\"", "limitations": "See weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is simple and easy to follow. Though there is a small gap between the motivation and the proposed method (see Weakness), overall I buy the story of combining domain-specific knowledge with pure data-driven methods; the proposed combination of GA and GFlowNets makes sense to me.\n\n- The authors clearly have put a lot of efforts on conducting and designing the experiments, for both the main results and the ablation studies. The proposed study on designing inhibitors against SARS-CoV-2 looks interesting to me. Though it is hard for me to judge whether the in silico score functions indeed have a strong correlation with real-world performance in the biological sense, including such efforts/results is already something intriguing from a machine learning perspective.\n\n- The paper is in a good shape. The authors did a good job on describing necessary background knowledge for GFlowNet. Sufficient technical details are provided in the paper.", "weaknesses": "- One weakness is the gap between \"domain-specific knowledge\" and \"GA method\" mentioned in the introduction. When I read the introduction, I am expecting the authors will propose some new genetic operators that are well-designed and specific to domain-specific tasks, with clear indicator that these operators have strong correlation with domain-specific knowledge. It turns out that the authors are still using the standard genetic operators such as crossover and mutation. Are there any special care taken that I may have missed? for example, the crossover fragments are not purely random but are collected from motifs specific to tasks. If no, I feel the statement of \"GA method can encode domain-specific knowledge\" is very vague, or at least it not being discussed comprehensively if previous works have explored. \n\n- Another thing missing in the paper is enough qualitative results. I can only see Fig 4 & 5 contain some final generated molecules for two targets. It will be great if the authors can include more visual results, especially the trajectory of the sampled molecules, with highlights on what fragments have been changed during the process. Some analysis on why certain fragments are favored (if any) or remain at the final optimized structure is also very helpful for the readers to verify the effectiveness of the proposed method.\n\n- Another interesting study to show is how the proposed pipeline can be incorporated with grammar-based representations, such as STONED, SELFIES, and many more if search \"molecular grammar\", rather than SMILES. Since these manually designed grammar consists of more \"domain-specific knowledge\" compared to pure atom-based SMILES string, I would expect the experimental results will be better. It would be great to include such analysis in the paper to provide more evidence on the key motivation of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Thanks for the valuable comments.*** We address concerns and questions as follows.\n\n> One weakness is the gap between \"domain-specific knowledge\" and \"GA method\" mentioned in the introduction. When I read the introduction, I am expecting the authors will propose some new genetic operators... It turns out that the authors are still using the standard genetic operators such as crossover and mutation. \n\nWe identify two potential but distinct approaches for integrating genetic algorithms with deep learning: (1) automating genetic algorithms (GA) using deep learning and (2) incorporating domain-specific GA's search capabilities into deep learning as an inductive bias. Our research focuses on the latter, leveraging powerful GAs designed based on chemical domain knowledge to enhance deep neural network policies for more sample-efficient molecular optimization strategies. We also acknowledge that the first approach, automating genetic algorithms with deep learning, could be a valuable direction for future research. We will revise our manuscript to articulate these points clearly.\n\n> Are there any special care taken that I may have missed? for example, the crossover fragments are not purely random but are collected from motifs specific to tasks. If no, I feel the statement of \"GA method can encode domain-specific knowledge\" is very vague, or at least it not being discussed comprehensively if previous works have explored.\n\nCrossover and mutation operations are conducted according to predefined patterns. For instance, when altering bond order (one type of mutation), the possible transformations are specified to ensure valid changes that adhere to chemical rules. Defining these valid operations requires domain knowledge and careful design of the logic.\n\nGraph GA employs two crossover operations and seven mutation operations based on SMARTS patterns (e.g., inserting an atom into a double bond: `[*;!H0:1]~[*:2]>>[*:1]=X-[*:2]`). Creating and utilizing these patterns requires expertise in chemistry to ensure accurate representation and manipulation of molecular structures. Additionally, validation and sanitization steps ensure that only chemically plausible and stable molecules are considered. Further details about genetic operations can be found in the original Graph GA paper.\n\nTo make our manuscript self-contained, **we will include a detailed explanation of how the crossover and mutation operations are designed** to guarantee molecular validity in the Appendix, accompanied by **Fig.2 in our supplementary material**.\n\n> Another thing missing in the paper is enough qualitative results. I can only see Fig 4 & 5 contain some final generated molecules for two targets. It will be great if the authors can include more visual results, especially the trajectory of the sampled molecules, with highlights on what fragments have been changed during the process. \n\nWe provide **additional visual results in Fig. 4 of our supplementary material**. Due to space constraints, only the top three samples for 50, 100, 500, and 1000 steps are reported.\n\nOur observations from the generated candidates for both targets are as follows.\n- Many molecules include heterocyclic rings, which contain nitrogen or other non-carbon atoms. These structures may play a role in the molecules' interactions with the target protein.\n- Benzene rings with various substituents (e.g., methyl, hydroxyl) are frequently observed. These substitutions could provide diverse interaction points with the target protein, although their exact contribution to binding affinity needs further investigation.\n- There seems to be a trend of increasing molecular complexity and functional diversity over iterations. For example, at step 100, more complex substituents on aromatic rings are introduced compared to the generated candidates at step 50. After 1000 steps, we observe the addition of bulkier groups, such as tert-butyl and sulfone groups.\n\nWe plan to provide more visual results in the revised manuscript.\n\n> Another interesting study to show is how the proposed pipeline can be incorporated with grammar-based representations, such as STONED, SELFIES, and many more if search \"molecular grammar\", rather than SMILES. .... It would be great to include such analysis in the paper to provide more evidence on the key motivation of the paper.\n\nWe have included the results of Genetic GFN with SELFIES in Appendix F.5. As pointed out in the work of Gao et al. [1], despite of clear benefits of SELFIES, SMILES often shows competitive and better performances in sample efficient molecular optimization tasks. For instance, SMILES-REINVENT outperforms SELFIES-REINVENT, and SMILES-LSTM-HC(hill climbing) outperforms SELFIES-LSTM-HC; please see Table 3 and the analysis in Section 3.2 in the work of Gao et al. [1].\n\nWe also additionally provide experiments that incorporating STONED (GA with SELFIES) as an exploration strategy to guide GFN training instead of Graph GA. Note that STONED only utilize mutations (designing valid crossover with string representation is difficult). \n\n|  | AUC1 | AUC10 | AUC100 |\n| ---- | ---- | ---- | ---- |\n| Genetic GFN | 16.527 ± 0.043 | 16.213 ± 0.042 | 15.516 ± 0.041 |\n| Genetic GFN (STONED) |15.806 ± 0.037 | 15.439 ± 0.037 | 14.870 ± 0.036 |\n| Mol GA | 16.001 ± 0.027 |15.686 ± 0.025 | 15.021 ± 0.025 |\n| SMILES-REINVENT | 15.686 ± 0.035 | 15.185 ± 0.035 | 14.306 ± 0.033 |\n\n> line 55, please remove \"expectional\"\n\nThanks for the suggestion. we removed the term in the revised manuscript.\n\n> line 118, no references to \"previous works\"\n\nThank you for bringing this to our attention. The \"previous works\" indicate the string generation approaches, which are usually high-ranked in the benchmark, like REINVENT, SMILES-LSTM hill climbing, and GEGL. We added these references in the revised version.\n\n### References\n[1] Gao et al. \"Sample efficiency matters: a benchmark for practical molecular optimization.\" Advances in neural information processing systems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RZ9mDiCiyL", "reviewer": "Reviewer_gNyQ", "summary": "This paper presents a novel approach called Genetic-guided GFlowNets (Genetic GFN) for sample-efficient molecular optimization. The method integrates domain-specific genetic algorithms to guide a GFlowNet policy toward higher-reward molecular samples.", "questions": "- Is the GA implemented on CPU or GPU? If it’s on CPU, how slow it is?\n- Would the model's performance improve if combined with modern RNN architectures like Mamba or xLSTM?\n- How sensitive is the graph genetic algorithm to the way molecules are fragmented and the resolution of fragmentation?\n- How might this approach be adapted to other discrete optimization problems beyond molecular design? (such as TSP in combinatorial optimization.  A quick intuition would be sufficient)\n- How does the method's performance scale with the size and complexity of the molecular space being explored?\n- Is there potential for incorporating human feedback or preferences into the optimization process?", "limitations": "See above.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is very pedagogical and easy to read, clearly explaining the proposed method and its rationale.\n- Extensive experiments demonstrate the effectiveness of Genetic GFN, showing state-of-the-art performance on benchmark tasks.\n- The approach offers a promising direction for enhancing sample efficiency in molecular design.\n- The method shows promising results in designing SARS-CoV-2 inhibitors, demonstrating potential real-world impact.\n- The approach offers controllability of the score-diversity trade-off, which is valuable for practical applications.", "weaknesses": "- The proposed method is limited to molecular optimization and is not readily generalizable to other domains.\n- While the graph GA is based on prior work, the paper would benefit from being more self-contained by including an algorithm or visualization of the mutation and crossover steps. For example, how does the GA ensures the validity of the molecules.\n- A schematic diagram illustrating the entire training process, including both pretraining and GFlowNet training, would improve clarity.\n- The anonymous link to code provided is not accessible, limiting reproducibility.\n- Lack of efficiency/runtime analysis and comparison with the baseline methods", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Thanks for the valuable comments.***\n\n> The proposed method is limited to molecular optimization and is not readily generalizable to other domains.\n\nOur method focuses on integrating strong domain-specific search heuristics into deep neural network policies using the off-policy nature of GFlowNets for sample-efficient optimization. This approach is adaptable to any task where a powerful domain-specific search heuristic is available. For example, in jailbreaking tasks on LLMs, one of the state-of-the-art methods is a genetic algorithm [1]; we could use this to train GFlowNets for jailbreaking policies. We indeed agree that the potential of developing automated genetic algorithm methods that can be applied across general domains, leveraging deep learning, as a promising direction for future work.\n\n[1] Liu et al. \"Autodan: Generating stealthy jailbreak prompts on aligned large language models.\" arXiv preprint arXiv:2310.04451 (2023).\n\n> the paper would benefit from being more self-contained by including an algorithm or visualization of the mutation and crossover steps.\n\nThanks for the helpful suggestion. We will include the new figure, provided in the attached additional material (Fig. 2), and more detailed explanation of Graph GA. \n\n> A schematic diagram illustrating the entire training process would improve clarity.\n\nThanks for helpful suggestion. We will include the new diagram, which is **provided in the attached additional material (Fig.1)**.\n\n> The anonymous link to code provided is not accessible, limiting reproducibility.\n\nWe apologize for the inconvenience caused. There was an error in the link; please refer to this link (https://anonymous.4open.science/r/genetic_gfn).\n\n> Lack of efficiency/runtime analysis and comparison with the baseline methods\n\nIn sample efficient molecular optimization, the main computational bottleneck is evaluating Oracle functions, so it is common to compare efficiency based on the number of samples. Also, note that the main metric, the area under the curve (AUC), is defined to measure sample efficiency (the higher AUC score means the higher sample efficiency).\n\nOur average runtimes are as follows. Though we tested all algorithms using similar computational environments, we did not rigorously control the computation resource.\n\n| | Avg (sec) | \n| --- | --- | \n| Genetic GFN | 827.50 | \n| SMILES-REINVENT | 252.88 | \n| Mol GA | 9803.26 | \n| Graph GA | 165.00 | \n| GPBO | 1519.65 | \n\n- The runtime of Mol GA is significantly longer than Graph GA, despite both utilizing the same crossover and mutation operations. This difference arises because Mol GA has a much smaller offspring size, and we observed a tendency to generate the already discovered molecules repeatedly (i.e., early convergence before reaching the maximum number of calls).\n- Compared to REINVENT, our runtime is increased, but it is not significantly longer than other baselines.\n- Note that the methods have different early termination rules, complicating direct comparisons.\n\n> Is the GA implemented on CPU or GPU? If it’s on CPU, how slow it is?\n\nGA is implemented on the CPU (we adopt the implementation of Graph GA without parallel computation). As shown in the table above, Graph GA has a relatively short runtime. The runtime increase compared to REINVENT comes from replay training, which is roughly twice longer than genetic search time.\n\n> Would the model's performance improve if combined with modern RNN architectures like Mamba or xLSTM?\n\nThank you for the suggestion. We believe that utilizing modern RNN architectures like Mamba or xLSTM can indeed enhance performance. Recent work [2] has demonstrated that Mamba outperforms vanilla RNNs in their methodology. \n\n[2] Guo & Schwaller (2024). Saturn: Sample-efficient Generative Molecular Design using Memory Manipulation. arXiv preprint arXiv:2405.17066.\n\n> How sensitive is the graph genetic algorithm to the way molecules are fragmented and the resolution of fragmentation?\n\nFirst of all, we adopt the original implementation of Graph GA. The Graph GA crossover divides each parent molecule into two fragments, either by cutting within a ring or arbitrarily along non-ring edges. Mutations are applied mostly at the atom level. \n\n> How might this approach be adapted to other discrete optimization problems beyond molecular design? \n\nWe can utilize constructive RL policy, like AM [3], which sequentially adds elements into partial solution to complete solution. Notably, there is a work that trains AM with GFN [4]. To guide GFN training, we can utilize domain-inspired genetic algorithms, such as edge assembly crossover [5] or hybrid genetic search [6].\n\n[3] Kool et al. (2019). Attention, learn to solve routing problems! ICLR\n[4] Kim et al. (2023). Symmetric Replay Training: Enhancing Sample Efficiency in Deep Reinforcement Learning for Combinatorial Optimization. ICML\n[5] Nagata & Kobayashi, (2013). A powerful genetic algorithm using edge assembly crossover for the traveling salesman problem. INFORMS Journal on Computing\n[6] Vidal et al. (2012). A hybrid genetic algorithm for multidepot and periodic vehicle routing problems. Operations Research\n\n> How does the method's performance scale with the size and complexity of the molecular space being explored?\n\nThe maximum SMILES length is set to 140, consistent with REINVENT, corresponding to approximately 70-100 atoms. For JNK3 (#10), consisting of relatively large molecules, SMILES lengths range from 50 to 130, with 35 to 100 atoms. In contrast, isomers_c7h8n2o2 (#8) typically contain about 10-15 atoms.\n\n> Is there potential for incorporating human feedback or preferences into the optimization process?\n\nOur method follows the (unsupervised) pretraining and fine-tuning framework, similar to approaches like RL with human feedback (RLHF) and direct preference optimization. One possible approach is incorporating the reward model used in RLHF as our oracle function and fine-tuning the policy with Genetic GFN.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PSqMT7s1n5", "reviewer": "Reviewer_4tbn", "summary": "In this paper, the authors present a sample-efficient molecular optimization method using GFlowNets and a genetic algorithm. The authors demonstrate its effectiveness in generating inhibitors against SARS-CoV-2, with fewer reward calls than other methods, as well as on the PMO benchmark, where the method outperforms all other methods on the 23 tasks (at least on top-1, -10, and -100 AUC). The paper is strong, novel, and well-written, with also very nice figures which are also well-annotated. Sadly, no code provided.", "questions": "* In Table 1, the authors demonstrate that their approach outperforms MolGA and REINVENT on most tasks from the PMO benchmark when using the AUC top-10 metric. Do these results still hold for the AUC top-1 (Table 14 suggests they do), and if so, is it by similar amounts or are the gaps wider between the top 3 models?\n* How does the model scale with the number of atoms? For instance, can it scale well to other modalities or macromolecules?\n* It is interesting to see that for task #22 (valsartan_smarts), only the genetic GFN and REINVENT perform somewhat well on this. Do the authors have any idea why the genetic GFN and REINVENT are the only models which do not completely fail on this task? I think it would be interesting to dig more deeply into some of the specific oracles/tasks and look at the molecules being generated by the genetic GFN and REINVENT - for instance, do they find the same solutions here?", "limitations": "* Despite the excellent results, the authors fail to provide any accompanying code, which is a shame (the link in the paper points to a non-existing anonymized repo). The lack of openly-shared code casts doubt on the performance of the model and some of the claims made in the paper, hence the reason for the low score. Hard to say how well-documented the code is or how useful it will be to researchers.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* A molecular optimization method that actually computes the sample efficiency - well done! And great results outperforming SOTA with a well-motivated strategy.\n* The authors postulate that genetic algorithms can more effectively navigate the chemical space through domain-specific genetic operations, something which deep generative models generally lack. As such, their approach leverages the ability of GFlowNets to generate diverse molecules, while instead leveraging the optimization power of genetic algorithms. Neat idea.\n* The ability for users to control the score-diversity trade-off via the inverse temperature parameter is very useful. Nonetheless, I believe that some of the other methods shown in Figure 3 also have that feature (e.g., REINVENT4 also has a controllable temperature parameter), which many be misleading in the current text/plot that suggests only the genetic GFN has it.", "weaknesses": "* How sensitive is the genetic GFN to the hyperparameters? I saw that many defaults were chosen to compare directly to REINVENT, did these also happen to be good options for GGFN or could even better results have been obtained? Some discussion around hyperparameter sensitivity would be interesting, perhaps I missed it. Similarly, a discussion around the computational complexity of the model would be interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Thanks for the valuable comments.*** We address concerns as follows.\n\n> Nonetheless, I believe that some of the other methods shown in Figure 3 also have that feature (e.g., REINVENT4 also has a controllable temperature parameter), which many be misleading in the current text/plot that suggests only the genetic GFN has it\n\nThanks for the suggestion, and we agree on your feedback. We will include the explanation of temperature-like parameter in REINVENT4.\n\n> How sensitive is the genetic GFN to the hyperparameters? I saw that many defaults were chosen to compare directly to REINVENT, did these also happen to be good options for GGFN or could even better results have been obtained? \n\nIn the manuscript, we have searched additional parameters only (GA parameters and the number of replay training); the results are provided in Appendix F.6. \n\nAs commented, we further conducted hyperparameter sensitivity analysis (batch size, learning rate, and varying the number of layers) with 3 independent runs. Since we have not searched all hyperparameters, there might be the better combination of hyperparameter setup.\n\n| batch 64, lr 0.0005 (main) | batch 128, lr 0.0005 | batch 64, lr 0.001 |\n| --- | --- | --- |\n| 16.088 ± 0.025 | 15.801 ± 0.016 | 15.900 ± 0.035 |\n\n| 2 layers | 3 layers (main) | 4 layers |\n| --- | --- | --- |\n| 15.628 ± 0.021 | 16.088 ± 0.025 | 16.012 ± 0.030 |\n\nThe results show that our method consistently achieves competitive performance under the different hyperparameter setups.\n\n> Similarly, a discussion around the computational complexity of the model would be interesting.\n\nUnfortunately, rigorous analysis of computational complexity presents significant challenges for the following reasons:\n- The number of iterations is performance-dependent, terminating either when the maximum reward calls are reached (where repeated samples do not necessitate additional calls) or when early termination conditions are met.\n- Graph GA involves RDKit API calls for tasks such as converting molecules to SMILES or removing certain components, (e.g., kelkulization).\n\n> Do these results still hold for the AUC top-1, and if so, is it by similar amounts or are the gaps wider between the top 3 models?\n\nHere, we provide the results of the top3 models. For each metric, the performance gaps are similar.\n\n|  | AUC Top1 | AUC Top10 | AUC Top100 |\n| --- | --- | --- | --- |\n| Genetic GFN | 16.527 ± 0.043 | 16.213 ± 0.042 | 15.516 ± 0.041 |\n| Mol GA | 16.001 ± 0.027 |15.686 ± 0.025 | 15.021 ± 0.025 |\n| SMILES-REINVENT | 15.686 ± 0.035 | 15.185 ± 0.035 | 14.306 ± 0.033 |\n\n|  | Avg. Top1 | Avg. Top10 | Avg. Top100 |\n| --- | --- | --- | --- |\n| Genetic GFN | 17.924 ± 0.054 | 17.760 ± 0.054 | 17.481 ± 0.054 |\n| Mol GA | 17.252 ± 0.032 | 17.116 ± 0.030 | 16.816 ± 0.029 |\n| SMILES-REINVENT | 17.345 ± 0.040 | 17.149 ± 0.042 | 16.763 ± 0.043 |\n\n> How does the model scale with the number of atoms? For instance, can it scale well to other modalities or macromolecules?\n\nThe maximum length of SMILES is set to 140, consistent with the setting used in REINVENT. This length corresponds to approximately 70-100 atoms, depending on the molecules. For JNK3 (#10), which consists of relatively large molecules, the generated SMILES lengths range from 50 to 130, with the number of atoms varying from approximately 35 to 100. In contrast, for the isomers_c7h8n2o2 (#8), the generated molecules typically contain about 10-15 atoms.\n\n> It is interesting to see that for task #22 (valsartan_smarts), only the genetic GFN and REINVENT perform somewhat well on this. ... I think it would be interesting to dig more deeply into some of the specific oracles/tasks and look at the molecules being generated by the genetic GFN and REINVENT - for instance, do they find the same solutions here?\n\nThanks for the insightful comments. The valsartan SMARTS targets molecules containing a SMARTS pattern related to valsartan while being characterized by physicochemical properties corresponding to the sitagliptin molecule [1]. It measures the arithmetic means of several scores, including (1) binary score about whether it contains a certain SMARTS structure, (2) LogP, (3) TPSA, and (4) Bertz score. Since we utilize a TDC oracle function for evaluations, we provide our empirical observations here.\n\n- Difficulty of the task: Due to the binary score (1 if the certain SMARTS pattern is included), many tries terminate with 0. Especially with a limited number of oracle calls, generating molecules containing a certain sub-structure is notoriously hard. Other literature shows that other methods achieve high scores with more oracle calls [2]. With 10K calls, even REINVENT and Genetic only succeed to find non-zero score molecules once out of five independent runs. \n- Another observation is that methods (REINVENT, Genetic GFN, and GEGL) achieving non-zero scores all generate SMILES with RNN-based models. Thus, we have a conjecture that SMILES generation is effective in generating a certain SMARTS pattern. \n\nWe provide examples of generated molecules with non-zero valsartan_smarts scores. Note that the other four seeds failed. Each run generates similar molecules (see Top1,10,100 samples in Fig.3 in the additional material), but the samples between the two runs (REINVENT and Genetic GFN) have different structures (the molecule-distance between Top1 samples is 0.854).\n\n[1] Brown et al. (2019). GuacaMol: benchmarking models for de novo molecular design. Journal of chemical information and modeling.\n\n[2] Hu et al. (2024). De novo drug design using reinforcement learning with multiple GPT agents. Advances in Neural Information Processing Systems, 36.\n\n> Despite the excellent results, the authors fail to provide any accompanying code, which is a shame (the link in the paper points to a non-existing anonymized repo).\n\nWe apologize for the inconvenience caused. There was an error in the provided link; please refer to this link (https://anonymous.4open.science/r/genetic_gfn).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General Response

We are sincerely grateful to the reviewers for their valuable feedback on our manuscript. We are pleased to hear that the reviewers found our paper **well-written** (4tbn, gNyQ, kTKB, FCer) and supported by **extensive experiments with state-of-the-art performance** (4tbn, gNyQ, kTKB, FCer). We appreciate the recognition of the **usefulness** (4tbn, gNyQ), **simplicity** (4tbn, kTKB), **novelty** (4tbn), and **rationale** (gNyQ) of our method. Additionally, we are encouraged by the acknowledgment of our research as **well-motivated** and **promising with potential real-world impact** (4tbn, gNyQ).


### Answers for the common concern and feedback


- Further explanation of Graph GA: We adopt the original implementation of Graph GA [1]. To make the manuscript self-contained, we will include detailed explanations of Graph GA along with figures; please see the summary of Graph GA andFig. 2 in the supplementary material.
- Provided link does not work: We apologize for the inconvenience. Our code is available at https://anonymous.4open.science/r/genetic_gfn. (directories: PMO `pmo/main/genetic_gfn`, multi-objective `multi_objective/genetic_gfn`, SARS-CoV-2 `sars_cov2/genetic_gfn`)
- Analysis of generated molecules: We will include more detailed explanations along with visual results (Fig. 3 and 4 in the supplementary material).

Additionally, in our supplementary material, we provide a semantic overview of pretraining and fine-tuning with Genetic GFN (Fig. 1), examples of Graph GA operations (Fig. 2), visual results for valsartan_SMARTS (oracle ID: #22) in Fig. 3, and visual results of SARS-CoV-2 inhibitor designs in Fig. 4.


***Summary of Graph GA:*** Graph GA is implemented by utilizing predefined SMARTS patterns for operations. During crossover, the algorithm randomly selects either non_ring_crossover or ring_crossover with equal probability. Non_ring_crossover involves cutting an arbitrary non-ring edge of two parent molecules and combining the subgraphs, while ring_crossover cuts two edges within a ring and combines the subgraphs from different parents. For mutations, the algorithm randomly applies one of seven modifications: atom_deletion, atom_addition, atom_insertion, atom_type_change, ring_bond_deletion, ring_bond_addition, and bond_order_change. Invalid molecules resulting from mutation are discarded and the mutation is re-applied.

Meta-review: The paper combines genetic algorithms and GFlowNets, which gives state-of-the-art results in the field of generative modeling for molecular data. This is a natural marriage, as GFlowNet is an off-policy method that allows to naturally introduce genetic algorithms into the sampling procedure.

All reviewers voted for acceptance, highlighting strong empirical results and combining the strengths of deep learning (high reward) and genetic algorithms (using domain-specific knowledge that helps navigate chemical spaces). As such, it is my pleasure to recommend acceptance of the paper.
Decision: Accept (poster)