Title: Genetic-guided GFlowNets for Sample Efficient Molecular Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Molecular Optimization

Sample Efficiency

Generative Flow Networks

Genetic Algorithm
Abstract: The challenge of discovering new molecules with desired properties is crucial in domains like drug discovery and material design. Recent advances in deep learning-based generative methods have shown promise but face the issue of sample efficiency due to the computational expense of evaluating the reward function. This paper proposes a novel algorithm for sample-efficient molecular optimization by distilling a powerful genetic algorithm into deep generative policy using GFlowNets training, the off-policy method for amortized inference. This approach enables the deep generative policy to learn from domain knowledge, which has been explicitly integrated into the genetic algorithm. Our method achieves state-of-the-art performance in the official molecular optimization benchmark, significantly outperforming previous methods. It also demonstrates effectiveness in designing inhibitors against SARS-CoV-2 with substantially fewer reward calls.

Genetic-guided GFlowNets for Sample Efficient Molecular Optimization

Abstract

The challenge of discovering new molecules with desired properties is crucial in domains like drug discovery and material design. Recent advances in deep learning-based generative methods have shown promise but face the issue of sample efficiency due to the computational expense of evaluating the reward function. This paper proposes a novel algorithm for sample-efficient molecular optimization by distilling a powerful genetic algorithm into deep generative policy using GFlowNets training, the off-policy method for amortized inference. This approach enables the deep generative policy to learn from domain knowledge, which has been explicitly integrated into the genetic algorithm. Our method achieves state-of-theart performance in the official molecular optimization benchmark, significantly outperforming previous methods. It also demonstrates effectiveness in designing inhibitors against SARS-CoV-2 with substantially fewer reward calls.

1 Introduction

Discovering new molecules is one of the fundamental tasks in the chemical domain, with applications in drug discovery [1] and material design [2]. Particularly, de novo molecular design focuses on generating novel molecules with desired properties from scratch. In this context, deep learning-based generative methods have emerged, showing promising results (e.g., [3-7]). However, these methods still face a key challenge: the reward function is computationally expensive (e.g., assessing binding affinity through docking simulations), while the molecule space is combinatorially large.

Sample-efficient molecular optimization is thus crucial for discovering high-reward molecular structures with limited reward calls, especially for real-world applicability. The recently proposed benchmark, Practical Molecular Optimization (PMO) [8], has extensively assessed the sample efficiency of various algorithms, including reinforcement learning [3, 9], active learning [10], variational autoencoders [4, 5], generative flow networks (GFlowNets) [6, 11], and classical optimization methods like Bayesian optimization [12] and genetic algorithms [13, 14]. Interestingly, the PMO benchmark has revealed a shift in algorithm rankings, with classical algorithms often outperforming recently proposed methods such as GFlowNets when the sample efficiency is considered.

Recent investigations [15, 16], including those highlighted by the PMO benchmark [8], indicate that the classical frameworks, especially genetic algorithms (GA), still exhibit competitive performances compared to recently proposed deep learning methods. These studies underscore that GAs effectively navigate the chemical space using domain-specific genetic operators. In contrast, deep learning methods usually do not leverage domain-specific knowledge, relying instead on deep networks to autonomously learn these insights. It can lead to inefficient training processes due to the lack of expert guidance [17]. To address this limitation, Genetic Expert Guided Learning (GEGL) [17] has been introduced, which enhances deep learning by distilling GA-generated samples into a deep generative policy using maximum likelihood estimation. This approach enables the deep generative policy to

Policy

𝜋𝜃

⋯

⋯

⋯

GRU

GRU

GRU

SMILES

CC=Cc1ccc(C[NH2+]

⋯

NC(=O)NC1=CC=C

⋯

O=C1O[C@@H]

⋯

Buffer

𝒟

with reward evaluation

a. SMILES generation with policy

Population

Crossover & Mutation

Offspring

(GFlowNet Training)

Rank-based sampling

Original

Weighted

Industrial & Systems Engineering implicitly utilize domain-specific knowledge from the GA as a form of guidance. However, despite its successes, GEGL may face challenges in generalizing to unexplored regions and in learning a peaky distribution from samples, as it maximizes likelihood equally across all high-reward samples without adequately configuring the reward landscape.

To address this, we propose a novel molecular optimization algorithm that integrates domainspecialized genetic algorithms into GFlowNets, an off-policy training method that trains policy to sample proportional to their rewards. As illustrated in Fig. 1, we first generate diverse candidates using the current policy and refine the candidates into higher-reward samples using GAs. Subsequently, we fine-tune the policy with a GFlowNet using the collected samples. Unlike the MLE training, which can be stuck in local modes when the reward landscape is peaky, GFlowNet trains the policy to sample molecules proportional to rewards . To enhance sample efficiency, we perform unsupervised pretraining on chemical datasets and regularize the GFlowNets policy with KL-divergence to align generative probability with the dataset distribution, focusing on the compact valid space.

Our contributions can be interpreted from both perspectives of GFlowNets and GAs:

GA increases the exploitation power of GFlowNets. The proposed algorithm incorporates an effective off-policy exploration into GFlowNets based on domain-specialized GA. This approach aligns with recent studies that utilize off-policy explorations to guide toward the high-reward regions [18]. Our key contribution lies in explicitly leveraging domain knowledge about chemical structures and effectively distilling it into GFlowNets, which enable exceptional performance in real-world tasks beyond small-scale molecular generations. This contribution is crucial for the field of GFlowNets, which have struggled with sample-efficient molecular optimization tasks, even when using proxy reward models for active learning [10, 8]; see Section 4.2.

GFlowNets increases population diversity of GA. The proposed algorithm generates diversified samples using GFlowNets, enabling GA to effectively improve samples; our method can be regarded as a new genetic algorithm with deep generative policy-based population resetting. Our GFlowNet policy, parameterized over the whole space, periodically resets the population by diversely sampling individuals proportional to rewards, mitigating premature convergence of GA [19, 20]. Experiments show that ours outperforms recent GAs in the PMO benchmark; see Section 5.1.

Our extensive experiments demonstrate the effectiveness and practical applicability of the proposed method. First, our method achieves the highest total score of 16.213 across 23 oracles in the Practical Molecular Optimization benchmark [8], outperforming all other baselines. Second, we conduct in in silico experiments for designing SARS-CoV-2 inhibitors. The proposed method successfully generates inhibitors with ten times fewer reward calls. Moreover, our method effectively balances optimization and diversity, achieving higher scores with increased diversity compared to previously top-ranked methods.

2 Background

2.1 Sample efficient de novo molecular optimization

Molecular design is the process of proposing new molecules likely to exhibit desirable outcomes. Compared to traditional virtual screening approaches, which identify suitable molecules from virtual

13

libraries with a large number of molecules known a priori, de novo approaches seek to generate molecule structures anew. The desired properties can be measured using score functions O , called oracle. Formally, molecular design can be formulated as arg max x ∈X O ( x ) , where x is a molecule, and X denotes the chemical space which comprises all possible molecules.

The publication of standard benchmarks and datasets has facilitated the assessment of de novo design methods (e.g., GuacaMol [21], Therapeutics Data Commons (TDC) [22]). The score functions are designed to consider various properties, such as the presence and absence of substructures, similarity, isomers, structural features, physicochemical properties, biological activity, and binding affinity (i.e., docking score). Notably, the PMO benchmark [8] offers a unified framework that comprehensively evaluates the sample efficiency of a range of molecular design methods.

2.2 Generative flow networks

Generative flow networks (GFlowNets) [23] are introduced as a new class of probabilistic models to sample a discrete compositional object x ∈ X from the target distribution, i.e., P ( x ) ∝ e -E ( x ) . In general, direct sampling from the target distribution is challenging since the partition function Z = ∑ x ∈X e -E ( x ) is intractable when the sample space is combinatorially large. Hence, GFlowNets sample an object from an unnormalized distribution as a constructive generative process, where discrete actions iteratively modify a state -apartially constructed object. We define a trajectory as τ = ( s 0 , . . . , s T ) , where s T is a terminal state corresponding to a fully constructed object x .

A GFlowNet models flow F of particles along a directed acyclic graph (DAG). The source and sink nodes of the DAG correspond to the initial state s 0 and terminal states s T , respectively. The trajectory flow F ( τ ) is defined as a flow through the trajectory τ , and the state flow F ( s ) is defined as the sum of trajectory flows that include the state s , i.e., F ( s ) = ∑ s ∈ τ F ( τ ) . The edge flow F ( s → s ′ ) the sum of trajectory flows through the edge from state s to s ′ , i.e., F ( s → s ′ ) = ∑ ( s,s ′ ) ∈ τ F ( τ ) .

From the flow function F , we derive two policy distributions. The forward policy P F ( s ′ | s ) is the probability of transitioning from state s to its child state s ′ , defined as the edge flow F ( s → s ′ ) normalized by the state flow F ( s ) , i.e., P F ( s ′ | s ) = F ( s → s ′ ) /F ( s ) . Similarly, the backward policy P B ( s | s ′ ) is the probability of moving from state s ′ to its parent state s , defined as P B ( s | s ′ ) = F ( s → s ′ ) /F ( s ′ ) . Utilizing these forward and backward policies, GFlowNets can derive an optimal sampler P ( s T ) = ∏ P F ( s t | s t -1 ) = R ( s T ) /Z if balance conditions (e.g., [6, 24-27]) are satisfied.

Trajectory balance loss [24]. One of the most popular conditions is trajectory balance (TB) , which directly parameterizes P F , P B , and flow of initial state (i.e., partition function) Z to satisfy the following trajectory balance condition:

Then, this equation is converted into a loss function to be minimized along sampled trajectories, i.e.,

In GFlowNet training, employing exploratory behavior policies or replay training is allowed since GFlowNet can be trained in an off-policy manner, which is a key advantage [23, 24, 28, 18].

3 Genetic-guided GFlowNets

This section describes how the desired molecules are discovered with Genetic GFN. We model the generation process of molecules as a string-based constructive process. First, we pretrain the policy to learn the distribution of valid molecules. During the optimization phase, we iteratively generate molecules and update the policy with GFlowNet training using high-reward molecules. Particularly, we introduce graph-based genetic search to refine generated samples.

Algorithm 1 Genetic GFN training with limited reward calls

3.1 Factorized string-based generative policy and unsupervised pretraining

Building on insights from previous works [3, 17], we employ a string-based representation strategy, simplifying the molecular generation process by reducing it to a one-directional sequence generation. Weadopt a sequence generative policy using a string-based assembly strategy, especially the simplified molecular-input line-entry system (SMILES) [29]. Motivated by REINVENT [3], we parameterize the policy using a recurrent neural network architecture [30]. Then, the probability π θ ( x ) of generating a molecule, can be factorized to ∏ n t =1 π θ ( x t | x 1 , . . . , x t -1 ) , where x 1 , . . . , x n are characters of SMILES representation of x .

As demonstrated in previous studies, including [3, 17, 8], pretraining is inevitable since training the generative policy from scratch is excessively sample-inefficient. Therefore, our policy is pre-trained to maximize the likelihood of valid molecules on existing chemical datasets D pre; note that pretraining does not require oracle information . Precisely, the policy is pretrained to minimize the following:

3.2 GFlowNet training of the generative policy with graph-based genetic search

To generate desirable molecules with limited reward calls, we iteratively generate samples using two distinct strategies (Section 3.2.1) and fine-tune the policy, initialized with π pre, using a GFlowNet by replaying collected samples (Section 3.2.2). The overall procedure is described in Algorithm 1.

3.2.1 Molecule generation strategies in Genetic GFN

We employ two distinct molecule generation strategies, SMILES generation with our training policy and graph-based genetic search. These two strategies are synergized to generate diversified and high-reward samples, efficiently searching the vast chemical space.

SMILES generation with policy. The training policy π θ generates SMILES sequences. Since the policy is trained using the trajectory balance loss (see the following subsection), it is the same as sampling x from ∏ T t =1 P F ( s t | s t -1 ) ∝ R ( s T = x ) , where s t -1 is represented by previously collected SMILES token, and log R ( x ) = -β O ( x ) with an inverse temperature β .

Graph-based genetic search. To effectively search the higher-reward region, we employ a genetic algorithm that iteratively evolves populations through crossover , mutation , and selection . We adopt the operations of the graph-based genetic algorithm, Graph GA [13], which has proven to effectively search the molecule space with finely designed genetic operations; please refer to the original paper for details. The genetic search is performed as follows:

Initialize a population D pop: The initial population is selected from the whole buffer D , consisting of samples from the policy and previous genetic search.

Generate offspring D off: A child is generated from randomly chosen two parent molecules by combining the fragments ( crossover ). Then, the child is randomly modified ( mutation ).

Select a new population D ′ pop : Sample from D pop ∪ D off, and go back to 2.

One key advantage is that offspring can have a large distance from the parents in the 1D string space, even if the molecule distances are small, which is beneficial to avoid being stuck in local optima; see the experimental results in Table 3b.

3.2.2 Updating the generative policy with GFlowNets training

Using the generated samples, the policy is fine-tuned using the trajectory balance loss. The off-policy property of GFlowNet losses enables the utilization of refined samples from the genetic search. In particular, for better sample efficiency, we employ replay training with a rank-based reweighed buffer.

TB loss with KL-divergence penalty. The generative policy is trained using the trajectory balance loss in Eq. (1). Note that we set P B to 1 for simplicity since SMILES generations are conducted in one direction. To ensure that the policy does not deviate excessively from the pretrained policy during training, we introduce a Kullback-Leibler (KL) divergence penalty inspired by the works in language model fine-tuning [31, 32]. Thus, our model is updated to minimize the following loss function:

where π pre denotes the the pre-trained policy. As a result, π θ is trained to generate desired (by L TB) and valid (by π pre) molecules. Note that trajectories on which the proposed loss is minimized are sampled from the experience buffer.

Rank-based reweighed experience buffer. The rank-based reweighting biases the samples towards high-reward candidates by assigning greater weight to trajectories with higher ranks, thereby enhancing the focus on more promising solutions [33, 34]. The weight is computed as follows:

Here, k is a weight-shifting factor, and rank O , D ( x ) is a relative rank of value of O ( x ) in the dataset D . Note that we also utilize rank-based sampling in the genetic search (steps 1 and 3).

4 Related works

4.1 Genetic algorithms for molecular optimization

Genetic algorithm (GA) is a representative meta-heuristic inspired by the evolutionary process. This subsection focuses on discussing the application of GA in molecular optimization. As one of the seminal works, a graph-based GA (Graph GA) was proposed with sophisticatedly designed operations based on chemical knowledge [13]. Note that our method also adopts Graph GA operations in the genetic search. Various strategies for molecular assembly, not limited to graphs, have been utilized in GA [35, 14, 36]. A recent contribution by [16] introduces an enhanced version of Graph GA. They introduce quantile-uniform sampling to bias the population towards containing higher reward samples while maintaining diversity. Experimental results from Mol GA demonstrate the effectiveness of GAs as strong baselines, achieving state-of-the-art (SOTA) performance in the PMO benchmark.

4.2 GFlowNets for molecular optimization

Generative Flow Networks (GFlowNets or GFN) [23, 6] have drawn significant attention in scientific discovery [37], particularly in molecular optimization and biological sequence design [6, 10, 38, 18, 39, 40, 11, 41]. GFlowNets, which are off-policy variational inference methods [42], are closely related to value-based reinforcement learning within soft Markov Decision Processes (soft MDPs) [43, 44], focusing on learning maximum entropy agents [45, 46]. This allows GFlowNets to generate

0

2000

4000

6000

8000

10000

Num. of oracle calls

0.2

0.4

0.6

0.8

1.0

Avg. Top-10

isomers_c9h10n2o2pf2cl

Genetic GFN (Ours)

SMILES-REINVENT

GP BO

0

2000

4000

6000

8000

10000

Num. of oracle calls

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Avg. Top-10

celecoxib_rediscovery

0

2000

4000

6000

8000

10000

Num. of oracle calls

0.5

0.6

0.7

0.8

0.9

Avg. Top-10

amlodipine_mpo

0

2000

4000

6000

8000

10000

Num. of oracle calls

0.4

0.5

0.6

0.7

0.8

Avg. Top-10

scaffold_hop

diverse candidates for chemical and biological structures. Advances in GFlowNets have included new objective functions [6, 24, 25], improved credit assignments [26, 27], and enhanced off-policy exploration strategies [18, 47-49]. Our approach, an improved off-policy exploration technique for GFlowNets, enhances sample efficiency and scalability for moderate-scale chemical discovery. We present extensive experiments targeting the discovery of larger-scale molecules, with SMILES strings approximately 100 characters long, surpassing the scope of smaller molecule generation tasks commonly addressed in GFlowNets literature [6, 38].

5 Experiments

This section provides extensive experimental results, including experiments on the official sampleefficient molecular optimization benchmark and in silico design for SARS-CoV-2 inhibitors. The codes are available at https://github.com/hyeonahkimm/genetic_gfn.

5.1 Sample efficient molecular optimization

In this experiment section, we compare Genetic GFN with various molecular optimization methods from the perspective of sample efficiency. The performance is primarily quantified by the area under the curve (AUC), with the number of score function calls limited to 10K. Note that we rigorously follow the Practical Molecular Optimization (PMO) benchmark [8].

5.1.1 Main results in the official benchmark of PMO

As baselines, we employ Top-8 methods from the PMO benchmark since they recorded the best AUC Top-10 in at least one oracle. The baseline methods include various ranges of algorithms and representation strategies. First, REINVENT [3] is an RL method that tunes the policy with adjusted likelihood. Graph GA [13], STONED [14], SMILES GA [21], and SynNet [36] are genetic algorithms that utilize different assembly strategies; they use fragment-based graphs, SELFIES, SMILES, and synthesis, respectively. Additionally, a hill climbing method (SMILES-LSTM-HC [21]) and Bayesian optimization (GP BO [12]) are included. SMILES-LSTM-HC iteratively generates samples and imitates high-reward samples, while GP BO uses a surrogate model with the Gaussian process (GP) and Graph GA to optimize the GP acquisition functions in the inner loop.

Moreover, we adopt additional methods, Mol GA [16] and GEGL [17]. Mol GA is an advanced version of Graph GA and outperforms other baselines in the PMO benchmark. On the other hand, GEGL is an ablated version of our approach that utilizes imitation learning with a reward-priority queue instead of GFlowNet training with rank-based sampling. For both, we adopt the original implementations 23 with hyperparameters searches following the guidelines; see Appendix C.

The main results in Table 1 report the AUC score of Top-10 candidates with independent five runs with different seeds. In addition, Fig. 2 visually presents the Top-10 average score across the computational budget, i.e., the number of oracle calls, providing a concise overview of the results. Due to the lack of space, the best five results are provided; please check Appendix G.2 for the rest of the results. As shown in Table 1, Genetic GFN outperforms the other baselines with a total of 16.213 and attains the highest AUC Top-10 values in 14 out of 23 score functions. The results of diversity and SA score for each oracle are presented in Appendix G.5.

5.1.2 Controllability of the scores-diversity trade-off and ablation studies

Controllability of the scores-diversity trade-off.

In the benchmark, we found a pronounced tradeoff between attaining high evaluation scores within a limited budget and generating diverse molecular candidates. This section demonstrates the controllability of the score-diversity trade-off through adjustments in the inverse temperature β . Decreasing the inverse temperature gives more diverse candidates. The results in Fig. 3 demonstrate adjustments of β can control the trade-off between score and diversity, achieving Pareto-frontier to other baselines in the benchmark. Notably, Genetic GFN with β = 30 achieves a higher AUC Top-10 with a greater diversity compared to the SOTA GA method (Mol GA: 15.686 with a diversity of 0.465) and RL method

0.45

0.50

0.55

0.60

0.65

Diversity

16.0

16.5

17.0

17.5

Avg. Top-10

Genetic GFN (

=50)

Genetic GFN (

=30)

Genetic GFN (

=10)

Genetic GFN (

=5)

SMILES-REINVENT

GP BO

β

= 1

β

= 5

β

= 10

β

= 30

β

= 50

AUC-10

11.083

14.597

14.735

15.815

16.213

Div.

0.812

0.670

0.663

0.528

0.432

(REINVENT: 15.185 with a diversity of 0.468). Similarly, the weight-shifting factor k in rank-based sampling can control the trade-off; see Appendix G.1.

Ablation studies. The ablation studies investigate the essential components of our framework: the genetic search (GS) and the KL-divergence penalty. To assess the effectiveness of the genetic search, we also compare its performance against the exploration strategy used in previous GFlowNet studies

(a) Average and standard deviation of AUC scores ( ↑ )

(b) Search distances ( ↑ )

[6, 10]. It samples actions from a GFlowNet sampler mixed with a uniform distribution, similar to ϵ -greedy in RL. The results, shown in Table 2, reveal that the removal of either component results in a decline in performance, underscoring the importance of employing a suitable exploration strategy. Detailed results, including statistical analysis, are provided in Appendix G.4.

5.1.3 Comparisons with GFlowNets variants

We compare Genetic GFN with the graph-based GFlowNet [6], GFlowNet-AL [10], and the local search GFlowNet (LS-GFN) [18] using SMILES representations. LS-GFN utilizes Monte Carlo Markov Chain (MCMC) techniques, incorporating partial backtracking and reconstructing solution trajectories with the training policy as the proposal distribution [18]. The experiments are conducted on the PMO benchmark, and we implement LS-GFN with SMILES by replacing our genetic search with a local search. Note that while the original LS-GFN employs the prepend-append MDP, which does not directly apply to SMILES, we use the same one-directional SMILES generation as ours.

As shown in Table 3a, Genetic GFN outperforms other GFlowNet variants. Notably, generating SMILES is significantly more advantageous than generating graph-based fragments. The performance gap between Genetic GFN and LS-GFN highlights the importance of a proper exploratory policy. To further analyze, we measure the distance between samples before and after searches in GSK3 β and JNK3. The normalized Levenshtein distances for SMILES and Tanimoto similarity for molecules are reported in Table 3b. The results show that the local search may be inefficient in effectively searching in moderate-scale chemical spaces because its capabilities heavily depend on the current policy, leading to suboptimal search performance. In contrast, our approach leverages a domain-specialized genetic search within the molecule graph space, working as an effective off-policy exploration - the samples with SMILES representation are used to train the string-based generative policy.

5.1.4 Sample efficient multi-objective molecular optimization

According to Zhu et al. [11], we apply our method to multi-objective tasks: GSK3 β +JNK3 and GSK3 β +JNK3+QED+SA. Notably, GSK3 β and JNK3 are potential targets of Alzheimer's Disease treatments [50]. We use a linear combination of each objective with given coefficients, and the performance is measured by hypervolumes with 1K evaluations. We obtained the results from five independent trials using different seeds. Even though Genetic GFN is not designed for multiobjective molecular optimization, it demonstrates notable performance using proper scalar-valued score functions; please see Appendix E for details.

5.1.5 Further analysis

Active learning with Genetic GFN. Similar to GFlowNet-AL, ours can work as a generative model in multi-round active learning. We compare Genetic GFN-AL with other model-based and active learning methods; please refer to Appendix D.

Genetic GFN with SELFIES representation. Genetic GFN with SELFIES generation achieves the improved sample efficiency to other SELFIES-based methods; see Appendix G.6.

Table 4: Average and standard deviation of hypervolumes ( ↑ ) for each task. The baseline results are directly from the HN-GFN paper [11]. The bold text indicates the best value.

Score: 0.903

(docking: -12.7, QED: 0.383, SA: 2.720)

Score: 0.909

(docking: -12.6, QED: 0.412, SA: 2.251)

Score: 0.917

(docking: -12.0, QED: 0.859, SA: 3.376)

Score: 0.903

Score: 0.909

Score: 0.917

NH

NH

NH

NH

(docking: -12.7, QED: 0.383, SA: 2.720)

(docking: -12.6, QED: 0.412, SA: 2.251)

(docking: -12.0, QED: 0.859, SA: 3.376)

Score: 0.892

(docking: -13.0, QED: 0.268, SA: 3.226)

Score: 0.908

(docking: -13.5, QED: 0.370, SA: 3.153)

Score: 0.894

(docking: -13.7, QED: 0.197, SA: 2.951)

Score: 0.892

(docking: -13.0, QED: 0.268, SA: 3.226)

Score: 0.908

(docking: -13.5, QED: 0.370, SA: 3.153)

Score: 0.894

(docking: -13.7, QED: 0.197, SA: 2.951)

M

NH

NH

NH

Industrial & Systems Engineering 21 Industrial & Systems Engineering 21 Sensitivity analysis. We provide the experimental results by varying the hyperparameters, such as the offspring size and the number of training loops; please refer to Appendix G.8.

5.2 Designing inhibitors against SARS-CoV-2 targets

In this subsection, we conduct drug discovery experiments for Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-Cov-2), known as the novel coronavirus. One desired property is maximizing the binding affinity to the target protein. The binding affinity is measured with a docking score, which is calculated based on the energies of the interaction between the ligand and the receptor. Typically, the computation of docking scores is expensive since it involves predicting the spatial orientation and binding affinity of the molecule in the active site of the target protein. We employ Quick Vina 2 [51] docking software to assess generated molecules.

Additionally, QED (Quantitative Estimate of Drug-likeness) and SA (Synthetic Accessibility) are considered to quantify the drug-likeness and difficulties of synthesizing. The higher QED, which ranges [0, 1], and the lower SA, which ranges [0, 10], are desired. Therefore, we define the score function s ( x ) for designing SARS-Cov-2 inhibitors as a linear combination of normalized scores according to the previous work [52]. Following [53] and [52], the target proteins are selected: PLPro_7JIR, a critical enzyme in the life cycle of SARS-CoV-2, and RdRp_6YYT, which is essential for the replication and the transcription of genes.

The experiments are conducted with up to 1000 update steps with 128 batch size [52]. As shown in Table 5, ours achieves the highest Top-100 average scores only with 100 steps, which is 10 times fewer than others. Note that the score is recalculated based on the normalized score function in Eq. (5) using average values in the MolRL-MGPT paper [52]; the full results and a more detailed experimental setup are provided in Appendix F. We also report the best candidates of 100 steps in Fig. 4 and in Fig. 5. The final molecules correspond to the Top-1 score molecules from 3 independent runs.

6 Discussion

This paper introduces a Genetic-guided GFlowNet (Genetic GFN), which integrates a domain-specific genetic algorithm to guide the GFlowNet policy toward higher-reward samples. The method employs off-policy training with a rank-based reweighted buffer, enhancing the policy as a powerful amortized inference sampler for chemical discovery. Extensive experiments demonstrate that Genetic GFN effectively generates desirable molecules within the high-dimensional chemical space, including long chemical structure sequences (e.g., ≥ 100). On the other hand, our approach can be considered as

a novel population reinitialization strategy for genetic algorithms using GFlowNets, which sample diverse objects proportional to rewards.

Limitations and future works. Our method assumes the existence of effective genetic algorithms, which is valid for the molecular design domain. However, designing domain-specific operators for genetic algorithms can be challenging in other fields. One possible future work is to enhance genetic algorithms using the neural policy similar to recent studies [54, 55]. Another direction is to extend our approach to other domains, such as combinatorial optimization. For instance, we could utilize a powerful GA, hybrid genetic search [56], to design a GFlowNet-based solver for routing problems.

Broader Impact. This paper introduces a new generative model, significantly enhancing sample efficiency in molecular optimization. This advance is likely to hold substantial promise for this field, potentially accelerating the development of new therapies and advanced materials. Our research is currently focused on in-silico experiments. The potential safety concerns of discovered molecules are further examined in the subsequent processes, such as in-vitro experiments and pre-clinical tests.

Acknowledgments and Disclosure of Funding

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024-00410082). The authors are grateful to Austin Tripp and Xiuyuan Hu for their help with baselines.

References

James P Hughes, Stephen Rees, S Barrett Kalindjian, and Karen L Philpott. Principles of early drug discovery. British Journal of Pharmacology , 162(6):1239-1249, 2011.

Alexander W Hains, Ziqi Liang, Michael A Woodhouse, and Brian A Gregg. Molecular semiconductors in organic photovoltaic cells. Chemical Reviews , 110(11):6689-6735, 2010.

Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of Cheminformatics , 9(1):1-14, 2017.

Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science , 4(2):268-276, 2018.

Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International Conference on Machine Learning , pages 2323-2332. PMLR, 2018.

Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. In Advances in Neural Information Processing Systems , volume 34, pages 27381-27394, 2021.

Seul Lee, Jaehyeong Jo, and Sung Ju Hwang. Exploring chemical space with score-based out-ofdistribution generation. In International Conference on Machine Learning , pages 18872-18892. PMLR, 2023.

Wenhao Gao, Tianfan Fu, Jimeng Sun, and Connor Coley. Sample efficiency matters: a benchmark for practical molecular optimization. Advances in Neural Information Processing Systems , 35:21342-21357, 2022.

Zhenpeng Zhou, Steven Kearnes, Li Li, Richard N Zare, and Patrick Riley. Optimization of molecules via deep reinforcement learning. Scientific Reports , 9(1):1-10, 2019.

Moksh Jain, Emmanuel Bengio, Alex Hernández-Garcıa, Jarrid Rector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with GFlowNets. In International Conference on Machine Learning , pages 9786-9801. PMLR, 2022.

Yiheng Zhu, Jialu Wu, Chaowen Hu, Jiahuan Yan, Tingjun Hou, Jian Wu, et al. Sample-efficient multi-objective molecular optimization with GFlowNets. Advances in Neural Information Processing Systems , 36, 2024.

Austin Tripp, Gregor N. C. Simm, and José Miguel Hernández-Lobato. A fresh look at de novo molecular design benchmarks. In NeurIPS 2021 AI for Science Workshop , 2021.

Jan H Jensen. A graph-based genetic algorithm and generative model/Monte Carlo tree search for the exploration of chemical space. Chemical Science , 10(12):3567-3572, 2019.

AkshatKumar Nigam, Robert Pollice, Mario Krenn, Gabriel dos Passos Gomes, and Alan Aspuru-Guzik. Beyond generative models: superfast traversal, optimization, novelty, exploration and discovery (STONED) algorithm for molecules using SELFIES. Chemical Science , 12(20):7079-7090, 2021.

AkshatKumar Nigam, Pascal Friederich, Mario Krenn, and Alan Aspuru-Guzik. Augmenting genetic algorithms with deep neural networks for exploring the chemical space. In International Conference on Learning Representations , 2020.

Austin Tripp and José Miguel Hernández-Lobato. Genetic algorithms are strong baselines for molecule generation. arXiv preprint arXiv:2310.09267 , 2023.

Sungsoo Ahn, Junsu Kim, Hankook Lee, and Jinwoo Shin. Guiding deep molecular optimization with genetic exploration. In Advances in Neural Information Processing Systems , volume 33, pages 12008-12021, 2020.

Minsu Kim, Taeyoung Yun, Emmanuel Bengio, Dinghuai Zhang, Yoshua Bengio, Sungsoo Ahn, and Jinkyoo Park. Local search GFlowNets. In International Conference on Learning Representations , 2024.

David B Fogel. An introduction to simulated evolutionary optimization. IEEE Transactions on Neural Networks , 5(1):3-14, 1994.

Hari Mohan Pandey, Ankit Chaudhary, and Deepti Mehrotra. A comparative review of approaches to prevent premature convergence in GA. Applied Soft Computing , 24:1047-1077, 2014.

Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. GuacaMol: benchmarking models for de novo molecular design. Journal of Chemical Information and Modeling , 59(3):1096-1108, 2019.

Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf H Roohani, Jure Leskovec, Connor W. Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) , 2021.

Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. Journal of Machine Learning Research , 24(210):1-55, 2023.

Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in GFlowNets. In Advances in Neural Information Processing Systems , volume 35, pages 5955-5967, 2022.

Kanika Madan, Jarrid Rector-Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, Andrei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability. In International Conference on Machine Learning , pages 23467-23483. PMLR, 2023.

Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of GFlowNets with local credit and incomplete trajectories. In International Conference on Machine Learning , pages 26878-26890. PMLR, 2023.

Hyosoon Jang, Minsu Kim, and Sungsoo Ahn. Learning energy decompositions for partial inference of GFlowNets. In International Conference on Learning Representations , 2024.

Heiko Zimmermann, Fredrik Lindsten, Jan-Willem van de Meent, and Christian A Naesseth. A variational perspective on generative flow networks. Transactions on Machine Learning Research , 2023.

David Weininger. SMILES, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of Chemical Information and Computer Sciences , 28(1):31-36, 1988.

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 , 2014.

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems , volume 36, pages 53728-53741, 2023.

Austin Tripp, Erik Daxberger, and José Miguel Hernández-Lobato. Sample-efficient optimization in the latent space of deep generative models via weighted retraining. In Advances in Neural Information Processing Systems , volume 33, pages 11259-11272, 2020.

Minsu Kim, Federico Berto, Sungsoo Ahn, and Jinkyoo Park. Bootstrapped training of scoreconditioned generator for offline design of biological sequences. In Advances in Neural Information Processing Systems , volume 36, pages 67643-67661, 2023.

Naruki Yoshikawa, Kei Terayama, Masato Sumita, Teruki Homma, Kenta Oono, and Koji Tsuda. Population-based de novo molecule generation, using grammatical evolution. Chemistry Letters , 47(11):1431-1434, 2018.

Wenhao Gao, Rocío Mercado, and Connor W. Coley. Amortized tree generation for bottomup synthesis planning and synthesizable molecular design. In International Conference on Learning Representations , 2022.

Moksh Jain, Tristan Deleu, Jason Hartford, Cheng-Hao Liu, Alex Hernandez-Garcia, and Yoshua Bengio. GFlowNets for AI-driven scientific discovery. Digital Discovery , 2(3):557-577, 2023.

Max W Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving GFlowNet training. In International Conference on Machine Learning , pages 30956-30975. PMLR, 2023.

Pouya M Ghari, Alex Tseng, Gökcen Eraslan, Romain Lopez, Tommaso Biancalani, Gabriele Scalia, and Ehsan Hajiramezanali. Generative flow networks assisted biological sequence editing. In NeurIPS 2023 Generative AI and Biology (GenBio) Workshop , 2023.

Miruna Cretu, Charles Harris, Julien Roy, Emmanuel Bengio, and Pietro Liò. SynFlowNet: Towards molecule design with guaranteed synthesis pathways. In ICLR 2024 Workshop on Generative and Experimental Perspectives for Biomolecular Design , 2024.

Hyeonah Kim, Minsu Kim, Sungsoo Ahn, and Jinkyoo Park. Symmetric replay training: Enhancing sample efficiency in deep reinforcement learning for combinatorial optimization. In Proceedings of the 41st International Conference on Machine Learning , pages 24110-24136. PMLR, 2024.

Nikolay Malkin, Salem Lahlou, Tristan Deleu, Xu Ji, Edward Hu, Katie Everett, Dinghuai Zhang, and Yoshua Bengio. GFlowNets and variational inference. In International Conference on Learning Representations , 2023.

Sobhan Mohammadpour, Emmanuel Bengio, Emma Frejinger, and Pierre-Luc Bacon. Maximum entropy GFlowNets with soft Q-learning. In International Conference on Artificial Intelligence and Statistics , pages 2593-2601. PMLR, 2024.

Tristan Deleu, Padideh Nouri, Nikolay Malkin, Doina Precup, and Yoshua Bengio. Discrete probabilistic inference as control in multi-path environments. In The 40th Conference on Uncertainty in Artificial Intelligence , 2024.

Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In Advances in Neural Information Processing Systems , volume 30, 2017.

Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning , pages 1352-1361. PMLR, 2017.

Jarrid Rector-Brooks, Kanika Madan, Moksh Jain, Maksym Korablyov, Cheng-Hao Liu, Sarath Chandar, Nikolay Malkin, and Yoshua Bengio. Thompson sampling for improved exploration in GFlowNets. In ICML 2023 Structured Probabilistic Inference & Generative Modeling (SPIGM) Workshop , 2023.

Minsu Kim, Joohwan Ko, Taeyoung Yun, Dinghuai Zhang, Ling Pan, Woochang Kim, Jinkyoo Park, Emmanuel Bengio, and Yoshua Bengio. Learning to scale logits for temperatureconditional GFlowNets. In International Conference on Machine Learning , 2024.

Shuai Guo, Jielei Chu, Lei Zhu, and Tianrui Li. Dynamic backtracking in GFlowNet: Enhancing decision steps with reward-dependent adjustment mechanisms. arXiv preprint arXiv:2404.05576 , 2024.

Yibo Li, Liangren Zhang, and Zhenming Liu. Multi-objective de novo drug design with conditional graph generative model. Journal of Cheminformatics , 10:1-24, 2018.

Amr Alhossary, Stephanus Daniel Handoko, Yuguang Mu, and Chee-Keong Kwoh. Fast, accurate, and reliable molecular docking with QuickVina 2. Bioinformatics , 31(13):2214-2216, 2015.

Xiuyuan Hu, Guoqing Liu, Yang Zhao, and Hao Zhang. De novo drug design using reinforcement learning with multiple GPT agents. In Advances in Neural Information Processing Systems , volume 36, pages 7405-7418, 2023.

David M Rogers, Rupesh Agarwal, Josh V Vermaas, Micholas Dean Smith, Rajitha T Rajeshwar, Connor Cooper, Ada Sedova, Swen Boehm, Matthew Baker, Jens Glaser, et al. SARS-CoV2 billion-compound docking. Scientific Data , 10(1):173, 2023.

Tianfan Fu, Wenhao Gao, Connor Coley, and Jimeng Sun. Reinforced genetic algorithm for structure-based drug design. In Advances in Neural Information Processing Systems , volume 35, pages 12325-12338, 2022.

AkshatKumar Nigam, Robert Pollice, and Alán Aspuru-Guzik. Parallel tempered genetic algorithm guided by deep neural networks for inverse molecular design. Digital Discovery , 1(4):390-404, 2022.

Thibaut Vidal, Teodor Gabriel Crainic, Michel Gendreau, Nadia Lahrichi, and Walter Rei. A hybrid genetic algorithm for multidepot and periodic vehicle routing problems. Operations Research , 60(3):611-624, 2012.

Joshua Meyers, Benedek Fabian, and Nathan Brown. De novo molecular design and generative models. Drug discovery today , 26(11):2707-2715, 2021.

a.UnsupervisedPretraining

b. Genetic GFN Fine-tuning with Oracle

MaximumLikelihood

LTB+ αKL(πe(x) II πpre(x))

You

RU

G

GRU

GRU

G

BufferD

Rewardevaluation

ZINC250K(w/oOracle)

PolicyTpre

PolicyT

Graph-basedgeneticsearch

A Implementation details of Genetic GFN

Genetic GFN is implemented on top of the PMO benchmark source code (MIT license). 4 Mostly, we adopt the REINVENT implementation including the RNN models and experience buffer; the code is included in the benchmark, and the original implementation is also accessible with Apache-2.0 license. 5 See the following subsections for details.

A.1 Network architecture and pretraining

Network architecture. Our policy network is parameterized using a recurrent neural network containing multiple GRU cells [30]. In molecular optimization, RNN-based models with string molecular representations have proven to be successful [3, 17, 52]. In experiments, we employ the same hyperparameters to directly compare with REINVENT, whose input embedding dimension is 128 and hidden dimension is 512 with three layers.

Pretraining. According to the PMO benchmark guidelines [8], the pre-training is conducted on ZINC 250K. The overall framework is illustrated in Fig. 6. Since the network architecture is the same as REINVENT, we adopt the provided pretrained model for REINVENT in the PMO benchmark. This allows the direct comparison of fine-tuning approaches.

A.2 Hyperparameters

We mostly follow the hyperparmeter setup of REINVENT and GEGL. For instance, the batch size and learning rate are set as 64 and 0.0005 according to REINVENT in the PMO benchmark. On the other hand, the mutation rate and the number of training loops are set to 0.01 and 8 following GEGL. We use 64 samples for the replay training and population size, the same as the batch size without tuning. Lastly, the learning rate of Z , the partition function, is set to 0.1, also without tuning.

In contrast, we have searched several hyperparameters, offspring size, the number of GA generations, and KL-divergence coefficient α . We provide the sensitivity analysis for the offspring size and the number of GA generations in Appendix G.8. Furthermore, we use the inverse temperature β = 10 and the weight-shifting factor k = 0 . 01 , but they can be differently used to control score-diversity trade-off, as explained in Section 5.1.2.

A.3 Computing resource

Throughout the experiments, we utilize a 48-core CPU, Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz, and a single GPU. In the PMO benchmark, runtime varies from less than 10 minutes to several hours for 10K evaluations, depending on tasks and algorithms. However, most of the runtime is consumed in evaluating score functions-the motivation for why the sample efficiency matters. On the other hand, in the SARS-CoV-2 inhibitor design tasks, 1000 training steps with a batch size of 128 require more than 1 day for PlPro_7JIR and 2 days for RdRp_6YYT; more than 95% of the time is used to evaluation [52].

✓

✓

✓

✓

Ring crossover

Non-ring crossover

(a) Crossover

Delete atom

Append atom

Change bond order

Delete cyclic bond

Add ring

Change atom

(b) Mutations

HC

OH

HC—CH3

HO

HO

CH

HC=CH2

H2C

OH

1*

OH

HC

CH

CH

1*

HC

CH3

HC

CH3

HO-—CH3

HS ——CH3

OH

HN.

OH

HC

OH

HC

OH

B Genetic operations

This section details each operation in our genetic search; see Fig. 7 for illustration. Note that we adopt Graph GA of [13], which has demonstrated its powerful performances and has been adopted by GA-related works like Mol GA [16] and GEGL [17].

B.1 Crossover

A crossover operation is conducted to generate a new candidate (called offspring) by exchanging the genetic information of a pair of selected individuals (parents). This process mimics the crossover of genetic material in biological reproduction. In the context of molecular optimization with graphs, the crossover operation is conducted in two types: 'ring crossover' and 'non-ring crossover with a 50% probability.

These two main crossover operations perform crossover between two parent molecules by cutting and recombining ring substructures. Ring crossover performs a ring cut specifically designed to target ring structures within the molecule. The ring-cut operation cuts the molecule along two different ring patterns, selected randomly. One of the ring patterns checks for a specific arrangement of four consecutive ring atoms, and the other pattern checks for a ring atom connected to two other ring atoms with a single bond. If a suitable ring pattern is found, it cuts the molecule along that pattern, resulting in two fragments. On the other hand, non-ring crossover cuts a single bond, meaning it is not part of a cyclic (ring) structure within the molecule. The obtained fragments from both parents are recombined to create new molecules by applying predefined reaction SMARTS patterns. These operations are repeated for validity to ensure that the resulting molecules meet structural and size constraints.

B.2 Mutation

The mutation is a random change that is introduced to the genetic information of some individuals. This step adds diversity to the population and helps explore new regions of the solution space. In this work, we employ seven different mutation processes and randomly select one of these mutations to modify the offspring molecules slightly. The operations consist of atom and bond deletions, appending new atoms, inserting atoms between existing ones, changing bond orders, deleting cyclic bonds, adding cyclic rings, and altering atom types.

Deletion of atom: it selects one of five deletion SMARTS patterns, each representing the removal of a specific number of atoms or bonds. These patterns include the removal of a single atom, a single bond, a bond with two attached atoms, and bonds with multiple attached atoms. The selected pattern is applied to the molecule, deleting the specified atom(s) or bond(s).

Appending atom: it introduces a new atom to the molecule. The type of atom (e.g., C, N, O) and the type of bond (single, double, or triple) are chosen based on predefined probabilities. The function then generates a reaction SMARTS pattern to append the selected atom to the molecule, forming a new bond.

Inserting atom: it inserts a new atom between two existing atoms in the molecule. Similar to the appending atom, it selects the type of atom and bond based on predefined probabilities and generates a reaction SMARTS pattern to insert the atom.

Changing bond order: it randomly selects one of four SMARTS patterns, each representing a change in the bond order between two atoms. These patterns include changing a single bond to a double bond, a double bond to a triple bond, and vice versa.

Deletion of cyclic bond: it deletes a bond that is a part of a cyclic structure within the molecule. The SMARTS pattern represents the breaking of a cyclic bond while retaining the atoms connected by the bond.

Adding ring: it introduces a new cyclic ring into the molecule by selecting one of four SMARTS patterns, each representing the formation of a specific ring type. These patterns create different types of cyclic structures within the molecule.

Changing atom: it randomly selects two types of atoms from a predefined list and generates a SMARTS pattern to change one atom into another. This operation modifies the atom type within the molecule.

C Hyperparameter setup for baseline methods

In this subsection, we present detailed descriptions for hyperparameter tuning of baselines. Except for Mol GA and GEGL, we adopt all the hyperparameters, initial datasets, and pre-trained models provided in the PMO benchmark. For hyperparameters that affect the sample efficiency, such as population size, we have searched for the proper hyperparameters following the guidelines suggested by [8]. In detail, we tune Mol GA and GEGL on the average AUC Top-10, the main performance metric, from 3 independent runs of two oracles, zaleplon_mpo and perindopril_mpo . The best configurations are used in the main experiments.

Mol GA. As mentioned in Section 5.1, we set the offspring size as 5, the most crucial hyperparameter, according to the original paper [16]. Then, we searched the starting population size in [100, 200, 500, 1000] and the population size [100, 200, 500]. As shown in Fig. 8, we found the best configuration to be 500 and 100 for the starting population size and the population size, respectively.

population_size

starting_population_size

avg_auc

500

1,000-

1.115

450

900

1.110

1.105

400

800

1.100

700

1.095

350

600

1090

300

1.085

500

1.080

250

400

200

300

1.070

1.065

150

1.060

100

1.055

GEGL. In GEGL, the policy sampling size is set as the expert sampling size, and both priority queue sizes (denoted as num_keep ) are the same as the original implementation. Thus, we searched the expert sampling size in [64, 128, 512] and the priority queue size in [128, 512, 1024]. Originally, they were set as 8192 and 1024, which are improper to the sample efficient setting. For the training batch size, we use 64, which is the same as ours. We use the pretrained policy provided in the original code and adapt the setup of the rest of the hyperparameters, including mutation rate, learning rate, and the number of training loops. Based on the results in Fig. 9, we set the expert sampling size and priority queue size as 128.

xpert_sampling_batch_size

num_keep

avg_auc

260

550-

一

1.03

240

500

1.01

1.02

220

450

1.00

200

660

400

0.98

180

350

0.97

160

0.96

140

300

0.95

120

250

0.93

0.94

100

200

0.92

91

80

150

0.98

60

100

0.89

D Comparison Genetic GFN-AL with active learning and model-based algorithms

D.1 Implementation of Genetic GFN-AL

We mostly adopt the implementation of GFlowNet-AL, including the training method and the utilization of an acquisition function [10]. The source code is accessible online with an MIT license. 6 Additionally, the number of training proxy and generative models are aligned with the PMO benchmark standards, while the replay training and genetic search hyperparameters are set to those used in our method in Genetic GFN. The pseudo-code is as follows.

Hyperparmeters. We use the same hyperparameters for the generative model (i.e., Genetic GFN). Since active learning approaches introduce various hyperparameters, such as the training iterations of the proxy and generative models, not only introduce the proxy model, we tried to keep the setup of GFlowNet-AL in the PMO benchmark (e.g., proxy learning rate). We provide hyperparameters in Table 6; note that our proxy model predicts the score using SMILES rather than fragments, unlike the original GFlowNet-AL. Additionally, we adopt γ , the ratio of offline (oracle-touched) data in the inner loop training, and the acquisition function (related with κ ); see the original GFlowNet-AL paper [10].

D.2 Experimental results

Wecompare Genetic GFN-AL with various model-based and active learning methods. Here, GP BO is regarded as a model-based method of Graph GA because GP BO uses Graph when optimizing the GP acquisition function. As shown in Table 7, SMILES Genetic GFN-AL achieves significantly improved performance compared to fragment GFlowNet-AL in the PMO benchmark. It is noteworthy that we further utilize the acquisition function, not directly use the proxy prediction as a reward, and mix the oracle-touched data, as described in the previous section. Therefore, to verify the effectiveness of genetic search in the active learning setup, we implement another baseline, SMILES GFN-AL using

ϵ -greedy exploration, similar to our self-ablation studies in Section 5.1.2. The results demonstrate that Genetic GFN is beneficial to enhancing sample efficiency in the active learning setting. As pointed out in the PMO benchmark, though the model-based methods (or active learning methods) are known as more sample efficient, they require careful design to achieve superior performances.

E Multi-objective sample efficient molecular optimization tasks

This section provides details on multi-objective sample efficient molecular optimization experiments. Even though Genetic GFN targets single (i.e., scalar-valued) objective optimization tasks, ours directly applies to multi-objective tasks using well-defined coefficients. Inspired by the work of Zhu et al. [11], we conduct experiments on two multi-objective tasks: GSK3 β + JNK3 and GSK3 β + JNK3 + QED + SA. Genetic GFN is implemented on top of the original code (MIT license). 7

Interestingly, GSK3 β and JNK3 are machine-learning-based oracles that estimate inhibiting scores against the Glycogen synthase kinase 3 beta target and the c-Jun N-terminal kinase 3 target. Designing dual inhibitors for both targets can be beneficial to designing treatments for Alzheimer's Disease [50]. We define scalar-valued score functions using the linear combination of oracle functions with given coefficients. The cost coefficients α are set following the HN-GFN paper, i.e., α = (1 , 1) for GSK3 β + JNK3 and α = (3 , 4 , 2 , 1) for GSK3 β + JNK3 + QED + SA. Since the reward scales become larger, we reduce the inverse temperature from 50 to 25.

According to the experiment setup of hypernetwork-based GFlowNets (HN-GFN) [11], we limit the reward calls to 1000. Consequently, the batch size, replay training loops, population, and offspring size are adjusted to half. The performance is measured using hypervolumes with five independent runs. Note that we use the same pretrained model in the main experiment for PMO.

F Designing of SAS-Cov-2 inhibitors

According to the previous work [52], we define the score function s ( x ) for designing SARS-Cov-2 inhibitors as a linear combination of normalized scores, i.e.,

The target proteins are as follows:

PLPro_7JIR : PLPro (papain-like protease) is a critical enzyme in the life cycle of SARSCoV-2, which can help in studying the enzyme's function and in designing inhibitors that could potentially disrupt the virus's ability to replicate and evade the immune system. The 7JIR represents a C111S mutant version of PLPro. 8

RdRp_6YYT : RdRp (RNA-dependent RNA polymerase) is essential for the replication of the genome and the transcription of genes in SARS-CoV-2. The protein structure of RdRp is cataloged in the Protein Data Bank (PDB) under the identification code 6YYT. 9

Genetic GFN is implemented on top of the implementation of MolRL-MGPT (Molecular design using Reinforcement Learning with Multiple GPT agents). 10 We employ the same hyperparameters with the experiments on the PMO benchmark, except for the weight-shifting factor (we use k = 0 . 05 ).

In Table 5, the scores of baselines are computed according to the Eq. (5) using the average values in the MolRL-MGPT [52]. We also provide average scores and standard deviation of docking, QED, SA, and diversity in Table 8 and Table 9.

We provide additional visual results in Fig. 10. There seems to be a trend of increasing molecular complexity and functional diversity over iterations.

50 steps

50 steps

100 steps

100 steps

500 steps

500 steps

1000 steps

1000 steps

RdRp

RdRp

(a) PLPro_7JIR

(b) RdRp_6YYT

G Additional results

G.1 Controllability of the score-diversity using the weight-shifting factor k

As mentioned in Section 5.1.2, the weight-shifting factor k in Eq. (4) also can control the scorediversity trade-off. Increasing k gives more diverse candidates by increasing the probability of high-ranked samples. Similar to adjusting the inverse temperature, the results in Table 10 and Fig. 11 demonstrate that adjusting k with fixed β also can effectively control the score-diversity trade-off.

0.45

0.50

0.55

0.60

0.65

0.70

Diversity

16.0

16.5

17.0

17.5

Avg. Top-10

Genetic GFN (

k

=0.01)

Genetic GFN (

k

=0.05)

Genetic GFN (

k

=0.1)

Genetic GFN (

k

=0.5)

SMILES-REINVENT

GP BO

G.2 Additional results for the PMO benchmark

G.2.1 Statistical analysis

We provided the results of the t-tests of AUC Top-10 with Mol GA (2nd place) and SMIELS REINVENT (3rd place). As shown in Table 11, both p-values are less than 0.05, so Genetic GFN outperforms baselines with statistical significance.

G.2.2 Full results of the PMO benchmark

0

2000

4000

6000

8000

10000

0.4

0.6

0.8

1.0

albuterol_similarity

Genetic GFN (Ours)

SMILES-REINVENT

GP BO

SELFIES-REINVENT

Graph GA

SMILES-HC

STONED

SynNet

SMILES GA

GFlowNet

GFlowNet-AL

0

2000

4000

6000

8000

10000

0.3

0.4

0.5

0.6

0.7

0.8

0.9

amlodipine_mpo

0

2000

4000

6000

8000

10000

0.2

0.4

0.6

0.8

celecoxib_rediscovery

0

2000

4000

6000

8000

10000

0.6

0.7

0.8

0.9

deco_hop

0

2000

4000

6000

8000

10000

0.0

0.2

0.4

0.6

0.8

1.0

drd2

0

2000

4000

6000

8000

10000

0.6

0.7

0.8

0.9

fexofenadine_mpo

0

2000

4000

6000

8000

10000

0.2

0.4

0.6

0.8

1.0

gsk3b

0

2000

4000

6000

8000

10000

0.0

0.2

0.4

0.6

0.8

1.0

isomers_c7h8n2o2

0

2000

4000

6000

8000

10000

0.0

0.2

0.4

0.6

0.8

1.0

isomers_c9h10n2o2pf2cl

0

2000

4000

6000

8000

10000

0.0

0.2

0.4

0.6

0.8

1.0

jnk3

0

2000

4000

6000

8000

10000

0.1

0.2

0.3

0.4

median1

0

2000

4000

6000

8000

10000

0.15

0.20

0.25

0.30

0.35

median2

0

2000

4000

6000

8000

10000

0.2

0.4

0.6

0.8

1.0

mestranol_similarity

0

2000

4000

6000

8000

10000

0.6

0.7

0.8

0.9

osimertinib_mpo

0

2000

4000

6000

8000

10000

0.3

0.4

0.5

0.6

0.7

perindopril_mpo

0

2000

4000

6000

8000

10000

0.65

0.70

0.75

0.80

0.85

0.90

0.95

qed

0

2000

4000

6000

8000

10000

0.2

0.4

0.6

0.8

ranolazine_mpo

0

2000

4000

6000

8000

10000

0.4

0.5

0.6

0.7

0.8

scaffold_hop

0

2000

4000

6000

8000

10000

0.0

0.2

0.4

0.6

0.8

sitagliptin_mpo

0

2000

4000

6000

8000

10000

0.2

0.3

0.4

0.5

0.6

0.7

thiothixene_rediscovery

0

2000

4000

6000

8000

10000

0.2

0.3

0.4

0.5

0.6

0.7

troglitazone_rediscovery

0

2000

4000

6000

8000

10000

0.2

0.0

0.2

0.4

0.6

valsartan_smarts

0

2000

4000

6000

8000

10000

0.1

0.2

0.3

0.4

0.5

0.6

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

G.2.3 Ranks with various metrics

According to the PMO benchmark, we also provide the rank of each method with various metrics. The results in Table 14 show that Genetic GFN achieves the first place in total, not only in the AUC Top-10.

G.3 Further studies on valsartan_smarts (#22)

Notably, we have observed that only a few methods achieve non-zero scores on valsartan_smarts (#22) in Table 1. The valsartan SMARTS targets molecules containing a SMARTS pattern related to valsartan while being characterized by physicochemical properties corresponding to the sitagliptin molecule [57]. It measures the arithmetic means of several scores, including (1) binary score about whether it contains a certain SMARTS structure, (2) LogP, (3) TPSA, and (4) Bertz score. Since we utilize a TDC oracle function for evaluations, we provide our empirical observations here.

Due to the binary score (1 if the certain SMARTS pattern is included), many tries terminate with 0. Especially with a limited number of oracle calls, generating molecules containing a certain substructure is notoriously hard. Other literature shows that other methods achieve high scores with more oracle calls [52]. With 10K calls, even REINVENT and Genetic only succeed in finding non-zero score molecules once out of five independent runs. Another observation is that methods (REINVENT, Genetic GFN, and GEGL) achieving non-zero scores all generate SMILES with RNN-based models. Thus, we have a conjecture that SMILES generation is effective in generating a certain SMARTS pattern. We provide examples of generated molecules with non-zero valsartan_smarts scores. Note that the other four seeds failed. Each run generates similar molecules (see Top1,10,100 samples in Fig. 13 in the additional material), but the samples between the two runs (REINVENT and Genetic GFN) have different structures (the molecule distance between Top1 samples is 0.854).

(a) REINVENT (

seed

0

)

(b) Genetic GFN (

seed

4

)

NH,

NH

HN

Top1:0.987

Top10:0.981

Top100:0.799

Top1:0.987

Top10:0.981

Top100:0.799

G.4 Additional results for ablation studies

G.4.1 Statistical analysis

We provided the results of the t-tests of ablation studies. As shown in Table 15, the p-values for ablating genetic search are less than 0.05, so our genetic search is a statistically significant component.

G.5 Results for diversity and synthesizability

We report the diversity and synthetic accessibility (SA) score of Top-100 molecules on each oracle.

albuterol_similarity

amlodipine_mpo

celecoxib_rediscovery

deco_hop

drd2

fexofenadine_mpo

gsk3b

isomers_c7h8n2o2

isomers_c9h10n2o2pf2cl

jnk3

median1

median2

mestranol_similarity

osimertinib_mpo

perindopril_mpo

qed

ranolazine_mpo

scaffold_hop

sitagliptin_mpo

thiothixene_rediscovery

troglitazone_rediscovery

valsartan_smarts

Genetic GFN (Ours)

SMILES-REINVENT

GPBO

SELFIES-REINVENT

Graph GA

SMILES-HC

STONED

SynNet

SMILES GA

GFlowNet

GFlowNet-AL

Avg. Diversity of T op-100

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

albuterol_similarity

amlodipine_mpo

celecoxib_rediscovery

deco_hop

drd2

fexofenadine_mpo

gsk3b

isomers_c7h8n2o2

isomers_c9h10n2o2pf2cl

jnk3

median1

median2

mestranol_similarity

osimertinib_mpo

perindopril_mpo

qed

ranolazine_mpo

scaffold_hop

sitagliptin_mpo

thiothixene_rediscovery

troglitazone_rediscovery

valsartan_smarts

Genetic GFN (Ours)

SMILES-REINVENT

GPBO

SELFIES-REINVENT

Graph GA

SMILES-HC

STONED

SynNet

SMILES GA

GFlowNet

GFlowNet-AL

Avg. SA Score

3

4

5

6

7

8

9

10

G.6 Genetic GFN with SELFIES generation

As our method employs a string-based sequence generation model, it is applicable to SELFIES representation. Following the same procedure and setup described in Section 3 and Section 5.1. The results in Table 16 demonstrate that Genetic GFN significantly outperforms SELFIES-REINVENT by achieving 14.986 compared to 14.152, not only the other high-ranked method, GP BO (14.264). Notably, our AUC Top-10 is higher than that of SELFIES-REINVENT in 19 oracles.

G.7 Genetic GFN with string-based genetic search

We also additionally provide experiments that incorporate STONED (GA with SELFIES)[14] as an exploration strategy to guide GFN training instead of Graph GA. Note that STONED only utilizes mutations since designing valid crossover with string representation is challenging.

G.8 Experiments with varying hyperparameters

In this subsection, we verify the robustness of Genetic GFN for differing hyperparameter setups. We conduct experiments by varying the number of GA generation (refining loops), offspring size, and the number of training inner loops. The results show that our results are robust to each hyperparameter setup by achieving similar or better performance compared to Mol GA in all tested configurations.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state our main research claim, which is consistent with the experimental results.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We include our limitations and future works in the discussion section.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide implementation details, including pseudo-codes.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the anonymized link of our implementation. Also, the paper uses the public dataset and benchmark; we provide accessible links in the supplemental material.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all details regarding experiments in the appendix and the anonymized codes.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The experimental results include standard deviation with independent multiple runs. In addition, statistical tests are conducted for the main results and self-ablation studies.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the computer resources in the appendix (due to lack of spaces).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have checked the paper according to the NeurIPS Code of Ethics; our study does not include human participation and privacy-related data.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impact of this study at the end of the paper.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our study is conducted using the public dataset, and all experimental tasks are from previous publications.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper clearly states the source and license of the original code, data, and models in the appedix.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our experiments are conducted using the already published datasets and benchmarks.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our study does not involve crowdsourcing nor human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our study does not involve crowdsourcing nor human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors present a sample-efficient molecular optimization method using GFlowNets and a genetic algorithm. The authors demonstrate its effectiveness in generating inhibitors against SARS-CoV-2, with fewer reward calls than other methods, as well as on the PMO benchmark, where the method outperforms all other methods on the 23 tasks (at least on top-1, -10, and -100 AUC). The paper is strong, novel, and well-written, with also very nice figures which are also well-annotated. Sadly, no code provided.\n\nSTRENGTHS:\n* A molecular optimization method that actually computes the sample efficiency - well done! And great results outperforming SOTA with a well-motivated strategy.\n* The authors postulate that genetic algorithms can more effectively navigate the chemical space through domain-specific genetic operations, something which deep generative models generally lack. As such, their approach leverages the ability of GFlowNets to generate diverse molecules, while instead leveraging the optimization power of genetic algorithms. Neat idea.\n* The ability for users to control the score-diversity trade-off via the inverse temperature parameter is very useful. Nonetheless, I believe that some of the other methods shown in Figure 3 also have that feature (e.g., REINVENT4 also has a controllable temperature parameter), which many be misleading in the current text/plot that suggests only the genetic GFN has it.\n\nWEAKNESSES:\n* How sensitive is the genetic GFN to the hyperparameters? I saw that many defaults were chosen to compare directly to REINVENT, did these also happen to be good options for GGFN or could even better results have been obtained? Some discussion around hyperparameter sensitivity would be interesting, perhaps I missed it. Similarly, a discussion around the computational complexity of the model would be interesting.\n\nQUESTIONS:\n* In Table 1, the authors demonstrate that their approach outperforms MolGA and REINVENT on most tasks from the PMO benchmark when using the AUC top-10 metric. Do these results still hold for the AUC top-1 (Table 14 suggests they do), and if so, is it by similar amounts or are the gaps wider between the top 3 models?\n* How does the model scale with the number of atoms? For instance, can it scale well to other modalities or macromolecules?\n* It is interesting to see that for task #22 (valsartan_smarts), only the genetic GFN and REINVENT perform somewhat well on this. Do the authors have any idea why the genetic GFN and REINVENT are the only models which do not completely fail on this task? I think it would be interesting to dig more deeply into some of the specific oracles/tasks and look at the molecules being generated by the genetic GFN and REINVENT - for instance, do they find the same solutions here?\n\nLIMITATIONS:\n* Despite the excellent results, the authors fail to provide any accompanying code, which is a shame (the link in the paper points to a non-existing anonymized repo). The lack of openly-shared code casts doubt on the performance of the model and some of the claims made in the paper, hence the reason for the low score. Hard to say how well-documented the code is or how useful it will be to researchers.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach called Genetic-guided GFlowNets (Genetic GFN) for sample-efficient molecular optimization. The method integrates domain-specific genetic algorithms to guide a GFlowNet policy toward higher-reward molecular samples.\n\nSTRENGTHS:\n- The paper is very pedagogical and easy to read, clearly explaining the proposed method and its rationale.\n- Extensive experiments demonstrate the effectiveness of Genetic GFN, showing state-of-the-art performance on benchmark tasks.\n- The approach offers a promising direction for enhancing sample efficiency in molecular design.\n- The method shows promising results in designing SARS-CoV-2 inhibitors, demonstrating potential real-world impact.\n- The approach offers controllability of the score-diversity trade-off, which is valuable for practical applications.\n\nWEAKNESSES:\n- The proposed method is limited to molecular optimization and is not readily generalizable to other domains.\n- While the graph GA is based on prior work, the paper would benefit from being more self-contained by including an algorithm or visualization of the mutation and crossover steps. For example, how does the GA ensures the validity of the molecules.\n- A schematic diagram illustrating the entire training process, including both pretraining and GFlowNet training, would improve clarity.\n- The anonymous link to code provided is not accessible, limiting reproducibility.\n- Lack of efficiency/runtime analysis and comparison with the baseline methods\n\nQUESTIONS:\n- Is the GA implemented on CPU or GPU? If it’s on CPU, how slow it is?\n- Would the model's performance improve if combined with modern RNN architectures like Mamba or xLSTM?\n- How sensitive is the graph genetic algorithm to the way molecules are fragmented and the resolution of fragmentation?\n- How might this approach be adapted to other discrete optimization problems beyond molecular design? (such as TSP in combinatorial optimization.  A quick intuition would be sufficient)\n- How does the method's performance scale with the size and complexity of the molecular space being explored?\n- Is there potential for incorporating human feedback or preferences into the optimization process?\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a combination of genetic algorithm (GA) and GFlowNets for molecular optimization, with an emphasis on sample efficiency (achieving high property value by few number of reward calls). The key motivation is that GA can incorporate domain specific knowledge by designing the mutation operations, which is the key to improve sample efficiency, while GFlowNets model the overall distribution of the molecular space in a data-driven manner. The pipeline thus consists of two components: a SMILES-based generative model using GFlowNets and a graph-based genetic search. Several techniques for training this pipeline, including unsupervised pretraining, experience buffer for off-policy training, and TB + KL loss functions, are proposed. Experimental results on PMO benchmark and in silico design for SARS-CoV-2 inhibitors are conducted to demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n- The proposed method is simple and easy to follow. Though there is a small gap between the motivation and the proposed method (see Weakness), overall I buy the story of combining domain-specific knowledge with pure data-driven methods; the proposed combination of GA and GFlowNets makes sense to me.\n\n- The authors clearly have put a lot of efforts on conducting and designing the experiments, for both the main results and the ablation studies. The proposed study on designing inhibitors against SARS-CoV-2 looks interesting to me. Though it is hard for me to judge whether the in silico score functions indeed have a strong correlation with real-world performance in the biological sense, including such efforts/results is already something intriguing from a machine learning perspective.\n\n- The paper is in a good shape. The authors did a good job on describing necessary background knowledge for GFlowNet. Sufficient technical details are provided in the paper.\n\nWEAKNESSES:\n- One weakness is the gap between \"domain-specific knowledge\" and \"GA method\" mentioned in the introduction. When I read the introduction, I am expecting the authors will propose some new genetic operators that are well-designed and specific to domain-specific tasks, with clear indicator that these operators have strong correlation with domain-specific knowledge. It turns out that the authors are still using the standard genetic operators such as crossover and mutation. Are there any special care taken that I may have missed? for example, the crossover fragments are not purely random but are collected from motifs specific to tasks. If no, I feel the statement of \"GA method can encode domain-specific knowledge\" is very vague, or at least it not being discussed comprehensively if previous works have explored. \n\n- Another thing missing in the paper is enough qualitative results. I can only see Fig 4 & 5 contain some final generated molecules for two targets. It will be great if the authors can include more visual results, especially the trajectory of the sampled molecules, with highlights on what fragments have been changed during the process. Some analysis on why certain fragments are favored (if any) or remain at the final optimized structure is also very helpful for the readers to verify the effectiveness of the proposed method.\n\n- Another interesting study to show is how the proposed pipeline can be incorporated with grammar-based representations, such as STONED, SELFIES, and many more if search \"molecular grammar\", rather than SMILES. Since these manually designed grammar consists of more \"domain-specific knowledge\" compared to pure atom-based SMILES string, I would expect the experimental results will be better. It would be great to include such analysis in the paper to provide more evidence on the key motivation of the paper.\n\nQUESTIONS:\n- line 55, please remove \"expectional\"\n- line 118, no references to \"previous works\"\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This work proposes a variant of GFlowNet, called genetic GFN, for molecular property optimization. Specifically, the authors use genetic search to guide the GFlowNet to explore high-reward regions, addressing the over-exploration issue in GFlowNet. Besides, the authors incorporate some effective training strategies to improve the performance further. The proposed genetic GFN achieves achieves state-of-the-art performance in practical molecular optimization (PMO) benchmark.

STRENGTHS:
- This paper studies an important scientific problem, i.e., molecular optimization.
- The paper is well-organized and easy to follow.
- The experimental results are comprehensive and strong.

WEAKNESSES:
- The novelty is somewhat limited, as the proposed Genetic GFN simply combines some existing techniques, that have been studied independently.
- In Table 4, Genetic GFN is worse than Graph GA and MARS on GSK3 + JNK3. The reason should be discussed.
- I am more curious about the difference between increasing KL term in loss function and directly using the logP of the reference distribution as part of the reward (Please see equation (4) in DPO [1]). In this situation, what's the different between GFlowNet and PPO.

[1] Direct preference optimization: Your language model is secretly a reward model

QUESTIONS:
See weaknesses

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12