Title: Sample-efficient Multi-objective Molecular Optimization with GFlowNets
Venue: NeurIPS 2023
Accepted: yes
Keywords: drug discovery

multi-objective molecular optimization

Bayesian optimization

generative flow networks
Abstract: Many crucial scientific problems involve designing novel molecules with desired properties, which can be formulated as a black-box optimization problem over the *discrete* chemical space. In practice, multiple conflicting objectives and costly evaluations (e.g., wet-lab experiments) make the *diversity* of candidates paramount. Computational methods have achieved initial success but still struggle with considering diversity in both objective and search space. To fill this gap, we propose a multi-objective Bayesian optimization (MOBO) algorithm leveraging the hypernetwork-based GFlowNets (HN-GFN) as an acquisition function optimizer, with the purpose of sampling a diverse batch of candidate molecular graphs from an approximate Pareto front. Using a single preference-conditioned hypernetwork, HN-GFN learns to explore various trade-offs between objectives. We further propose a hindsight-like off-policy strategy to share high-performing molecules among different preferences in order to speed up learning for HN-GFN. We empirically illustrate that HN-GFN has adequate capacity to generalize over preferences. Moreover, experiments in various real-world MOBO settings demonstrate that our framework predominantly outperforms existing methods in terms of candidate quality and sample efficiency. The code is available at https://github.com/violet-sto/HN-GFN.

Sample-efficient Multi-objective Molecular Optimization with GFlowNets

Abstract

Many crucial scientific problems involve designing novel molecules with desired properties, which can be formulated as a black-box optimization problem over the discrete chemical space. In practice, multiple conflicting objectives and costly evaluations (e.g., wet-lab experiments) make the diversity of candidates paramount. Computational methods have achieved initial success but still struggle with considering diversity in both objective and search space. To fill this gap, we propose a multi-objective Bayesian optimization (MOBO) algorithm leveraging the hypernetwork-based GFlowNets (HN-GFN) as an acquisition function optimizer, with the purpose of sampling a diverse batch of candidate molecular graphs from an approximate Pareto front. Using a single preference-conditioned hypernetwork, HN-GFN learns to explore various trade-offs between objectives. We further propose a hindsight-like off-policy strategy to share high-performing molecules among different preferences in order to speed up learning for HNGFN. We empirically illustrate that HN-GFN has adequate capacity to generalize over preferences. Moreover, experiments in various real-world MOBO settings demonstrate that our framework predominantly outperforms existing methods in terms of candidate quality and sample efficiency. The code is available at https://github.com/violet-sto/HN-GFN .

1 Introduction

Designing novel molecular structures with desired properties, also referred to as molecular optimization, is a crucial task with great application potential in scientific fields ranging from drug discovery to material design. Molecular optimization can be naturally formulated as a black-box optimization problem over the discrete chemical space, which is combinatorially large [52]. Recent years have witnessed the trend to leverage computational methods, such as deep generative models [31] and combinatorial optimization algorithms [62, 30], to facilitate optimization. However, the applicability of most prior approaches in real-world scenarios is hindered by two practical constraints: (i) Chemists commonly seek to optimize multiple properties of interest simultaneously [60, 33]. For example, in addition to effectively inhibiting a disease-associated target, an ideal drug is desired to be easily synthesizable and non-toxic. Unfortunately, as objectives often conflict, in most cases, there is no single

𝑀:𝜇,𝜎

Training :

𝜆~𝐷𝑖𝑟 𝑎

Sampling :

𝜆𝑡𝑎𝑟𝑔𝑒𝑡

∈ Λ

Hypernetwork

ℎ(∙ ; 𝜙)

𝜃𝑝𝑟𝑒𝑑

𝒇𝟏

𝒇𝟐

…

𝒇𝑴

?

?

…

?

=

×𝑏

represents the initial state and        represents the complete object.

GFlowNet

𝜋𝜃

:

𝜃 = (𝜃 𝑚𝑝𝑛𝑛 , 𝜃

𝑝𝑟𝑒𝑑 )

...

…

𝑆1

𝑆0

…

...

...

𝑆2

𝑆𝑛-1

...

...

𝑅𝜆

𝑥1

𝑅𝜆

𝑥2

𝑅𝜆

𝑥3

Evaluated batch

𝑅𝜆

𝑥

= 𝑎 𝛽 (𝜇

𝑠 𝜆

𝑥

, 𝜎

𝑠 𝜆

𝑥

; 𝛭)

optimal solution, but rather a set of Pareto optimal solutions defined with various trade-offs [18, 45]. (ii) Realistic oracles (e.g., wet-lab experiments and high-fidelity simulations) require substantial costs to synthesize and evaluate molecules [22]. Hence, the number of oracle evaluations is notoriously limited. In such scenarios, the diversity of candidates is a critical consideration.

Bayesian optimization (BO) [34, 54] provides a sample-efficient framework for globally optimizing expensive black-box functions. The core idea is to construct a cheap-to-evaluate surrogate model approximating the true objective function (also known as the oracle ) on the observed dataset and optimize an acquisition function (built upon the surrogate model) to obtain informative candidates with high utility for the next round of evaluations. Due to the common large-batch and low-round settings in biochemical experiments [3], batch BO is prioritized to shorten the entire cycle of optimization [25]. MOBOalso received broad attention and achieved promising performance in continuous problems by effectively optimizing differentiable acquisition functions [12, 13]. Nevertheless, it is less prominent in discrete problems, where no gradients can be leveraged to navigate the search space.

Although existing discrete molecular optimization methods can be adopted as the acquisition function optimizer to alleviate this issue, they can hardly consider diversity in both search and objective space: 1) many neglect the diversity of proposed candidates in search space [30, 33]; 2) many multi-objective methods simply rely on a predefined scalarization function (e.g., mean) to turn the multi-objective optimization (MOO) problem into a single-objective one [61, 20]. As the surrogate model cannot exactly reproduce the oracle's full behaviors and the optimal trade-off is unclear before optimization (even with domain knowledge), it is desired to not only propose candidates that bring additional information about the search space but also explore more potential trade-offs of interest. To achieve this goal, we explore how to extend the recently proposed generative flow networks (GFlowNets) [6, 7], a class of generative models that aim to learn a stochastic policy for sequentially constructing objects with a probability proportional to a reward function, to facilitate multi-objective molecular optimization. Compared with traditional combinatorial optimization methods, GFlowNets possess merit in generating diverse and high-reward objects, which has been verified in single-objective problems [6, 28].

In this work, we present a MOBO algorithm based on GFlowNets for sample-efficient multi-objective molecular optimization. We propose a hypernetwork-based GFlowNet (HN-GFN) as the acquisition function optimizer within MOBO to sample a diverse batch of candidates from an approximate Pareto front. Instead of defining a fixed reward function as usual in past work [6], we train a unified GFlowNet on the distribution of reward functions ( random scalarizations parameterized by preference vectors ) and control the policy using a single preference-conditioned hypernetwork. While sampling candidates, HN-GFN explores various trade-offs between competing objectives flexibly by varying the input preference vector. Inspired by hindsight experience replay [2], we further introduce a hindsight-like off-policy strategy to share high-performing molecules among different preferences

and speed up learning for HN-GFN. As detailed in our reported experiments, we first empirically verify that HN-GFN is capable of generalizing over preference vectors, then apply the proposed framework to real-world scenarios and demonstrate its effectiveness and efficiency over existing methods. Our key contributions are summarized below:

We introduce a GFlowNet-based MOBO algorithm to facilitate real-world molecular optimization. We propose HN-GFN, a conditional variant of GFlowNet that can efficiently sample candidates from an approximate Pareto front.

We introduce a hindsight-like off-policy strategy to speed up learning in HN-GFN.

Experiments verify that our MOBO algorithm based on HN-GFN can find a high-quality Pareto front more efficiently compared to state-of-the-art baselines.

2 Related Work

Molecular optimization. Molecular optimization has been approached with a wide variety of computational methods, which can be mainly grouped into three categories: 1) Latent space optimization (LSO) methods [24, 31, 58, 66] perform the optimization over the low-dimensional continuous latent space learned by generative models such as variational autoencoders (V AEs) [36]. These methods require the latent representations to be discriminative, but the training of the generative model is decoupled from the optimization objectives, imposing challenges for optimization [58]. Instead of navigating the latent space, combinatorial optimization methods search for the desired molecular structures directly in the explicit discrete space with 2) evolutionary algorithms [30] and 3) reinforcement learning (RL) [62] guiding the search. However, most prior methods only focus on a single property, from non-biological properties such as drug-likeliness (QED) [8] and synthetic accessibility (SA) [19], to biological properties that measure the binding energy to a protein target [6]. Multi-objective molecular optimization has recently received wide attention [33, 61, 20]. For example, MARS [61] employs Markov chain Monte Carlo (MCMC) sampling to find novel molecules satisfying several properties. However, most approaches require a notoriously large number of oracle calls to evaluate molecules on the fly [33, 61]. In contrast, we tackle this problem in a sample-efficient manner.

GFlowNet. GFlowNets [6, 7] aim to sample composite objects proportionally to a reward function, instead of maximizing it as usual in RL [56]. GFlowNets are related to the MCMC methods due to the same objective, while amortizing the high cost of sampling (mixing between modes) over training a generative model. GFlowNets have made impressive progress in various applications, such as biological sequence design [28], discrete probabilistic modeling [63], and Bayesian structure learning [15]. While the concept of conditional GFlowNet was originally discussed in Bengio et al. [7], we are the first to study and instantiate this concept for MOO, in parallel with Jain et al. [29]. Compared with them, we delicately design the conditioning mechanism and propose a hindsightlike off-policy strategy that is rarely studied for MOO (in both the RL and GFlowNet literature). Appendix D provides a detailed comparison.

Bayesian optimization for discrete spaces. The application of BO in discrete domains has proliferated in recent years. It is much more challenging to construct surrogate models and optimize acquisition functions in discrete spaces, compared to continuous spaces. One common approach is to convert discrete space into continuous space with generative models [24, 16, 44]. Besides, one can directly define a Gaussian Process (GP) with discrete kernels [46, 53] and solve the acquisition function optimization problem with evolutionary algorithms [35, 57].

Multi-objective Bayesian optimization. BOhas been widely used in MOO problems for efficiently optimizing multiple competing expensive black-box functions. Most popular approaches are based on hypervolume improvement [12], random scalarizations [37, 49], and entropy search [27, 5]. While there have been several approaches that take into account parallel evaluations [11, 38] and diversity [38], they are limited to continuous domains.

3 Background

3.1 Problem formulation

Weaddress the problem of searching over a discrete chemical space X to find molecular graphs x ∈ X that maximize a vector-valued objective f ( x ) = ( f 1 ( x ) , f 2 ( x ) , . . . , f M ( x ) ) : X → R M , where f m is a black-box function (also known as the oracle) evaluating a certain property of molecules. Practically, realistic oracles are extremely expensive to evaluate with either high-fidelity simulations or wet-lab experiments. We thus propose to perform optimization in as few oracle evaluations as possible, since the sample efficiency is paramount in such a scenario.

There is typically no single optimal solution to the MOO problem, as different objectives may contradict each other. Consequently, the goal is to recover the Pareto front - the set of Pareto optimal solutions which cannot be improved in any one objective without deteriorating another [18, 45]. In the context of maximization, a solution f ( x ) is said to Pareto dominates another solution f ( x ′ ) iff f m ( x ) ≥ f m ( x ′ ) ∀ m = 1 , . . . , M and ∃ m ′ such that f m ′ ( x ) > f m ′ ( x ′ ) , and we denote f ( x ) ≻ f ( x ′ ) . A solution f ( x ∗ ) is Pareto optimal if not Pareto dominated by any solution. The Pareto front can be written as P ∗ = { f ( x ∗ ) : { f ( x ) : f ( x ) ≻ f ( x ∗ ) } = ∅} .

The quality of a finite approximate Pareto front P is commonly measured by hypervolume [67] - the M-dim Lebesgue measure λ M of the space dominated by P and bounded from below by a given reference point r ∈ R M : HV ( P , r ) = λ M ( ∪ |P| i =1 [ r , y i ]) , where [ r , y i ] denotes the hyper-rectangle bounded by r and y i = f ( x i ) .

3.2 Batch Bayesian optimization

Bayesian optimization (BO) [54] provides a model-based iterative framework for sample-efficient black-box optimization. Given an observed dataset D , BO relies on a Bayesian surrogate model M to estimate a posterior distribution over the true oracle evaluations. Equipped with the surrogate model, an acquisition function a : X → R is induced to assign the utility values to candidate objects for deciding which to be evaluated next on the oracle. Compared with the costly oracle, the cheap-to-evaluate acquisition function can be efficiently optimized. We consider the scenario where the oracle is given an evaluation budget of N rounds with fixed batches of size b .

To be precise, we have access to a random initial dataset D 0 = { ( x 0 i , y 0 i ) } n i =1 , where y 0 i = f ( x 0 i ) is a true oracle evaluation. In each round i ∈ { 1 , . . . , N } , the acquisition function is maximized to yield a batch of candidates B i = { x i j } b j =1 to be evaluated in parallel on the oracle y i j = f ( x i j ) . The observed dataset D i -1 is then augmented for the next round: D i = D i -1 ∪ { ( x i j , y i j ) } b j =1 .

4 Method

In this section, we present the proposed MOBO algorithm based on hypernetwork-based GFlowNet (HN-GFN), shown in Figure 1. Due to the space limitation, we present the detailed algorithm in Appendix A. Our key idea is to extend GFlowNets as the acquisition function optimizer for MOBO, with the objective of sampling a diverse batch of candidates from the approximate Pareto front. To begin, we introduce GFlowNets in the context of molecule design, then describe how GFlowNet can be biased by a preference-conditioned hypernetwork to sample molecules according to various trade-offs between objectives. Next, we propose a hindsight-like off-policy strategy to speed up learning in HN-GFN. Finally, we introduce the surrogate model and acquisition function.

4.1 Preliminaries

GFlowNets [6] seek to learn a stochastic policy π for sequentially constructing discrete objects x ∈ X with a probability π ( x ) ∝ R ( x ) , where X is a compositional space and R : X → R ≥ 0 is a non-negative reward function. The generation process of object x ∈ X can be represented by a sequence of discrete actions a ∈ A that incrementally modify a partially constructed object, which is denoted as state s ∈ S . Let the generation process begin at a special initial state s 0 and terminate with a special action indicating that the object is complete ( s = x ∈ X ) , the construction of an object x can be defined as a complete trajectory τ = ( s 0 → s 1 →··· → s n = x ) .

Following fragment-based molecule design [6, 61], the molecular graphs are generated by sequentially attaching a fragment, which is chosen from a predefined vocabulary of building blocks, to an atom of the partially constructed molecules. The maximum trajectory length is 8, with the number of actions varying between 100 and 2000 depending on the state, making |X| up to 10 16 . There are multiple action sequences leading to the same state, and no fragment removal actions, the space of possible action sequences can thus be denoted by a directed acyclic graph (DAG) G = ( S , E ) , where the edges in E are transitions s → s ′ from one state to another. To learn the aforementioned desired policy, Bengio et al. [6] propose to see the DAG structure as a flow network .

Markovian flows. Bengio et al. [7] first define a trajectory flow F : T → R ≥ 0 on the set of all complete trajectories T to measure the unnormalized density. The edge flow and state flow can then be defined as F ( s → s ′ ) = ∑ s → s ′ ∈ τ F ( τ ) and F ( s ) = ∑ s ∈ τ F ( τ ) , respectively. The trajectory flow F determines a probability measure P ( τ ) = F ( τ ) ∑ τ ∈T F ( τ ) . If flow F is Markovian , the forward transition probabilities P F can be computed as P F ( s ′ | s ) = F ( s → s ′ ) F ( s ) .

Flow matching. A flow is consistent if the following flow consistency equation is satisfied ∀ s ∈ S :

where Pa G ( s ) is a set of parents of s in G . As proved in Bengio et al. [6], if the flow consistency equation is satisfied with R ( s ) = 0 for non-terminal state s and F ( x ) = R ( x ) ≥ 0 for terminal state x , a policy π defined by the forward transition probability π ( s ′ | s ) = P F ( s ′ | s ) samples object x with a probability π ( x ) ∝ R ( x ) . GFlowNets propose to approximate the edge flow F ( s → s ′ ) using a neural network F θ ( s, s ′ ) with enough capacity, such that the flow consistency equation is respected at convergence. To achieve this, Bengio et al. [6] define a temporal difference-like [56] learning objective, called flow-matching (FM):

One can use any exploratory policy ˜ π with full support to sample training trajectories and obtain the consistent flow F θ ( s, s ′ ) by minimizing the FM objective [6]. Consequently, a policy defined by this approximate flow π θ ( s ′ | s ) = P F θ ( s ′ | s ) = F θ ( s → s ′ ) F θ ( s ) can also sample objects x with a probability π θ (x) proportionally to reward R ( x ) . Practically, the training trajectories are sampled from a mixture between the current policy P F θ and a uniform distribution over allowed actions [6]. We adopt the FM objective in this work because the alternative trajectory balance [43] was also examined but gave a worse performance in early experiments. Note that more advanced objectives such as subtrajectory balance [42] can be employed in future work.

4.2 Hypernetwork-based GFlowNets

Our proposed HN-GFN aims at sampling a diverse batch of candidates from the approximate Pareto front with a unified model. A common approach to MOO is to decompose it into a set of scalar optimization problems with scalarization functions and apply standard single-objective optimization methods to gradually approximate the Pareto front [37, 64]. Here we consider the weighted sum (WS): s λ ( x ) = ∑ i λ i f i ( x ) and Tchebycheff [45]: s λ ( x ) = max i λ i f i ( x ) , where λ = ( λ i , · · · , λ M ) is a preference vector that defines the trade-off between the competing properties.

To support parallel evaluations, one can simultaneously obtain candidates according to different scalarizations. Practically, this approach hardly scales efficiently with the number of objectives for discrete problems. Taking GFlowNet as an example, we need to train multiple GFlowNets separately for each choice of the reward function R λ ( x ) = s λ ( x ) to cover the objective space:

Our key motivation is to design a unified GFlowNet to sample candidates according to different reward functions, even ones not seen during training. Instead of defining the reward function with a

fixed preference vector λ , we propose to train a preference-conditioned GFlowNet on the distribution of reward functions R λ , where λ is sampled from a simplex S M = { λ ∈ R m | ∑ i λ i = 1 , λ i ≥ 0 } :

Remarks. Assuming an infinite model capacity, it is easy to prove that the proposed optimization scheme (Equation (4)) is as powerful as the original one (Equation (3)), since the solutions to both loss functions coincide [17]. Nevertheless, the assumption of infinite capacity is extremely strict and hardly holds, so how to design the conditioning mechanism in practice becomes crucial.

4.2.1 Hypernetwork-based conditioning mechanism

We propose to condition the GFlowNets on the preference vectors via hypernetworks [26]. Hypernetworks are networks that generate the weights of a target network based on inputs. In vanilla GFlowNets, the flow predictor F θ is parameterized with the message passing neural network (MPNN) [23] over the graph of molecular fragments, with two prediction heads approximating F ( s, s ′ ) and F ( s ) based on the node and graph representations, respectively. These two heads are parameterized with multi-layer perceptrons (MLPs). For brevity, we write θ = ( θ mpnn , θ pred ) .

One can view the training of HN-GFN as learning an agent to carry out multiple policies that correspond to different goals (reward functions R λ ) defined in the same environment (state space S and action space A ). We thus propose only to condition the weights of prediction heads θ pred with hypernetworks, while sharing the weights of MPNN θ mpnn, leading to more generalizable state representations. More precisely, the hypernetwork h ( · ; ϕ ) takes the preference vector λ as inputs and outputs the weights θ pred = h ( λ ; ϕ ) of prediction heads. The parameters ϕ of the hypernetwork are optimized like normal parameters. Following Navon et al. [47], we parametrize h using an MLP with multiple heads, each generating weights for different layers of the target network.

4.2.2 As the acquisition function optimizer

Training. At each iteration, we first sample a preference vector λ from a Dirichlet distribution Dir( α ) . Then the HN-GFN is trained with the reward function R λ ( x ) = a ( µ ( s λ ( x )) , σ ( s λ ( x )); M ) , where µ and σ are the posterior mean and standard deviation estimated by M . In principle, we retrain HN-GFN after every round. We also tried only initializing the parameters of the hypernetwork and found it brings similar performance and is more efficient.

Sampling. At each round i , we use the trained HN-GFN to sample a diverse batch of b candidates. Let Λ i be the set of k target preference vectors λ i target . We sample b k molecules for each λ i target ∈ Λ i and evaluate them on the oracle in parallel. We simply construct Λ i by sampling from Dir( α ) , but it is worth noting that this prior distribution can also be defined adaptively based on the trade-off of interest. As the number of objectives increases, we choose a larger k to cover the objective space.

4.3 Hindsight-like off-policy strategy

Resorting to the conditioning mechanism, HN-GFN can learn a family of policies to achieve various goals, i.e., one can treat sampling high-reward molecules for a particular preference vector as a separate goal. As verified empirically in Jain et al. [28], since the FM objective is off-policy and offline , we can use offline trajectories to train the target policy for better exploration, so long as the assumption of full support holds. Our key insight is that each policy can learn from the valuable experience (high-reward molecules) of other similar policies.

Inspired by hindsight experience replay [2] in RL, we propose to share high-performing molecules among policies by re-examining them with different preference vectors. As there are infinite possible preference vectors, we focus on target preference vectors Λ i , which are based on to sample candidates, and build a replay buffer for each λ i target ∈ Λ i . After sampling some trajectories during training, we store in the replay buffers the complete objects x with the corresponding reward R λ i target ( x ) .

To be specific, at each iteration, we first sample a preference vector from a mixture between Dir ( α ) and a uniform distribution over Λ i : (1 -γ ) Dir ( α ) + γ Uniform. If Λ i is chosen, we construct half of the training batch with offline trajectories from the corresponding replay buffer of molecules

encountered with the highest rewards. Otherwise, we incorporate offline trajectories from the current observed dataset D i instead to ensure that HN-GFN samples correctly in the vicinity of the observed Pareto set. Our strategy allows for flexible trade-offs between generalization and specialization. As we vary γ from 0 to 1, the training distribution of preference vectors moves from Dir( α ) to Λ i . Exclusively training the HN-GFN with the finite target preference vectors Λ i can lead to poor generalization. In practice, although we only sample candidates based on Λ i , we argue that it is vital to keep the generalization to leverage the trained HN-GFN to explore various preference vectors adaptively. The detailed algorithm can be found in Appendix A.

4.4 Surrogate model and acquisition function

While GPs are well-established in continuous spaces, they scale poorly with the number of observations and do not perform well in discrete spaces [57]. There has been significant work in efficiently training non-Bayesian neural networks to estimate epistemic uncertainty [21, 39]. We benchmark widely used approaches (in Appendix C.3), and use a flexible and effective one: evidential deep learning [1]. As for the acquisition function, we use Upper Confidence Bound [55] to incorporate the uncertainty. To be precise, the objectives are modeled with a single multi-task MPNN, and the acquisition function is applied to the scalarization.

5 Experiments

We first verify that HN-GFN has adequate capacity to generalize over preference vectors in a singleround synthetic scenario. Next, we evaluate the effectiveness of the proposed MOBO algorithm based on HN-GFN in multi-round practical scenarios, which are more in line with real-world molecular optimization. Besides, we conduct several ablation studies to empirically justify the design choices.

5.1 Single-round synthetic scenario

Here, our goal is to demonstrate that we can leverage the HN-GFN to sample molecules with preference-conditioned property distributions. The HN-GFN is used as a stand-alone optimizer outside of MOBO to directly optimize the scalarizations of oracle scores. As the oracle cannot be called as many times as necessary practically, we refer to this scenario as a synthetic scenario. To better visualize the trend of the property distribution of the sampled molecules as a function of the preference weight, we only consider two objectives: inhibition scores against glycogen synthase kinase-3 beta (GNK3 β ) and c-Jun N-terminal kinase-3 (JNK 3 ) [40, 33].

Compared methods. We compare HN-GFN against the following methods. Preference-specific GFlowNets ( PS-GFN ) is a set of vanilla unconditional GFlowNets, each trained separately for a specific preference vector. Note that PS-GFN is treated as " gold standard " rather than a baseline, as it is trained and evaluated using the same preference vector. Concat-GFN and FiLM-GFN are two alternative conditional variations of GFlowNet based on the concatenation and FiLM [51], respectively. MOEA/D [64] and NSGA-III [14] are two multi-objective evolutionary algorithms that also incorporate preference information. We perform evolutionary algorithms over the 32-dim latent space learned by HierVAE [32], which gives better optimization performance than JT-VAE [31].

Metrics. All the above methods are evaluated over the same set of 5 evenly spaced preference vectors. For each GFlowNet-based method, we sample 1000 molecules per preference vector as solutions. We compare the aforementioned methods on the following metrics: Hypervolume indicator (HV) measures the volume of the space dominated by the Pareto front of the solutions and bounded from below by the preference point (0 , 0) . Diversity (Div) is the average pairwise Tanimoto distance over Morgan fingerprints. Correlation (Cor) is the Spearman's rank correlation coefficient between the probability of sampling molecules from an external test set under the GFlowNet and their respective rewards in the logarithmic domain [48]. See more details in Appendix B.1.2. In a nutshell, HV and Div measure the quality of the solutions, while Cor measures how well the trained model is aligned with the given preference vector. Each experiment is repeated with 3 random seeds.

Experimental results. As shown in Table 1, HN-GFN outperforms the baselines and achieves competitive performance to PS-GFN ( gold standard ) on all the metrics. Compared to GFlowNet-based

0.55

Preference-specific GFlowNet

HN-GFN

0.66

0.50

Average Top-K Reward

scores

0.64

0.45

Top-100 JNK3 s

0.40

0.62

0.35

0.60

on-policy

off-policy(y = 0)

0.30

off-policy(y = 0.2)

0.58

off-policy(y = 0.4)

off-policy(y = 0.6)

0.25

一

off-policy(y = 0.8)

0.1

0.3

0.5

0.7

0.9

0

2000

4000

6000

8000

PreferenceweightforJNK3

Training Steps

methods, evolutionary algorithms (MOEA/D and NSGA-III) fail to find high-scoring molecules, especially the MOEA/D. HN-GFN outperforms Concat-GFN and FiLM-GFN, and is the only conditional variant that can match the performance of PS-GFN, implying the superiority of the welldesigned hypernetwork-based conditioning mechanism. In Figure 2 (left), we visualize the empirical property (JNK 3 ) distribution of the molecules sampled by HN-GFN and PS-GFN conditioned on the set of evaluation preference vectors. We observe that the distributions are similar and the trends are consistent: the larger the preference weight, the higher the average score. The comparable performance and consistent sampling distributions illustrate that HN-GFN has adequate capacity to generalize over preference vectors. Since the runtime and storage space of PS-GFN scale linearly with the number of preference vectors, our unified HN-GFN provides a significantly efficient way to explore various trade-offs between objectives.

5.2 Multi-objective Bayesian optimization

Next, we evaluate the effectiveness of HN-GFN as an acquisition function optimizer within MOBO in practical scenarios. We consider the following objective combinations of varying sizes:

GNK3 β +JNK 3 : Jointly inhibiting Alzheimer-related targets GNK3 β and JNK 3 .

GNK3 β +JNK 3 +QED+SA: Jointly inhibiting GNK3 β and JNK 3 while being drug-like and easy-to-synthesize.

We rescale the SA score (initially between 10 and 1) so that all the above properties have a range of [0,1] and higher is better. For both combinations, we consider starting with |D 0 | = 200 random molecules and further querying the oracle N = 8 rounds with batch size b = 100 . Each experiment is repeated with 3 random seeds.

Baselines. We compare HN-GFN with three representative LSO methods ( q ParEGO [37], q EHVI [12], and LaMOO [65]), as well as a variety of state-of-the-art combinatorial optimization

β

+ JNK

3

0.7

HN-GFN w/ hindsight

HN-GFN

0.45

HN-GFN w/ hindsight

HN-GFN

MARS

Graph GA

0.40

=MARS

Graph GA

P-MOCO

PMOCO

0.6

qEHVI

qParEGO

0.35

qEHVI

qParEGO

LaMOO

LaMOO

Hypervolume

Hypervolume

0.5

0.30

0.25

0.4

0.20

0.3

0.15

0.2

0.10

200

3004005006007008009001000

00010060080040090090000000Z

TotalOracleEvaluations

TotalOracleEvaluations

methods: Graph GA [30] is a genetic algorithm, MARS [61] is a MCMC sampling approach, and P-MOCO [41] is a multi-objective RL method. We provide more details in Appendix B.2.

Experimental results. Figure 3 illustrates that HN-GFN achieves significantly superior performance (HV) to baselines, especially when trained with the proposed hindsight-like off-policy strategy. HN-GFN outperforms the best baselines P-MOCO and Graph GA of the two objective combinations by a large margin (0.67 vs. 0.50 and 0.42 vs. 0.34), respectively. Besides, HN-GFN is more sampleefficient, matching the performance of baselines with only half the number of oracle evaluations. All combinatorial optimization methods result in more performance gains compared to the LSO methods, implying that it is promising to optimize directly over the discrete space. In Table 2, we report the Div (computed among the batch of 100 candidates and averaged over rounds). For a fair comparison, we omit the LSO methods as they only support 160 rounds with batch size 5 due to memory constraints. Compared with Graph GA and P-MOCO, which sometimes propose quite similar candidates, the superior optimization performance of HN-GFN can be attributed to the ability to sample a diverse batch of candidates. Another interesting observation, in the more challenging setting (GNK3 β +JNK 3 +QED+SA), is that MARS generates diverse candidates via MCMC sampling but fails to find a high-quality Pareto front, indicating that HN-GFN can find high-reward modes better than MARS. The computational costs are discussed in Appendix B.4.

5.3 Ablations

Effect of the hindsight-like off-policy strategy. In the first round of MOBO, for each λ target ∈ Λ we sample 100 molecules every 500 training steps and compute the average Top-20 reward over Λ . In Figure 2 (right), we find that the hindsight-like off-policy strategy significantly boosts average rewards, demonstrating that sharing high-performing molecules among policies effectively speeds up the training of HN-GFN. On the other hand, further increasing γ leads to slight improvement. Hence, we choose γ = 0 . 2 for the desired trade-off between generalization and specialization.

Effect of Dir( α ) . Here we consider the more challenging GNK3 β +JNK 3 +QED+SA combination, where the difficulty of optimization varies widely for various properties. Table 3 shows that the distribution skewed toward harder properties results in better optimization performance. In our early experiments, we found that if the distribution of Λ i is fixed, HN-GFN is quite robust to changes in α .

Effect of scalarization functions s λ . In Table 3, we observe that WS leads to a better Pareto front than Tchebycheff. Although the simple WS is generally inappropriate when a non-convex Pareto front is encountered [10], we find that it is effective when optimized with HN-GFN, which can sample diverse high-reward candidates and may reach the non-convex part of the Pareto front. In addition, we conjecture that the non-smooth reward landscapes induced by Tchebycheff are harder to optimize.

6 Conclusion

We have introduced a MOBO algorithm for sample-efficient multi-objective molecular optimization. This algorithm leverages a hypernetwork-based GFlowNet (HN-GFN) to sample a diverse batch of candidates from the approximate Pareto front. In addition, we present a hindsight-like off-policy strategy to improve optimization performance. Our algorithm outperforms existing approaches in synthetic and practical scenarios. Future work includes extending this algorithm to other discrete optimization problems such as biological sequence design and neural architecture search. One limitation of the proposed HN-GFN is the higher computational costs than other training-free optimization methods. However, the costs resulting from model training are negligible compared to the costs of evaluating the candidates in real-world applications. We argue that the higher quality of the candidates is much more essential than lower costs.

Acknowledgments and Disclosure of Funding

This research was partially supported by National Key R&D Program of China under grant No.2018AAA0102102, National Natural Science Foundation of China under grants No.62176231, No.62106218, No.82202984, No.92259202, No.62132017 and U22B2034, Zhejiang Key R&D Program of China under grant No.2023C03053. Tingjun Hou's research was supported in part by National Natural Science Foundation of China under grant No.22220102001.

References

Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. Advances in Neural Information Processing Systems , 33:14927-14937, 2020.

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems , 30, 2017.

Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In International Conference on Learning Representations , 2020. URL https://openreview. net/forum?id=HklxbgBKvr .

Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wilson, and Eytan Bakshy. Botorch: a framework for efficient monte-carlo Bayesian optimization. Advances in neural information processing systems , 33:21524-21538, 2020.

Syrine Belakaria, Aryan Deshwal, and Janardhan Rao Doppa. Max-value entropy search for multi-objective Bayesian optimization. Advances in Neural Information Processing Systems , 32, 2019.

Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems , 34:27381-27394, 2021.

Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. Journal of Machine Learning Research , 24(210):1-55, 2023.

G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry , 4(2):90-98, 2012.

Julian Blank and Kalyanmoy Deb. Pymoo: Multi-objective optimization in python. IEEE Access , 8:89497-89509, 2020.

Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Eric Bradford, Artur M Schweidtmann, and Alexei Lapkin. Efficient multiobjective optimization employing Gaussian processes, spectral sampling and a genetic algorithm. Journal of global optimization , 71(2):407-438, 2018.

Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hypervolume improvement for parallel multi-objective Bayesian optimization. Advances in Neural Information Processing Systems , 33:9851-9864, 2020.

Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Parallel Bayesian optimization of multiple noisy objectives with expected hypervolume improvement. Advances in Neural Information Processing Systems , 34:2187-2200, 2021.

Kalyanmoy Deb and Himanshu Jain. An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part i: solving problems with box constraints. IEEE transactions on evolutionary computation , 18(4):577-601, 2013.

Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian Structure Learning with Generative Flow Networks. arXiv preprint , 2022.

Aryan Deshwal and Jana Doppa. Combining latent space and structured kernels for Bayesian optimization over combinatorial spaces. Advances in Neural Information Processing Systems , 34:8185-8200, 2021.

Alexey Dosovitskiy and Josip Djolonga. You only train once: Loss-conditional training of deep networks. In International conference on learning representations , 2019.

Matthias Ehrgott. Multicriteria optimization , volume 491. Springer Science & Business Media, 2005.

Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. Journal of cheminformatics , 1(1):1-11, 2009.

Tianfan Fu, Wenhao Gao, Cao Xiao, Jacob Yasonik, Connor W Coley, and Jimeng Sun. Differentiable scaffolding tree for molecule optimization. In International Conference on Learning Representations , 2022.

Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning , pages 1050-1059. PMLR, 2016.

Wenhao Gao, Tianfan Fu, Jimeng Sun, and Connor Coley. Sample efficiency matters: a benchmark for practical molecular optimization. Advances in Neural Information Processing Systems , 35:21342-21357, 2022.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272. PMLR, 2017.

Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS central science , 4(2):268-276, 2018.

Javier González, Zhenwen Dai, Philipp Hennig, and Neil Lawrence. Batch bayesian optimization via local penalization. In Artificial intelligence and statistics , pages 648-657. PMLR, 2016.

David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106 , 2016.

Daniel Hernández-Lobato, Jose Hernandez-Lobato, Amar Shah, and Ryan Adams. Predictive entropy search for multi-objective Bayesian optimization. In International conference on machine learning , pages 1492-1501. PMLR, 2016.

Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garcia, Jarrid Rector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning , pages 9786-9801. PMLR, 2022.

Moksh Jain, Sharath Chandra Raparthy, Alex Hernandez-Garcia, Jarrid Rector-Brooks, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. arXiv preprint arXiv:2210.12765 , 2022.

Jan H Jensen. A graph-based genetic algorithm and generative model/monte carlo tree search for the exploration of chemical space. Chemical science , 10(12):3567-3572, 2019.

Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning , pages 2323-2332. PMLR, 2018.

Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In International conference on machine learning , pages 48394848. PMLR, 2020.

Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multi-objective molecule generation using interpretable substructures. In International conference on machine learning , pages 4849-4859. PMLR, 2020.

Donald R Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization , 13(4):455-492, 1998.

Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Neural architecture search with Bayesian optimisation and optimal transport. Advances in neural information processing systems , 31, 2018.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Joshua Knowles. Parego: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Transactions on Evolutionary Computation , 10(1):50-66, 2006.

Mina Konakovic Lukovic, Yunsheng Tian, and Wojciech Matusik. Diversity-guided multiobjective Bayesian optimization with batch evaluations. Advances in Neural Information Processing Systems , 33:17708-17720, 2020.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems , 30, 2017.

Yibo Li, Liangren Zhang, and Zhenming Liu. Multi-objective de novo drug design with conditional graph generative model. Journal of cheminformatics , 10(1):1-24, 2018.

Xi Lin, Zhiyuan Yang, and Qingfu Zhang. Pareto set learning for neural multi-objective combinatorial optimization. In International Conference on Learning Representations , 2021.

Kanika Madan, Jarrid Rector-Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, Andrei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability. In International Conference on Machine Learning , pages 23467-23483. PMLR, 2023.

Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. arXiv preprint arXiv:2201.13259 , 2022.

Natalie Maus, Haydn T Jones, Juston S Moore, Matt J Kusner, John Bradshaw, and Jacob R Gardner. Local latent space Bayesian optimization over structured inputs. arXiv preprint arXiv:2201.11872 , 2022.

Kaisa Miettinen. Nonlinear multiobjective optimization , volume 12. Springer Science & Business Media, 2012.

Henry Moss, David Leslie, Daniel Beck, Javier Gonzalez, and Paul Rayson. Boss: Bayesian optimization over string spaces. Advances in neural information processing systems , 33: 15476-15486, 2020.

Aviv Navon, Aviv Shamsian, Ethan Fetaya, and Gal Chechik. Learning the pareto front with hypernetworks. In ICLR , 2021.

Andrei Cristian Nica, Moksh Jain, Emmanuel Bengio, Cheng-Hao Liu, Maksym Korablyov, Michael M Bronstein, and Yoshua Bengio. Evaluating generalization in gflownets for molecule design. In ICLR2022 Machine Learning for Drug Discovery , 2022.

Biswajit Paria, Kirthevasan Kandasamy, and Barnabás Póczos. A flexible framework for multi-objective Bayesian optimization using random scalarizations. In Uncertainty in Artificial Intelligence , pages 766-776. PMLR, 2020.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI Conference on Artificial Intelligence , 2018.

Pavel G Polishchuk, Timur I Madzhidov, and Alexandre Varnek. Estimation of the size of drug-like chemical space based on gdb-17 data. Journal of computer-aided molecular design , 27(8):675-679, 2013.

Binxin Ru, Xingchen Wan, Xiaowen Dong, and Michael Osborne. Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kernels. arXiv preprint arXiv:2006.07556 , 2020.

Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE , 104 (1):148-175, 2015.

Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: no regret and experimental design. In Proceedings of the 27th International Conference on International Conference on Machine Learning , pages 1015-1022, 2010.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Kevin Swersky, Yulia Rubanova, David Dohan, and Kevin Murphy. Amortized Bayesian optimization over discrete spaces. In Conference on Uncertainty in Artificial Intelligence , pages 769-778. PMLR, 2020.

Austin Tripp, Erik Daxberger, and José Miguel Hernández-Lobato. Sample-efficient optimization in the latent space of deep generative models via weighted retraining. Advances in Neural Information Processing Systems , 33:11259-11272, 2020.

Austin Tripp, Gregor NC Simm, and José Miguel Hernández-Lobato. A fresh look at de novo molecular design benchmarks. In NeurIPS 2021 AI for Science Workshop , 2021.

Robin Winter, Floriane Montanari, Andreas Steffen, Hans Briem, Frank Noé, and Djork-Arné Clevert. Efficient multi-objective molecular optimization in a continuous latent space. Chemical science , 10(34):8016-8024, 2019.

Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. Mars: Markov molecular sampling for multi-objective drug discovery. In International Conference on Learning Representations , 2021.

Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. Advances in neural information processing systems , 31, 2018.

Dinghuai Zhang, Nikolay Malkin, Zhen Liu, Alexandra Volokhova, Aaron Courville, and Yoshua Bengio. Generative flow networks for discrete probabilistic modeling. arXiv preprint arXiv:2202.01361 , 2022.

Qingfu Zhang and Hui Li. Moea/d: A multiobjective evolutionary algorithm based on decomposition. IEEE Transactions on evolutionary computation , 11(6):712-731, 2007.

Yiyang Zhao, Linnan Wang, Kevin Yang, Tianjun Zhang, Tian Guo, and Yuandong Tian. Multiobjective optimization by learning space partition. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=FlwzVjfMryn .

Yiheng Zhu, Zhenqiu Ouyang, Ben Liao, Jialu Wu, Yixuan Wu, Chang-Yu Hsieh, Tingjun Hou, and Jian Wu. Molhf: A hierarchical normalizing flow for molecular graph generation. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence , pages 5002-5010, 8 2023.

Eckart Zitzler and Lothar Thiele. Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE transactions on Evolutionary Computation , 3(4): 257-271, 1999.

A Algorithms

Algorithm 1 describes the overall framework of the proposed MOBO algorithm, where HN-GFN is leveraged as the acquisition function optimizer. Algorithm 2 describes the training procedure for HN-GFN within MOBO.

B Implementation details

B.1 Experimental settings

B.1.1 Molecule domain

Following Bengio et al. [6], the molecules are generated by sequentially attaching a fragment, which is chosen from a predefined vocabulary of building blocks, to an atom of the partially constructed molecules. The maximum trajectory length is 8, with the number of actions varying between 100 and 2000 depending on the state, making |X| up to 10 16 . We adopt the property prediction models released by Xie et al. [61] to evaluate the inhibition ability of generated molecules against GSK3 β and JNK 3 .

B.1.2 Metrics

Diversity. Diversity (Div) is the average pairwise Tanimoto distance over Morgan fingerprints. In the synthetic scenario, for each preference vector, we sample 1000 molecules, calculate the Div among the Top-100 molecules, and report the averages over preferences. In MOBO, the DiV is computed among the batch of 100 candidates per round, as Graph GA and MARS are not preferenceconditioned. And we believe this metric possibly is more aligned with how these methods might be used in a biology or chemistry experiment.

Correlation. Correlation (Cor) is the Spearman's rank correlation coefficient between the probability of sampling molecules from an external test set under the GFlowNet and their respective rewards in the logarithmic domain: Cor = Spearman's ρ log ( π ( x )) , log ( R ( x )) . The external test set is obtained in two steps: First, we generate a random dataset containing 300K molecules uniformly based on the number of building blocks; Next, we sample the test sets with uniform property distribution corresponding to GSK3 β and JNK 3 , respectively, from the 300K molecules.

B.2 Baselines

All baselines are implemented using the publicly released source codes with adaptations for our MOBO scenarios. The evolutionary algorithms (MOEA/D and NSGA-III) are implemented in PyMOO[9], and the LSO methods ( q ParEGO, q EHVI, and LaMOO) are implemented in BoTorch [4]. In MOBO scenarios, LaMOO and Graph GA utilize EHVI as the acquisition function. For all GPbased methods, each objective is modeled by an independent GP.

B.3 HN-GFN

We implement the proposed HN-GFN in PyTorch [50]. The values of key hyper-parameters are illustrated in Table 4.

Surrogate model: We use the 12-layer MPNN as the base architecture of the surrogate model in our experiments. In MOBO, a single multi-task MPNN is trained with a batch size of 64 using the Adam optimizer with a dropout rate of 0.1 and a weight decay rate of 1e-6. We apply early stopping to improve generalization.

HN-GFN: HN-GFN contains a vanilla GFlowNet and a preference-conditioned hypernetwork. The architecture of GFlowNet is a 10-layer MPNN, and the hypernetwork is a 3-layer MLP with multiple heads, each generating weights for different layers of the target network. The HN-GFN is trained with Adam optimizer to optimize the Flow Matching objective.

B.4 Empirical running time

The efficiency is compared on the same computing facilities using 1 Tesla V100 GPU. In the context of MOBO, the running time of three LSO methods (i.e., q ParEGO, q EHVI, and LaMOO) is around 3 hours, while Graph GA optimizes much faster and costs only 13 minutes. In contrast, the time complexity of deep-learning-based discrete optimization methods is much larger. MARS costs 32 hours, while our proposed HN-GFN costs 10 hours. With the hindsight-like training strategy, the running time of HN-GFN will increase roughly by 33%.

However, if we look at the problem in a bigger picture, the time costs for model training are likely negligible compared to those of evaluating molecular candidates in real-world applications. Therefore, we argue that the high quality of the candidates (the performance of the MOBO algorithm) is more essential than having a lower training cost.

C Additional results

C.1 Sampled molecules in MOBO experiments

We give some examples of sampled molecules from the Pareto front by HN-GFN in the GSK3 β + JNK3 + QED + SA optimization setting (Figure 4). The numbers below each molecule refer to GSK3 β , JNK3, QED, and SA scores respectively.

C.2 Synthetic scenario

As illustrated in Figure 5, the distribution of Top-100 GSK3 β scores shows a consistent trend in preference-specific GFlowNet and our proposed HN-GFN, although the trend is not as significant as the JNK 3 property.

Preference-specific GFlowNet

0.75

HN-GFN

scores

0.70

Top-100 GSK3β s

0.65

0.60

0.55

0.50

0.9

0.7

0.5

0.3

0.1

Preference weight for GSK3β

C.3 Effect of Surrogate models

We conduct ablation experiments to study the effectiveness of different surrogate models. We consider the following three surrogate models: evidential regression [1], Deep Ensembles [39], and GP based on the Tanimoto kernel [59]. As shown in Table 5, we can observe that evidential regression leads to better optimization performance than Deep Ensembles. While the HV of evidential regression and GP is comparable, evidential regression can propose more diverse candidates. Furthermore, we argue that GP is less flexible over discrete spaces than evidential regression and Deep Ensembles, as different kernels need to be designed according to the data structures.

D Comparison with parallel work

While the concept of Pareto GFlowNet was theoretically discussed in Bengio et al. [7], we are among the first to study and instantiate this concept for MOO, and we address practical challenges that are not discussed thoroughly in the original theoretical exposition to potentially make sample-efficient molecular optimization a reality. We extensively study the impact of the conditioning mechanism (Section 5.1) and surrogate models (Appendix C.3). Moreover, we delicately propose a hindsight-like off-policy strategy (Section 4.3) which is rarely studied for MOO (in both the RL and GFlowNet literature).

We note that there is a parallel work introduced in Jain et al. [29], which shares the idea of using GFlowNets for MOO. In Jain et al. [29], they propose two variants MOGFN-PC and MOGFN-AL for MOO in the single-round scenario and Bayesian optimization (BO) scenario, respectively. Indeed, when HN-GFN is used as a stand-alone optimizer outside of MOBO (Section 5.1), it is similar to MOGFN-PC except for using different conditioning mechanisms. As for MOBO, MOGFN-AL is a vanilla GFlowNet whose reward function is defined as a multi-objective acquisition function (NEHVI [13]). In addition to extending GFlowNet, we delicately propose a hindsight-like off-policy strategy (Section 4.3) which is rarely studied for MOO (in both the RL and GFlowNet literature).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMolecule generation involves the optimization of multiple potentially competing objectives simultaneously. As evaluating these objectives can be a time-consuming and costly task, sample efficiency is paramount. This work proposes a multi-objective Bayesian optimization approach leveraging GFlowNets to tackle this problem. The authors propose a preference-based decomposition and a hyper-network-based parameterization for incorporating preferences and conditioning on those. They also employ a hindsight-experience replay strategy to utilize offline data during learning. The hyper-network-based GFlowNet is then used as an acquisition function optimizer to find a diverse set of molecules on the empirical Pareto front.\n\nSTRENGTHS:\n- The paper covers an important and impactful topic of multi-objective optimization in molecule design.\n-  The experimental results, in principle, are strong.\n- It is an original combination of existing techniques.\n- Many benchmarks are compared.\n- Ablation study\n\nWEAKNESSES:\n- It is not directly evident to me how this work is different from Pareto GFlowNet that was proposed in [GFlowNet Foundations](https://arxiv.org/pdf/2111.09266.pdf)\n\n- Some parts remain somewhat unclear to me. How is the hyper-network trained? Is it stable, does it do what it is expected to do? \n\n- I am somewhat missing the link between the surrogate model and HN-GFN from what is described in the main paper. Why do we actually need the surrogate model? Can't the GFlownet propose diverse candidates by itself?\n\n- Perhaps the work would benefit from a clearer description and/or toy tasks to clear this up.\n\nQUESTIONS:\n- It would be nice if the authors could dedicate some more information about the test problem. What is the search space of the molecules, what atom types, how are impossible molecular structures avoided, etc? Are the building blocks created in such a way that they always can be connected to each other without violations?\n\n- Why a batch size of 100 and 8 rounds? In what setting would a practitioner be able to perform 100 wet lab experiments simultaneously? Perhaps with a simulator, this would be possible. But wouldn't the surrogate models (at least from the BO qEHVI and qParEGO) benefit from lower batch sizes and more iterations? This could change the benchmarks significantly.\n\nLIMITATIONS:\nThe authors have addressed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA GFlowNet for molecular optimization conditioned on the preference weights of multiple objectives is proposed. To be precise, the model is trained to sample molecules from a target space with probability proportional to some combination of reward functions (weighted sum or Chebyshev scalarization) and trained with varying weights of the rewards, with the GFlowNet conditioning achieved by a hypernetwork parametrization. The system is tested in both single-step and Bayesian optimization / active learning settings.\n\nSTRENGTHS:\n- The writing is generally clear, problem is introduced well and placed in context.\n- Good results on the tasks studied relative to the chosen baselines\n- Two interesting GFlowNet contributions not specific to molecules:\n  - Study of conditioning algorithms (FiLM and Concat variants), with the hypernetwork-based approach found to perform best.\n  - The use of a conditional replay buffer, which is, as far as I know, new in GFlowNet literature (a replay buffer was used with GFlowNets in [arXiv:2202.13903, UAI 2022] but in a vanilla RL manner).\n  - Both are illustrated well with ablation studies.\n- The code is a helpful addition that aids in understanding the details of what was done.\n\nWEAKNESSES:\nMeta-weakness: Due to my closely following the literature, I was aware of an earlier submission of this paper to ICLR 2023. The present submission is almost identical to ICLR version (post-rebuttal) and therefore cannot incorporate the feedback of the reviewers. That said, to avoid biasing myself here, I did not read the ICLR reviews and this assessment is entirely my own.\n\nComparison with prior work:\n- There is the paper \"Multi-objective GFlowNets\" [MO; arXiv:2210.12765, ICML 2023], which also studies GFlowNets for multiobjective Bayesian optimization by conditioning on scalarization weights. Can you comment on the differences in problem setting or method with that paper?\n- The attached code includes the implementations of two baselines, in `code/generator/gfn.py`, but they are not included in the paper:\n  - MOReinforce, which is a natural candidate for an RL-based baseline that can readily share the parametrization of the agent with the GFlowNet.\n  - GFlowNet with trajectory balance [TB; arXiv:2201.13259, NeurIPS 2022]. Trajectory balance is the objective used in nearly all work on GFlowNets for molecular optimization after [5], but for some reason it is not evaluated or mentioned in the text. In particular, it is evaluated on molecule synthesis in both [TB] and [MO].\n\nSmall issue in presentation/math: There are some inconsistencies in definitions related to GFlowNets. Equation (1) writes $R(s)$ for the reward received when terminating at $s$, as in [5]. However, the text immediately following it talks about terminal and nonterminal states (with nonterminals having reward 0), and the text above says there is a special termination action that leads to terminal states. This is more consistent with the convention from [TB], where terminal states have no children and are the only states with nonzero reward. This is not a major bug as the two conventions are equivalent, but it would be good to be careful.\n\nQUESTIONS:\nPlease see above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the use of GFlotNets to address the problem of sample-efficient multi-objective molecular optimization, an important problem in various scientific discovery application - such as materials design and drug discovery.\nThe key idea proposed in this work is to leverage hypernetwork-based GFlowNets - referred to as HN-GFN - to optimize the acquisition function for multi-objective Bayesian optimization (MOBO).\nThe goal is to enable efficient sampling of a diverse high-quality batch of molecular candidates from an approximate Pareto front.\n\nSTRENGTHS:\nThe proposed hyper network-based GFlowNets as a MOBO acquisition function optimizer provides a novel and intuitive way of using GFlowNets for sampling novel molecular candidates, where a \"unified\" GFlowNet is trained that considers the distribution of different reward functions that correspond to different preference vectors instead of relying on a single GFlowNet for a fixed preference vector.\nThis may allow the resulting model, HN-GFN, to naturally explore the various trade-offs between multiple objectives that may compete with one another by adapting to varying the input preference vector.\n\nThe proposed method builds on the recently proposed and widely popular GFlowNets and extends its flexibility for multi-objective molecular optimization by incorporating a hypernetwork-based approach.\nAdditionally, the adoption of a hindsight-like off-policy strategy is proposed to improve the learning efficiency, and ultimately, the multi-objective molecular optimization performance.\n\nOverall, the paper is well-organized and written in a clear manner, the proposed method is well-motivated and novel, and the performance evaluation results demonstrate the potential advantages of the proposed HN-GFN.\n\nWEAKNESSES:\nAlthough the batch size may significantly affect the overall computational cost as well as the optimization performance, there is no discussion on the impact of selecting a specific batch size nor any empirical evaluation based on different batch sizes.\n\nWhile the evaluation results provide some preliminary evidence of the potential advantages of HN-GFN, the evaluations in the current study are limited to (virtually) a single problem: i.e., inhibition of GSK3β + JNK3 (with potential additional considerations for synthesizability and drug-likeness).\nAdditional examples are needed to more convincingly demonstrate the general applicability (and merits) of the proposed HN-GFN.\nI suggest providing further evaluation results based on other benchmark problems often used for evaluating other generative models (e.g., JT-VAE, HierVAE, etc.)\n\nIn Figure 2, only trends for JNK3 are shown, but it would be helpful to show the optimization trends for GNK3β as well for completeness.\n\nQUESTIONS:\nPlease see the questions and suggestions in the weaknesses section.\n\nLIMITATIONS:\nThe conclusion section includes a very brief discussion of a limitation of the current method and suggests directions for future work.\nThe broader implication of the work is not explicitly discussed in the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses molecular design with GFLOWNET, an algorithm learning a sampling policy $\\pi$ proportional to the reward function, i.e. where $\\pi(x) \\propto R(x)$. In particular, this work tackles the essential yet under-adressed multiobjective optimization setting. They do so using a hypernetwork (conditioned on a preference vector) providing weights for the sampler.\n\nSTRENGTHS:\nIn my opinion the presentation is clear and the motivation behind this work are nicely presented. In addition, the paper is well written and easy to follow. As a side note, its presentation of the GFN is interesting and worth the reminder to ease the understanding of the paper. \nThe approach is both original and sound, boosting its potential use in practical application. \nFinally, the presented results are conclusive and seem to validate the intuition behind the method.\n\nWEAKNESSES:\nThe main weaknesses in the paper is in my opinion in the lack of diverse experiments or explanation about them. See questions.\n\nQUESTIONS:\n1- A part of the protocole is unclear to me: are the generated molecules evaluated on the objective using an actual oracle or using the learned surrogate function ?\n\n2 - How are the objectives computed ? Is it “physics” based or does it rely on learning a surrogate model ? Then, how are the generated molecules evaluated ?\n\n3 - What is the length of the generated molecules ? Is it a hyperparameter of the model ?\n\n4 - How sensitive are the results to the original dataset  $D_0$?  In my personal experience, Gflownets tend to be rather unstable or difficult to train due to the sparse feedback (and depends on the sequence size) and 200 initial molecules seem low to be able to learn a robust policy.\n \n5 - Does the proposed Hindsight-like off policy strategy amounts to the proposition of [1], i.e. to hybrid the learning between generated molecules vs already generated molecules ?\n\n6 - What are the variable predicted by the MPNN ? \n\n7 -  Can the authors discuss their implementation choice regarding the gflow network. For instance [1] use a MLP as sampler. \n\n\n[1]: Jain et al, Biological sequence designs with Gflownets.\n\nLIMITATIONS:\nA limitation of the paper is the lack of diversity in the experimental settings. For instance, the paper Gflownet for biological sequence design presents optimization results for several benchmark datasets such as GFP. Can the authors comment on the ability of their method to adapt to such an ‘off-line’ setting (which is an essential in biological sequence design) ?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}