Sample-efficient multi-objective molecular optimization with GFlowNets
Authors: Yiheng Zhu
Jialu Wu
Chaowen Hu
Jiahuan Yan
Chang-Yu Hsieh
Tingjun Hou
Jian Wu
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-objective molecular optimization
Bayesian optimization
generative flow networks

Abstract: Many crucial scientific problems involve designing novel molecules with desired properties, which can be formulated as an expensive black-box optimization problem over the discrete chemical space. Computational methods have achieved initial success but still struggle with simultaneously optimizing multiple competing properties in a sample-efficient manner. In this work, we propose a multi-objective Bayesian optimization (MOBO) algorithm leveraging the hypernetwork-based GFlowNets (HN-GFN) as an acquisition function optimizer, with the purpose of sampling a diverse batch of candidate molecular graphs from an approximate Pareto front. Using a single preference-conditioned hypernetwork, HN-GFN learns to explore various trade-offs between objectives. Inspired by reinforcement learning, we further propose a hindsight-like off-policy strategy to share high-performing molecules among different preferences in order to speed up learning for HN-GFN. Through synthetic experiments, we illustrate that HN-GFN has adequate capacity to generalize over preferences. Extensive experiments show that our framework outperforms the best baselines by a large margin in terms of hypervolume in various real-world MOBO settings.

Under review as a conference paper at ICLR 2023
SAMPLE-EFRICIENT MULTI-OBIECTIVE MOLECULAR OPTIMIZATION WITH GFLOWNETS
Anonymous authors Paper under double-blind review
ABSTRACT
Many crucial scientific problems involve designing novel molecules with desired properties, which can be formulated as an expensive black-box optimization prob- lem over the discrete chemical space. Computational methods have achieved ini- tial success but still struggle with simultaneously optimizing multiple compet- ing properties in sample-efficiet manner. In this work, we propose multi- objective Bayesian optimization (MOBO) algorithm leveraging the hypernetwork- acquisition function optimizer, with the pur-
based GelowNets (HN-GFN) pose of sampling a diverse batch of candidate molecular graphs from an approxi- mate Pareto front. Using a single iifference-conditionn hypernetwork, HN-GFN learns to explore various trade-offs between objectives. Inspired by reinforcement learning, further propose hindsight-like off-policy strategy share high- performing molecules among different preferences in order to speed up learning for HN-GFN. Through synthetic experiments, we illustrate that HN-GFN has ad- equate capacity to generalize over preferences. Extensive experiments show that our framework outperforms the best baselines by a large margin in terms of hy- pervolume in various real-world MOBO settings.
INTRODUCTION
Designing novel molecular structures with desired properties, also referred to molecular optimiza- tion. is crucial task with great application potential scientific fields ranging from drug discovery to material design. Molecular optimization can be naturally formulated as a black-box optimization problem over the discrete chemical space, which combinatorially large (Polishchuk et al. 2013).
Recent years have witnessed the trend of leveraging computational methods, such as deep generative models (Jin et al. 2018) and combinatorial optimization algorithms (You et al. 2018 8 Jensen 2019).
to facilitate the optimization. However, the applicability of most prior approaches in real-world scenarios is hindered by two practical constraints: (i) realistic oracles (e.g.. wet-lab experiments and high-fidelit simulations) require substantial costs to synthesize and evaluate molecules (Gao let al. 2022), and (ii) chemists commonly seek to optimize multiple properties of interest simulta- neously (Jin al., (0206)). For example, in addition to effectively inhibiting disease-associated target, an ideal drug is desired to be easily synthesizable and non-toxic.
Bayesian optimization (BO) (Jones al. 1998 Shahriari al. 2015) provides a sample-efficient framework for globally optimizing expensive black-box functions. The basic idea is to construct cheap-to-evaluute surrogate model, typically a Gaussian Process (GP) (Rasmussen) 2003), to ap- proximate the true function (also known as the oracle) on the observed dataset. The core objective of BO optimize an acquisition function (built upon the surrogate model) order to obtain formative candidates with high utility for the next round of evaluations. This loop is repeated until the evaluation budget is exhausted. Owing the fact that large batch of candidates can be evalu- ated in parallel in biochemical experiments, perform batch BO (with large-batch and low-round settings (Angermueller et al. 2020)) to significantly shorten the entire cycle of optimization.
As multi-objective optimization (MO0) problems are prevalent in scientific and engineering appli- cations, MOBO also received broad attention and achieved promising performance by effectively optimizing differentiable acquisition functions (Daulton et al. 2020). Nevertheless, it is less promi- nent in discrete problems, especially considering batch settings. The difficulty lies the fact that no gradients can be leveraged to navigate the discrete space for efficient and effective optimization of

---

Under review as a conference paper at ICLR 2023
Training: : (-Dir(a)
Gll 8 (Ompan Ppred)
Sampling i Arorger
$6
Sp-1
Hypemetwork
pridd
n(-:d)
xh
Evidential
Sunrogate Model
RaOG)
Rg
(2(s))
represents the initial represents the complete object.
Figure 1: MOBO loop for molecular optimization using an evidential surrogate model M for un- certainty estimation and HN-GFN for acquisition function optimization. In each round, the policy Th is trained with reward function Rx, where À is sampled from Dir(a) per iteration. A new batch of candidates is sampled from the approximate Pareto front according to Auaget A.
the acquisition function. Although most of the existing discrete molecular optimization methods can be adopted as the acquisition function optimizer alleviate this issue, they suffer from the following limitations. 1) Most approaches do not explicitly discuss the diversity of the proposed candidates, which is a key consideration in batch settings as the surrogate model cannot exactly reproduce the oracle's 's full behaviors. Therefore, we not only want to cover more high modes of the surrogate model but also to obtain candidates that bring additional information about the search space. 2) Most multi-objective methods (Xie et al. [22TT] Fu et al. 2022) simply rely on a scalarization func- tion, parameterized by a predefined preference vector reflecting the trade-off between objectives, and turn the MOO problem into single-objective one. Unfortunately, an ideal trade-off is unclear before optimization (even with domain knowledge), and many potential trade-offs of interest are worth exploring. In principle, it is possible independently train multiple optimization models, each conditioned on a distinct preference vector, to cover the objective space. Practically, this trivial strategy cannot efficiently scale with the number of objectives (Navon al. 2021).
The recently proposed GFlowNets (Bengio et al. 2021a) are a class of generative models over discrete objects (e.g., molecular graphs) that aim learn a stochastic policy for sequentially con- structing objects with probability proportional to reward function (e.g., the acquisition function). Hence, GFlowNets possess merit in generating diverse and high-reward objects, which makes them appealing in the batch BO context where exploration plays significant role (Jain et al., 2022].
In this work, we present a MOBO algorithm based on GFlowNets for sample-efficient multi- objective molecular optimization. We propose hypernetwork-base GFlowNet (HN-GFN) as the acquisition function optimizer within MOBO to sample a diverse batch of candidates from an ap- proximate Pareto front. Instead of defining a fixed reward function as usual in past work (Ben- gio et al. 2021a), we train a unified GFlowNet on the distribution of reward functions (random scalarizations parameterized by preference vectors) and control the policy using a single preference- conditioned hypernetwork. While sampling candidates, HN-GFN explores various trade-offs be- tween competing objectives flexibly by varying the input preference vector. Inspired by Hindsight Experience Replay (Andrychowicz et al. (017) in RI., we further introduce a hindsight- off- policy strategy share performnffn molecules among different preferences and speed up learn- ing for HN-GFN. As detailed in our reported experiments, we first evaluate HN-GFN through syn- thetic experiments
verify that HN-GFN is capable of generalizing over preference vectors, then apply the proposed framework to real-world scenarios. Remarkably. our framework outperforms the best baselines by 60% and 24% (relative improvement terms of hypervolume in the settings with two and four objectives), respectively. Our key contributions are summarized below:
We propose HN-GFN, a unified GFlowNet that can efficiently sample candidates from an approximate Pareto front using single hypernetwork.
2

---

Under review as a conference paper at ICLR 2023
We introduce hindsight-like off off--olliy strategy to speed up learning in HN-GFN.
Experiments verify that our MOBO algorithm based HN-GFN can find high-quality Pareto front more efficiently compared to state-of-the-ar baselines.
RELATED WORK
Molecular optimization. Recently molecular optimization has been approached with a wide va- riety of computational methods, which can be mainly grouped into three categories: 1) Latent space optimization (LSO) methods perform the optimization over the low-dimensional continuous latent space learned by generative models such variational autoencoders (VAEs) (Gómez-Bombarelli et al. 2018 Maus et al.] 2022). These methods require the latent representations to be discrim- inative, but the training of the generative model is decoupled from the optimization objectives, imposing challenges for optimization (Tripp et al.) 2020). Instead of navigating the latent space, combinatorial optimization methods search for the desired molecular structures directly in the ex- plicit discrete space with 2) evolutionary algorithms [Jensen] 2019) and 3) deep neural networks to guide the searching YYou et al., 2018). However, most prior methods only focus on optimizing single property, from non- non-biological properties such as drug- drug-likeliness (QED) (Bickerton et al., 2012] and synthetic accessibility (SA) (Ertl & Schuffenhauer 2209)) to biological properties that measure the binding energy protein target (Bengio et al.
(2221a). Despite the above advances, multi-objective molecular optimization has recently received wide attention (Jin et al., 2020b; Xic et al. 2021: Fu et al., 2022 For example, MARS (Xie et al. I02 employs Markov chain Monte Carlo (MCMC) sampling to find novel molecules satistying several properties. However, most ap- proaches require notoriously large number of oracle calls to evaluate molecules on-the-fly (Jin
ct al.
2020b Xie et al. 2021). In contrast, we tackle this problem in a sample-efficien manner.
GFlowNet. GFlowNets (Bengio et al. 202 aim to sample composite objects proportionally to reward function, instead of maximizing related to the MCMC methods due to the same objective, while amortizing the high cost of sampling (mixing between modes) over training generative model (Zhang et al.. 2022). GFlowNets have made impressive progress in various applications, such as active learning (Jain et al. 20022) discrete probabilistic modeling (Zhang et al. 2022), and Bayesian structure learning (Deleu et 2022). For a thorough discussion and mathematical treatment, we refer the readers to BBngio et al. 2021a b)
usual in RL (Sutton & Barto 2018). GFlowNets are
Bayesian Optimization for discrete spaces. While the application of BO in continuous domains has proliferated during the last decade. effort in applying it to discrete spaces is lacking. It is much more challenging to construct surrogate models and optimize acquisition functions in dis- crete spaces, compared to continuous spaces. One common approach is to define GPs with discrete kernels (Moss al. (2020) and solve the acquisition function optimization problem with evolution- ary algorithms (Kandasamy et al. 2018). Moreover, AmortizedBO (Swersky et al. 2020) proposes augment the evolutionary algorithms with RL.
Multi-objective Bayesian Optimization. BO has been widely used in MOO problems for eff- ciently optimizing multiple competing black-box functions. Most popular approaches are based on hypervolume improvement (Daulton et al. 2020), random scalarizations (Knowles) 2006 Paria ett al. 2020). and entropy search (Herráánezz-Lobae)
et al. 2016). While there have been several approaches that take parallel evaluations (Bradford et al. 2018; Konakovie Lukovic et al.) 2020) and diversity (Konakovic Lukovic et al. 2020) into account, they are limited to continuous domains.
BACKGROUND
3.1 PROBLEM FORMULATION
We address the problem of searching over a discrete chemical space X' to find molecular graphs x e
X' that maximize a vector-valued objective f(z) (f1(z), fz), fx)()) X -> R where fm is a black-box function (also known as the oracle) evaluating a certain property of molecules. Practically, realistic oracles are extremely expensive evaluate with either high-fidelity simulations
3

---

Under review as a conference paper at ICLR 2023
wet-lab experiments. We thus propose to perform optimization in as few oracle evaluations as possible, since the sample efficiency is paramount in such a scenario.
There is typically single optimal solution the MOO problem. different objectives may tradict each other. Consequently, the goal is to recover the Pareto front the set of Pareto optimal solutions which cannot be improved in any one objective without deteriorating another (Ehrgoti 2005 Miettinen 2012). In the context of maximization, a solution f(x) is said to Pareto dominates another solution Mx' iff fm (3:) 2 fm (x') Vm = 1, I,...,,, M and 3m' such that fmr(x) > (anr("),, and we denote (x) > f(z'). A solution f(x*) is Pareto optimal if not Pareto dominated by any
solution. The Pareto front can be written as p {f(x*)) {f(z) f(x) > f(x*)(()+))))))))))) 03.
The quality of finite approximate Pareto front d is commonly measured by hypervolume (HV) Zit-- zler Thiele 1999) the M-dimensional Lebesgue measure As of the space dominated by and bounded from below by a given reference point r E MR; HV(P, r) = Aur(U 2 sil). where [r, vi] denotes the hyper-rectangle bounded by r and y; = f(x)).
3.2 BATCH BAYESIAN OPTIMIZATION
Bayesian optimization (BO) (Shahriari et al. 2015) provides a model-based iterative framework for sample-efficient black-box optimization. Given observed dataset D, BO relies on Bayesian surrogate model M to estimate a posterior distribution over the true oracle evaluations. Equipped with the surrogate model, an acquisition function X' - R is induced assign the utility values to candidate objects for deciding which to next evaluate the oracle. Compared with the costly oracle, the theap-to-evaluate acquisition function can be efficiently optimized. We consider the scenario where the oracle is given an evaluation budget of N rounds with fixed batches of size b.
To be precise, we have access a random initial dataset Du {(2.. y/));= where y fx is true oracle evaluation. In each round i E NJ) the acquisition function is maximized to yield batch of candidates B; = {-}})= 1 to be evaluated parallel on the oracle 1/ = [(x)). The observed dataset D is then augmented for the next round: D. = Di-1 U {x1 11)
4 METHOD
In this section, we present the proposed MOBO algorithm based on snpernetwork-baseee GFlowNet (HN-GFN), shown in Figure [ Our key idea is to extend GFlowNets as the acquisition function optimizer for MOBO, with the objective to sample a diverse batch of candidates from the approx- imate Pareto front. To begin, we introduce GFlowNets in the context of molecule design, then de- scribe how GFlowNet can be biased by a preference-condiiionee hypernetwork to sample molecules according various trade-off's between objectives. Next, we propose a hindsight-like off-policy strategy to speed up learning in HN-GFN. Lastly, we introduce the evidential surrogate model.
4.1 PRELIMINARIES
GFlowNets (Bengio al. 2021a) seek learn a stochastic policy æ for sequentially constructing discrete objects I E X' with a probability <xx) x R(x), where X' is a compositional space and R X' R20 non-negative reward function. The generation process of object x X can be represented by a sequence of discrete actions a E A that incrementally modify partially con- structed object, which denoted as state state so and terminate with a special action indicating that the object is complete (s E X'), the construction of an object can be defined as a complete trajectory T = (No >> 81 - - 8n x)).
S. Let generation process begin at a special initial
Following fragment- based molecule design (Bengio et al. 2021a] Xie et al.) 2021), we first define vocabulary of building blocks (molecule fragments), then generate molecular graphs by sequentially attaching a fragment to atom of the partially constructed molecules. There are multiple action sequences leading to the same state, and no fragment deleting actions, the space of possible action sequences thus be denoted by directed acyclic graph (DAG) (S, where the edges in E are transitions s + s' from one state to another. To learn the aforementioned desired policy, Bengio et al. (202Ta) propose the DAG structure a flow network.
4

---

can also Under review as a conference paper at ICLR 2023
Markovian flows. Bengio et al. (2021b) first define trajectory flow FF T R2o the set of all complete trajectories to measure the unnormalized density. The edge flow and state flow can then be defined as F(s + s') = Es-seer F(T) and F(s) Exer F(),, respectively. The trajectory flow F determines probability measure P(r) Erer F(F) If flow F is Markovian, the forward transition probabilities PF can be computed as (s'ssss = F(a-te)
F(T)
F(a)
Flow matching objective. fied Vs S:
A flow is consistent if the following flow consistency equation is satis-
F(s)
Σ
F(8' -) - +
Σ
F(s - +8)
(1)
s'EPag(s)
stI Pag (8*)
where (sg(s) is set of parents in G. As proved in Bengio et al. (021a)) if the flow consistency equation is satisfied with IR(s) for non-terminal state and (()
R(x > 0 for terminal state Pr(s'ls) samples object æ with probability π (xx) R(x)). GFlowNets propose to approximate the edge flow F(s s') using
I, policy defined by the forward transition probability T(s'ls)
neural network (s, s') with enough capacity. such that the flow consistency equation is respected convergence. To achieve this, [Bengio et al. (202Ia) define temporal difference-like (Sutton &
Barto 2018) learning objective, called flow- flow-matching matching (FM):
LFM(S, R2;0) (108 log R(s) PPPo(s" Fo
EvEPoc(s) Po(s',8)
Bengio et al. (2021a) prove that we can use any exploratory policy मे with full support to sample training trajectories and obtain the consistent flow (s, by minimizing the FM objective. Con- sequently, policy defined by this approximate flow (T(s'(s) Pro (s' (s) Fe(s-a') sample objects x with a probability (x) proportionally to reward R(x). Practically, the training trajectories are sampled from exploratory policy which is mixture between PF, and uniform distribution over allowed actions (Bengio et al. 2021a).
Ps(a)
4.2 HYPERNETWORK BBSSED GFLOWNETS
Our proposed HN-GFN aims sampling a diverse batch of candidates from the approximate Pareto front with a unified model. A common approach to MOO is to decompose it into set of scalar optimization problems with different scalarization functions and apply standard objjctive optimization methods to gradually approximate the Pareto front (Knowles 20006 Z & Li 2007). We here consider convex combinations (weighted sum) of the objectives. Let (Ai,... AM) SM be a preference vector defining the trade-off between the competing properties, where SM =
(X E m Σ, A, 1, 2 0} the M simplex. Then the scalarization function can be formulated as (x (z)
To support parallel evaluations in BO, one can obtain candidates according to different scalariza- tions (Daulton al. 2020). Practically, this approach hardly scales efficiently with number of objectives for discrete problems. Taking GFlowNet as an example, we need to train multiple GFlowNets independently for each choice the reward function Rx(x) = sx(x) to cover the ob- jective space:
0 = arg min L Rx)
(3)
Our key motivation is to design unified GFlowNet sample candidates according to different reward functions, even ones not seen during training. Instead of defining the reward function with fixed preference vector X, we propose to train a weeference-conditoonn of reward functions Rx. where the preference vector À is sampled from simplex S
GFlowNet on a distribution
= arg min Exes, Eses RA)
(4)
Note that the preliminary concept of conditional GFlowNet was originally introduced in Bengio et al. (20216). We study and instantiate this concept, aiming to facilitate MOO in the context of molecule design.
5

---

Under review as a conference paper at ICLR 2023
Remarks. Our proposed optimization scheme of training single model to fit family of loss functions fits into the framework of YOTO (Dosovisskiy Djolonga 2019). As proved in Doso-
vitskiy & Djolonga 2019), assuming an infinite model capacity, the proposed optimization scheme (Eq. 4 is as powerful as the original one (Eq, D since the solutions to both loss functions coincide. Nevertheless, the assumption of infinite capacity is extremely strict and hardly holds, how to design the conditioning mechanism in practice becomes crucial.
4.2.1 HYPERNETWORK BASED CONDITIONING MECHANISM
We propose to condition the GFlowNets on the preference vectors via hypernetworks (Ha et al. 2016). Hypernetworks
deep networks that generate the weights of a target network based on inputs. In vanilla GetowNets, the flow predictor Fe is parameterized with the MPNN (Gilmer et al., 2017) over the graph of molecular fragments, with two prediction heads approximating F(s. s') and F(s) based on the node and graph representations respectively. These two heads are parameterized with multi-layer perceptrons (MLPs).
One can view the training of HN-GFN as learning an agent to perform multiple policies that corre- spond different goals (reward functions R) defined the same environment (state space and action space A). Therefore, we propose to only condition the weights of prediction heads 6 pred with hypernetworks, while sharing the weights of MPNN 0 leading to more generalizable state repre- sentations. More precisely, hypernetwork ln:; Φ) takes as inputs the preference vector À to output the weights fored = h(A; ø) of prediction heads in the flow predictor Fg. For brevity, we write 0 (Ompn pred) Following Navon et al. (2021)
parametrize h using MLP with multiple heads, each generating weights for different layers of the target network.
4.2.2 As THE ACQUISITION FUNCTION OPTIMIZER
Training. At each iteration, we first randomly sample a new preference vector À from a Dirichlet distribution Dir(a). Then the HN-GFN is trained in usual manner with the reward function set as (x) = a(u(sx(x)) (sx(x)): where a and a are posterior mean and standard deviation estimated by M.
Sampling. At each round i, we use the trained HN-GFN to sample diverse batch of b candidates. Let A' be the set of target preference vectors Xarges We sample & molecules per Xunget A and evaluate them on the oracle in parallel. In practice, we simply sample Xunget from Dir(a), but worth noting that this prior distribution can also be defined adaptively based on the trade-off of interest. As the number of objectives increases, we choose a larger to cover the objective space.
4.3 HINDSIGHT-LIKE OFT-POLICY STRATEGY
Resorting to the conditioning mechanism, HN-GFN can learn a family of policies to achieve various goals, i.e., one can treat sampling high-reward molecules for particular preference vector separate goal. As verified empirically in Dain et al. (0022), since the FM objective off-policy and offline, we can use offline trajectories to train the target policy for better exploration, so long the assumption full support holds. Our key insight is that each policy can learn from the valuable experience (high-reward molecules) of other similar policies.
To achieve this, inspired by Hindsight Experience Replay (Andrychowicz et al. 2017 in RI, we propose to share nigh-performing molecules among policies by re-examining them with different preference vectors. Because there are infinite possible preference vectors, here we only focus on A which are based on sample candidates round and build replay buffer for each Aurget E A After sampling some trajectories during training, we store in the replay buffers the complete object with the reward Rximm (x).
Algorithm describes the training procedure for HN-GFN with the proposed hindsight-like strategy. At each iteration, we first sample preference vector from mixture between Dir(a:) and uniform distribution over A (1 Dir (a) + Uniform. If A is chosen, we construct half of the training batch with offline trajectories from the corresponding replay buffer of molecules encountered with the highest rewards. Otherwise, we incorporate offline trajectories from the current observed dataset D, instead to ensure that HN-GFN samples correctly in the vicinity of the observed Pareto set.

---

Under review as a conference paper at ICLR 2023
0.53
Polerecsipack Growee
0.50
Ansrage TipK Reaard
0.45
0.64
0.35
0.50
0.30
0.8
0.2
8.1
0.3
0.5
077
0.9
2000
6C0O
6030
9000
Photo Senic WOITE for JMK3
Farnig Stops
Figure 2: Left: The distribution of Top-100 100 JNK3 scores over different preference vectors. Right: The progression of the average Top-20 rewards over the course of training of the HN-GFN in opti- mizing GSK33 and JNK3 with different strategies.
4.4 EVIDENTIAL SURROGATE MODEL
While GPs are well-established in continuous spaces, they scale poorly with the number of obser- vations and do perform well in discrete spaces (Swersky et al.) (020). There has been signif- icant work in efficiently training non-Bayesian neural networks to estimate the uncertainty (Gal & Ghahramani 2016). In this work, we use evidential deep learning (Amini al. 2020) to explicitly learn the epistemic uncertainty. Compared with the widely used MC Dropout (Gal & Ghahramani 2016) and Deep Ensembles (Lakshminarayanai advantages of faster inference speed and superior calibrated uncertainty (Soleimany et al. [2021) I). As for the acquisition function, we use Upper Confidence Bound (Srinivas et al., (0010) incorporate epistemic uncertainty. To be precise, the objectives are modeled with single multi-task network and the acquisition function is applied the scalarization. See Appendix for more discussion.
al. 2017), evidential deep learning presents the
EXPERIMENTS
We first verify that HN-GFN has adequate capacity to generalize over preference vectors in a syn- thetic scenario. Next, evaluate the effectiveness of the proposed MOBO algorithm based on HN-GFN in practical scenarios, which are more in line with real-world molecular optimization. Implementation details and additional results are provided in the Appendix.
5.1 SYNTHETIC SCENARIO
Here, our goal is to demonstrate that we can leverage the HN-GFN to sample molecules with rreferenee-oondttioeed property distributions. The HN-GFN is used a stand-alone optimizer outside of MOBO to directly optimize the scalarizations of oracle scores. As the oracle cannot be called many times as necessary practically, we refer this scenario as synthetic scenario. To better visualize the trend of the property distribution of the sampled molecules the preference vector, we only consider two objectives: inhibition scores against glycogen synthase kinase-3 beta (GNK33) and c-Jun N-terminal kinase-3 (JNK3) (Li et al. 2018 Jin et al. 2020b).
function of
Compared methods. We compare HN-GFN against the following methods. Preference-specific GFlowNet is vanilla GFlowNet trained independently for particular preference vector. Note that the preference-specific GelowNet is treated as "gold standard" rather than the baseline, as trained and evaluated using the same preference vector. Concat-GFN and FiLM-GFN are two variations of the conditional GFlowNet based on FiLM (Perez et al. 2018 and concatenation, re- spectively. MOEA/D (Zhang & Li 2007) and NSGA-III (Deb & Jain 2013) are two multi-objecctive evolutionary algorithms that also incorporate preference information. We perform evolutionary al- gorithms over the 32-dim latent space learned by HierVAE (Jin et al. 2020a). which gives better optimization performance than JT-VAE (Jin et al. 2018).
Metrics. All the above methods are evaluated over the same set of evenly spaced preference vectors. For each GFlowNetthassed method, sample 1000 molecules per preference vector as

---

Under review as a conference paper at ICLR 2023
Table 1: Evaluation of different methods the synthetic scenario
Method
HV
Div
Cor
MOEAID
0.182 # 0.045
n/a
n/a
NSGA-1II
0.364 = 0.041
n/a
n/a 0.653 + 0.003
ceeference-speeffe GFlowNet
0.545 = 0.055
0.786 0.013
Concat- FFNcat-GFN
0.534 # 0.069
0.786
0.004
0.646 a: 0.008
FiLM-GFN
0.431 = 0.045
0.795
0.014
0.633 # 0.009
HN-GFN
0.550 +: 0.074
0.797 0.015
0.666 + 0.010
Table 2: Evaluation of different methods on MOBO scenarios (mean std over runs)
GSK38
JNK3
GSK33 JNK3
QED SA
HV
Div
HV
Div
HicrVAEggParEG0
0,205:00015
n/a
0,186 + 0.009
n/a
HierVAE+/EHVI
0,341 # 0.072
n/a
0.211 # 0.006
n/a
LaMOO
0.279 = 0.090
n/a
0.190 : 0.069
n/a
GP-BO
0,368 0.020
0.347 = 0.059
0,335 0.021
0.562
0.031
MARS
0.418 0.095
0.653 0.072
0.273 0.020
0.754 0.027
HN-GEN
0.572 0.087
0.810 = 0.003
0,389 0.012
0.744 0.008
HN-GFN w/ hindsight
0.669 0.061
0.793 0.007
0.416 0.023
0.738 0.009
the solutions. We compare the aforementioned methods on the following metrics: Hypervolume indicator (HV) measures the volume of the space dominated by the Pareto front of the solutions and bounded from below by the preference point (0. 0). Diversity (Div) is the average pairwise Tanimoto distance over Morgan fingerprints. Correlation (Cor) is the Spearman's rank correla- tion coefficient between the probability of sampling molecules from an external test set under the GFlowNet and their respective rewards in the logarithmic domain (Nica et al.] 2022). See more details in Appendix B.2.2 In nutshell, HV and Div measure the quality of the solutions, while Cor measures how well the trained model is aligned with the given preference vector.
Experimental results. As shown in Table I HN-GFN outperforms the baselines and achieves competitive performance to the preference-specific GFlowNets (gold standard) on all the metrics. Compared the GFlowNet-based methods, the evolutionary algorithms (MOEA/D and NSGA- IIII fail to find high-scoring molecules, especially the MOEA/D. HN-GFN outperforms Concat- GFN and FiLM-GFN in terms of HV and Cor, implying the superiority of the well-designed hypernetwork-baat conditioning mechanism. The comparable performance of HN-GFN and preferencce-peccffe GFlowNets illustrates that HN-GFN can generalize over preference vectors. Therefore, the unified HN-GFN provides significantly efficient way to explore various trade-offs between objectives. In Figure (Left), we visualize the trend of the empirical distribution of JNK3 as the preference weight increases. Intuitively, HN-GFN and preférence-specific GFlowNets show consistent trends: the larger the preference weight, the higher the average score.
5.2 MULTI-OBIECTIVE BAYESIAN OPTIMIZATION
Next, we evaluate the effectiveness of HN-GFN as an acquisition function optimizer within MOBO in the practical scenarios, where there is limited evaluation budget for oracle. We consider the following objective combinations of varying size:
GNK33++JK33: Jointly inhibiting Alzheimer-related targets GNK333 and JNK3.
GNKK3/+NNK3-QH---S Jointly inhibiting GNK3,3 and JNK3 while being drug-like like and -onyttt-yynthesie...
We rescale the SA score such that all the above properties have a range of [0,1] and higher is bet- ter. For both combinations, we consider starting with DDo = 200 random molecules and further querying the oracle N 8 rounds with batch size h 100.
8

---

Under review as a conference paper at ICLR 2023
Baselines. We compare HN-GFN with the following methods (from each category men- tioned in section 2 as the acquisition function optimizer: HierVAE (Jin et al.) 2020a) with iParEGO/GEHVT (Daulton et al. 2020) and LaMOO (Zhao et al. 2022) are LSO methods. GP- BO (Tripp et al., (021) uses Graph GA Jensen 2019) to optimize the acquisition function defined based GP with Tanimoto kernel. MARS (Xie al., (021) applies MCMC sampling to opti- mize the acquisition function defined based on the same surrogate function as HN-GFN. Note that the RL-based P-MOCO (Xi Lin 2022
also implemented but fails optimize the properties.
Experimental results. Table 2 shows that HN-GFN achieves superior performance over the base- lines in terms of HV and Div, especially trained with the hindsight-like off-policy strategy. Note that the Div is computed among the batch of 100 candidates per round, we omit this metric for LSO methods as they only support 160 rounds with batch size due to memory constraint. Our HN-GFN w/ hindsight outperforms the best baselines MARS and GP-BO of the two objective combinations by a large margin (60.0% and 24.2% relative improvement) with respect to HV, respectively. The promising performance can be attributed the ability of HN-GFN to sample diverse batch of can- didates from the approximate Pareto front. Another interesting observation, in the more challenging settings where four objectives are optimized, that MARS generates diverse candidates via MCMC sampling but fails to find high-quality Pareto front, indicating that HN-GFN can find high-reward modes better than MARS. The computational costs are discussed in the Appendix B.4
5.3 ABLATIONS
Effeet of the hindsight-like strategy. In the first round of MOBO, for each Amaget E A we sample 100 molecules every 500 training steps and compute the average Top-20 reward over A. In Figure (Right), as we vary y from to 1, the hindsight-like strategy significantly boosts average rewards, demonstrating that sharing high-performing molecules among policies is effective for speeding up the training of HN-GFN. We choose ? = 0.2 for the desired trade-off between reward and general- ization, see Appendix C.2 for a detailed explanation.
Effect of a. Next, we study the effect of the prior distribution of preference vectors Dir(a). We consider the more challenging ENN33++JNK+OEE+SAA combination, where the difficulty of op- timization varies widely for various properties. Table 3 shows that the distribution skewed toward harder properties results in better optimization performance.
Effect of scalarization functions. In addition the weighted sum (WS), we consider the Tcheby- cheff (Miettinen 2012) that is also commonly used in MOO. Table ssows that Tchebycheff leads a worse Pareto front compared WS. We conjecture that the non-smooth reward landscapes induced by Tehebycheff are harder to optimize.
Table 3: Ablation study of the a and scalarization functions on GNK3
a
scalarization function
(1,1.1,1)
1,13,1.1
13,4,2,1)
WS
Tehehycheff
HV
0.312 0.039
0.385 + 0.018
0.416 0.023
0.416 0.023
0.304 0.075
Div 0.815 # 0.015
0.758 4 0.018
0.738 0.009
0.738 0.009
0.732 = 0.014
6 CONCLUSION
We have introduced a MOBO algorithm for sample-efficient multi-objective molecular optimization. This algorithm leverages a hypemetwork-bbssed GFlowNet (HN-GFN) to sample diverse batch of candidates from the approximate Pareto front. In addition, we present a hindsight-like off-policy strategy to improve optimization performance. Our algorithm outperforms existing approaches on synthetic and practical scenarios. Future work includes extending this algorithm to other discrete optimization problems such biological sequence design and neural architecture search.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Danicla Rus. Deep evidential regression. Advances in Neural Information Processing Systems, 33: 53:14927-14937, 2020.
Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. Advances neural information processing systems, 30, 2017.
Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In International Conference on Learning Representations, 2020. URL Utt.tttp::// //tprr/eeew//eeeeee.... www. new/forum? furum?
id=Hk xbr
Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew Wil- son, and Eytan Bakshy. Botorch: framework for efficient monte-carlo bayesian optimization. Advances in neural information processing systems, 33:21524-21534. 2020.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 14:27381-27399 2021a.
Yoshua Bengio, Tristan Deleu, Edward Hu, Salem Lahlou, Mo Tiwari, and Emmanuel Bengio.
Gflownet foundations. arXiv preprint arXiv:2111 1.09266, 2021b.
Richard Bickerton, Gaia Paolini, Jérémy Besnard, Sorel Muresan, and Andrew Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, (22):90-98, 2012.
Julian Blank and Kalyanmoy Deb. Pymoo: Multi-objective optimization in python. IEEE Access, 8:89497-89509, 2020.
Eric Bradford, Artur M Schweidtmann, and Alexei Lapkin. Efficient multiobjective optimization employing gaussian processes, spectral sampling and genetic algorithm. Journal of global optimization, 71(2):407-43, 2018.
Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hypervolume improvement for parallel multi-objective bayesian optimization. Advances in Neural Information Processing Systems, 33:9851-9866 2020.
Kalyanmoy Deb and Himanshu Jain. An evolutionary many-objective optimization algorithm using reference-pone cotenncepointt-aaaee nondominated sorting approach, part i: solving problems with box con- straints. IEEE transactions evolutionary computation, 18(4)5577-601, 2013.
Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian Structure Learning with Generative Flow Networks. arXiv preprint, 2022.
Alexey Dosovitskiy and Josip Djolonga. You only train once: Loss-conditional training of deep networks. In International conference on learning representations, 2019.
Matthias Ehrgott. Multicriteria optimization, volume 491. Springer Science & Business Media, 2005.
Peter Ert and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug- like molecules based on molecular complexity and fragment contributions. Journal of cheminformatics, 1(1):1-11, 2009.
Tianfan Fu, Wenhao Gao, Cao Xiao, Jacob Yasonik, Connor W Coley, and Jimeng Sun. Differ- entiable scaffolding tree for molecule optimization. In International Conference on Learning Representations, 2022.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
10

---

Under review as a conference paper at ICLR 2023
Wenhao Gao, Tianfan Fu, Jimeng Sun, and Connor W. Coley. Sample efficiency matters: Bench- marking molecular optimization. In ICML 2022 2nd Al for Science Workshop, 2022. URL
lttps://cpenrevier net heet//oouummidduu2crtttett
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp.
1263-1272. PMLR, 2017.
Rafael Gömez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling Dennis Sheberla, Jorge Aguilera-lparaguirre, Timothy D Hirzel, Ryan P Adams, and Alán Sapuru-Guzik Automatic chemical design using data-driver contin- uous representation of molecules. ACS central science, 4(2):268-21) 2018.
David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv: 1 100116699999000 2016.
Daniel Hernández-Looato. Jose Hernandez-Looato, Amar Shah, and Ryan Adams. Predictive en- tropy search for multi-objective bayesian optimization. International conference on machine learning, pp. 1492-1501. PMLR, 2016.
Moksh Jain, Emmanuel Bengio, Alex Hernandez-Gareia, Jarrid Rector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning, pp. 9786-9801. PMLR, 2022.
Jan H Jensen. A graph-based genetic algorithm and generative model/monte carlo tree search for the exploration of chemical space. Chemical science, 10(12)3357-3572,2222 2019.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junetion tree variational autoencoder for molecular graph generation. In International conference on machine learning, pp. 2323-2332. PMLR, 2018
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In International conference on machine learning, pp. 2020a.
PMLR,
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multi-objective molecule generation using interpretable substructures. In International conference on machine learning, pp. 4849-4859. PMLR, 2020b.
Donald R Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455-499 1998.
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Neural architecture search with bayesian optimisation and optimal transport. Advances in neural information processing systems, 31, 2018.
Joshua Knowles. Parego: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Transactions on Evolutionary Computation, 10(1): 50 50-66, 2006.
Mina Konakovic Lukovic, Yunsheng Tian, and Wojciech Matusik. Diversity-guided multi-objective bayesian optimization with batch evaluations. Advances in Neural Information Processing Systems, 33:17708- 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Yibo Li, Liangren Zhang, and Zhenming Liu. Multi-objective de novo drug design with conditional graph generative model. Journal of cheminformatics, 10(1):1-24, 2018.
Natalie Maus, Haydn T Jones, Juston S Moore, Matt J Kusner, John Bradshaw, and Jacob R Gardner. Local latent space bayesian optimization over structured inputs. arXiv preprint arXiv:2201. 11872, 2022.
11

---

Under review as a conference paper at ICLR 2023
Kaisa Miettinen. Nonlinear multiobjective optimization, volume 12. Springer Science & Business Media, 2012.
Henry Moss, David Leslie, Daniel Beck, Javier Gonzalez, and Paul Rayson. Boss: Bayesian op- timization over string spaces. Advances neural information processing systems, 33:15476- 15486, 2020.
Aviv Navon, Aviv Shamsian, Ethan Fetaya, and Gal Chechik, Learning the pareto front with hyper- networks. In ICLR, 2021.
Andrei Cristian Nica, Moksh Jain, Emmanuel Bengio, Cheng-Hao Liu, Maksym Korablyov, Michael M Bronstein, and Yoshua Bengio. Evaluating generalization in gflownets for molecule design. In ICLR2022 Machine Learning for Drug Discovery, 2022.
Biswajit Paria, Kirthevasan Kandasamy, and Barnabás Póczos. A flexible framework for multi- objective bayesian optimization using random scalarizations. In Uncertainty in Artificial Intelligence, pp. 766-776. PMIR, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems. 32, 2019.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Pavel G Polishchuk, Timur I Madzhidov, and Alexandre Varnek. Estimation of the size of drug- like chemical space based on gdb-17 data. Journal of computer-aided molecular design, 27(8): 675-679, 2013.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pp. 63-71. Springer, 2003.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):
145-175,7 2015.
Ava P Soleimany, Alexander Amini, Samuel Goldman, Daniela Rus, Sangeeta N Bhatia, and Con- nor Coley. Evidential deep learning for guided molecular property prediction and discovery. ACS central science, 13):11356 2021.
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process opti- mization in the bandit setting: no regret and experimental design. In Proceedings of the 27th International Conference on International Conference on Machine Learning, pp. 1015-1022, 2010.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Kevin Swersky, Yulia Rubanova, David Dohan, and Kevin Murphy. Amortized bayesian optimiza- tion over discrete spaces. Conference on Uncertainty Artificial Intelligence, pp. 769-778. PMLR, 2020.
Austin Tripp, Erik Daxberger, and José Miguel Heráández-Lobato. Sample-efficient optimiza- tion in the latent space of deep generative models via weighted retraining. Advances in Neural Information Processing Systems, 33:1 12111119911222 12222, 2020.
Austin Tripp, Gregor NC Simm, and José Miguel Hemándezz-obatte A fresh look de novo molecular design benchmarks. NeurIPS 2021 AI for Science Workshop, 2021.
Qingfu Zhang Xi Lin, Zhiyuan Yang. Pareto set learning for neural multi-objective combinatorial optimization. In International Conference on Learning Representations, 2022. URL https: //openreviee ew. net/fformm wttp:r/wwwmm.mmooooooooo0000.
12

---

Under review as a conference paper at ICLR 2023
Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. Mars: Markov molecular sampling for multi-objective drug discovery. International Conference on Learning Representations, 2021.
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. Advances in neural information processing systems, 31, 2018.
Dinghuai Zhang, Nikolay Malkin, Zhen Liu, Alexandra Volokhova, Aaron Courville, and Yoshua Bengio. Generative flow networks for discrete probabilistic modeling. arXiv preprint arXiv:2202.0136h 2022.
Qingfu Zhang and Hui Li. Moea/d: A multiobjective evolutionary algorithm based on decomposi- tion. IEEE Transactions evolutionary computation, 11(6):712-731, 2007.
Yiyang Zhao, Linnan Wang. Kevin Yang, Tianjun Zhang, Tian Guo, and Yuandong Tian. Multi- objective optimization by learning space partition. In International Conference on Learning Representations, 2022. /otpp//eee net. het::
Eckart Zitzler and Lothar Thiele. Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE transactions on Evolutionary Computation, 3(4):257-271, 1999.
13

---

Under review as a conference paper at ICLR 2023
A ALGORITHMS
describes the overall framework of the proposed MOBO algorithm, where HN-GFN leveraged as the acquisition function optimizer. Algorithm describes the training procedure for HN-GFN within MOBO.
Algorithm [
Algorithm MOBO based on HN-GFN
Input: oracle f = (f1..... fas) ). initial dataset Do = {(x (a acquisition function a, parameter of Dirichlet distribution a, number of rounds N. batch size b Initialization: surrogate model M. parameters of HN-GFN T
for i 1 to N do
Fit surrogate model M on dataset D;-1
Sample the set of target preference weights A Dir(a)
Train Te with reward function (x(x) = a(((s)(x)) (x)), (sx(2));
D Algorithm 2
Sample query batch B, { based on Atarge E
Evaluate batch B; with f and augment the dataset Di+1 = D; U ((t),
end for
Algorithm Training procedure for HN-GFN with the hindsight-like off-policy strategy
Input:
available dataset D. reward function R. minibatch size set of target preference vectors A. proportion of hindsight- strategy 7, replay buffers {Rx} Jaea while not converged do Flag ~ Bernoulli(y)
if Flag 1 then
A ~ A
Sample z trajectories from replay buffer RA
else
X Dir(a)
m trajectories from the available dataset D; Sample 2
end if
0 = (0mpen: o))
Sample mh trajectories from policy से and store terminal states x in R2 for all Compute reward R2 (x) on terminal states from each trajectory in the minibatch Update parameters mpma and Φ with a stochastic gradient descent step w.r.t Eq, 2
end while
B IMPLEMENTATION DETAILS
B.1 EXPERIMENTAL SETTINGS
B.1.1 MOLECULE DOMAIN
Following (Bengio al., 2021b), the molecules are generated based on a of 105 building blocks. The same substructure containing multiple stems (atoms for linking another building block) is served as separate building blocks. We allow the GFlowNet to sample molecules with 2-8 blocks. As for the oracles, adopt the property prediction models (random forest) released by (Xie et al. 2021 to evaluate the inhibition ability of generated molecules against GSK33 and JNK3.
1.1.2 METRICS
Diversity. Diversity (Div) is the average pairwise Tanimoto distance over Morgan fingerprints. In the synthetic scenario, for each preference vector, sample 1000 molecules, calculate the Div among the Top-100 molecules. and report the averages over preferences. In MOBO, the DiV is computed among the batch of 100 candidates per round, GP-BO and MARS are not preference- conditioned. And we believe this metric possibly more aligned with how these methods might be used in biology chemistry experiment.
14

---

Under review as a conference paper at ICLR 2023
Correlation. Correlation (Cor) is the Spearman's rank correlation coefficient between the prob- ability of sampling molecules from an external test set under the GFlowNet and their respective rewards in the logarithmic domain: Cor Spearman's (l R(()) The external test set is obtained in two steps: First, we generate a random dataset containing 300K molecules uniformly based the number of building blocks; Next, we sample the test sets with uniform property distri- bution corresponding to GSK33 and JNK3, respectively, from the 300K molecules. The final test contains 6062 molecules.
B.2 BASELINES
All the baselines are implemented using the publicly released source codes with adaptations for our
MOBO scenarios. Our evolutionary algorithms are implemented in PyMOO (Blank & Deb 2020). and the LSO methods are implemented BoTorch (Balandat et al., 2020). LaMOO and GP-BO utilize EHVI the acquisition function. For all GP-bar GP-based methods, each objective is modeled by an independent GP.
B.3 HN-GFN
We implement the proposed HN-GFN in PyTorch (Paskke et al. 2019). The values of key hyper- parameters are illustrated in Table
Surrogate model: We use the 12-layer MPNN as the base architecture of the surrogate model in our experiments. MOBO, a single multi-task MPNN is trained with batch size of 64 using the Adam optimizer with dropout rate of 0.1 and weight decay rate of le-6. We apply early stopping to improve generalization.
HN-GFN: HN-GFN contains vanilla GelowNet and inféerence-conditiionne hypernetwork. The architecture of GFlowNet is 10-layer MPNN, and the hypernetwork is a 3-layer MLP with multiple heads, each generating weights for different layers of the target network. The HN-GFN is trained with Adam optimizer to optimize the Flow Matching objective.
B.4 EMPIRICAL RUNNING TIME
The efficiency is compared on the same computing facilities using Tesla V100 GPU. In the context of MOBO, the running time of three LSO methods (i.e., HierVAE+gParEGO, HierVAE+gEHVI, and LaMOO) is around 3 hours, while GP-BO optimizes much faster and costs only 13 minutes. In contrast, the time complexity of deep-learning-bue MARS costs 32 hours, while our proposed HN-GFN costs 10 hours. With the hindsight-like training strategy, the running time HN-GFN will increase roughly by 33%.
discrete optimization methods much larger.
However, if we look at the problem in a bigger picture, the time costs for model training are most likely negligible comparison to those of evaluating the molecular candidates real-world appli- cations. Hence, we argue that the high quality of the candidates (the performance of the MOBO algorithm) is more essential than having a lower training cost.
C ADDITIONAL RESULTS
C.1 SYNTHETIC SCENARIO
As illustrated in Figure 3 the distribution of Top- GSK33 scores shows consistent trend in preference-specife GFlowNet and our proposed HN-GFN, although the trend not significant as the JNK3 property.
C.2 EFFECT OF THE HINDSIGHT-LIKE STRATEGY
There is a trade-off between reward and generalization. As we vary y from 0 to 1, the training distribution of preference vectors moves from Dir(a) to the of target preference vectors A.
15

---

Under review as a conference paper at ICLR 2023
Table 4: Hyper-paramettrs used in the real-world MOBO experiments.
Hyper-parameter
GSK33 + JNK3
GSK33 + JNK3 QED + SA
Surrogate model
Hidden size
64
64
Learning rate
2.5e-4
le-3
À for evidential regression
0.1
0.1
Number iterations
10000
10000
Early stop patience
500
500
Dropout
0.1
0.1
Weight decay
le-6
le-6
Acquisition function (UCB)
0.1
0.1
HN-GFN
Learning rate
5e-4
5e-4
Reward exponent
8
Reward norm
1.0
1.0
Trajectories minibatch size
8
Offline minibatch size hindsight y
8
8
0.2
0.2
Uniform policy coefficient
0.05
0.05
Hidden size for GFlowNet
256
256
Hidden size for hypernetwork
100
100
Training steps
5000
5000
(1,1)
(3,4,2,1)
Preference GRowket
0.75
Top 100
0.9
0.7
0.5
0.3
0.1
Preference weight GSK33
Figure 3: Comparison of the distribution of 1op-100 GSK33 scores sampled by different pref- erence vectors using preference-specific GFlowNets and HN-GFN.
Exclusively training the HN-GFN with the finite target preference vectors lead poor gen- cralization. In practice, although we only sample candidates based on A, we argue that it is vital to keep the generalization such that we can leverage the trained HN-GFN explore various preference vectors adaptively. In Figure 2 (Right), we found that increasing leads to slight (not significant) improvement average reward compared to 7 0.2. Hence, we believe 0.2 is the desired trade-off.
C.3 EFFECT OF SURROGATE MODELS
We conduct ablation experiments to study the effectiveness of different surrogate models. We sider the following three surrogate models: evidential regression (Amini et al., 2020). Deep En- sembles (Lakshminarayanna et al. 2017), and GP based on Tanimoto kernel (Tripp et al. 2021). As
16

---

Under review as a conference paper at ICLR 2023
shown in T observe that evidential regression leads to better optimization performance than Deep Ensembles. While the HV of evidential regression and GP comparable, evidential re- gression propose more diverse candidates. Furthermore, argue that GP is less flexible over discrete spaces than evidential regression and Deep Ensembles, as different kernels need to be de- signed according the data structures.
Table 5: Evaluation of different surrogate models MOBO scenarios
GSK38 JNK3
GSK33 JNK3
QED SA Div
Div
HV
HN-GFN (Evidential)
0.669 0.061
0.793 = 0.007
0.416 = 0,023
0.738 = 0.009
DFNGFN (Ensemble)
0,583 0.103
0.797 + 0.004
0.355 = 0,048
0.761 = 0.012
HN-GPN (GP)
0.662 0.054
0.739 = 0.008
0.421 0.037
0.683 = 0.0188
C.4 SAMPLED MOLECULES IN MOBO EXPERIMENTS
We give some examples of sampled molecules from the Pareto front by HN-GFN in the GSK3 JNK3 + QED + SA optimization setting (Figure 4 The numbers below each molecule refer to GSK33, JNK3, QED, and SA scores respectively.
HN
0.93, 0.34, 0.72, 0.83
0.79, 0.83, 0.59, 0.85
0.75, 0.42, 0.59, 0.94
0.84, 0.22, 0.83, 0.86
Figure 4: Sampled molecules from the approximate Pareto front by HN-GFN.
17


Reviews:
[{"review_id": "fd-k09hZLz", "reviewer": "Reviewer_4QvB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work investigate the possibility of using GFlowNets to tackle multi-objective sampling problems. Specifically, hypernetwork-based GFlowNet is proposed for solving multi-objective Bayesian optimization problems. Some insights from reinforcement learning is also involved. The proposed algorithm is evaluated in molecule generation tasks.", "summary_of_the_review": "The method of this paper is clean and straightforward, while this work still suffers from non-extensive experiments and insufficient ablation to justify some usage of the components. I understand that it is a tight schedule for the authors to rebuttal, but I would of course consider raising the score if some of the main concerns are addressed.", "strength_and_weaknesses": "## Strengths\n1. The methodology is well-formulated, with careful design for amortizing multi-objective problems and surrogate modeling.\n2. The off-policy property of GFlowNet is examined, including a mix of different training trajectory distribution, and the use of hindsight replay idea.\n\n## Weaknesses\n1. I would expect some illustration / analysis on at least toy tasks to show that the proposed method indeed learn the multi-objective behavior. E.g., when the conditioning preference is changed, the distribution of GFlowNet will change accordingly.\n2. The diversity should be measured for every fixed preference. If allowed different preference, there will be of course diverse samples generated by the GFlowNet. \n3. Is there any particular reason to model the preference conditioning with a hypernetwork?  Hypernetwork is not very stable in training. What's more, usually it is enough to directly augment the input with preference for the conditioning, which is also much easier. I would expect the authors to justify this choice with empirical evidence.\n4. Is there any particular reason to use evidental regression? According to my experience, sometimes it is not as reliable as more \"traditional\" methods like deep ensemble. Also, it seems HierVAE-based methods and GP-BO still uses GP, while the proposed method uses evidental regression. Is this an unfair comparison? I would expect at least some results about using HN-GFN with GP to justify this choice.\n\n### Minors\n- The experiments are limited to molecule generation. The multi-objective problems are much more than small graph generation. Other applications, such as protein, sequences, are also of great importance. \n- Is the set of target preference vectors $\\Lambda$ kept fixed? If not, how to update it?\n- In Fig.2 (left), why HN-GFN could be better than the \"gold-standard\" preference-specific GFlowNet? Do these two use the same GFlowNet architecture?\n- Fig.2 (right) shows the performance with $\\gamma=0, 0.2$, but it would be great to see a spectrum from 0 to 1. This is important for the choice of hyper parameter $\\gamma$, which seems not mentioned in the paper. Please correct me if I miss anything!\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YpFe6UMejWg", "reviewer": "Reviewer_44K1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Practical molecule generation involve optimization of multiple objectives simultaneously. These objectives are often expensive to evaluate making sample-efficiency key. The paper proposes a multi-objective Bayesian optimization method leveraging GFlowNets for optimizing the acquisition function. GFlowNets learn stochastic policies to generate discrete objects proportionally to their rewards, resulting in diverse candidates. The authors consider a preference-based decomposition of the MOO problem. The paper proposes a hypernetwork-based parameterization for conditioning on preferences. The authors also propose a hindsight-experience replay based strategy for leveraging offline data during learning. The authors use an evidential regressor as the surrogate model within the multi-objective Bayesian optimization context. The authors then present results on a molecule generation task with 4 objectives. ", "summary_of_the_review": "In summary, while the paper presents an interesting GFlowNet-based approach to tackle multi-objective optimization, there are several major shortcomings in the paper in terms of the empirical analysis, baselines and framing of contributions. In the current state I lean towards rejection but encourage the authors to incorporate the feedback to improve the paper during the discussion. ", "strength_and_weaknesses": "**Strengths**\n\n- The paper tackles an important and challenging problem of multi-objective optimization in the context of molecule generation. As shown in previous work, using GFlowNets for optimizing the acquisition function results in diverse candidates and sample-efficient optimization\n- The hypernetwork-based approach is an interesting way to implement conditioning, in contrast to FiLM based approaches. \n\n**Weaknesses**\n- The paper overall is not very clearly written (I discuss this in more detail in the next section)\n- The paper uses the preference-conditional GFlowNet formulation originally proposed in [1], but does not cite the paper where the preference-conditional GFlowNet is introduced. There are also other other inconsistent citations which I discuss in the next section,\n- Multi-Objective REINFORCE [2] is very closely related to the proposed HN-GFN approach, differing only in the learning objective, but is not discussed / included as a baseline. Additionally the authors also do not include recent approaches such as LaMOO [3] in the baselines. \n- Aside from the baselines, the experiments seem somewhat limited. While the method enjoys superior performance in the task studied in the paper, it is not clear how well it generalizes to different settings and even different rewards for instance. The authors also provide only limited ablations to investigate the method. For instance it is not clear how the set of preference vectors is selected and what is the effect of the distribution of preference vectors used in training. \n- Minor: The authors use evidential regression for the surrogate model claiming \"evidential deep learning presents the\nadvantages of faster inference speed and superior calibrated uncertainty\" however, recent work [4] has established that such approaches can be arbitrarily miscalibrated. \n\n\n[1]  - GFlowNet Foundations\n\n[2] - Pareto Set Learning for Neural Multi-Objective Combinatorial Optimization\n\n[3] - Multi-objective Optimization by Learning Space Partitions\n\n[4] - Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Pkt-3_ra1t8", "reviewer": "Reviewer_NQij", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a multi-objective Bayesian optimization approach for the molecules design problem. The proposed approach uses the hypernetwork-based GFlowNets as an acquisition function optimizer and uses a scalarization approach to combine the multiple objectives.\n", "summary_of_the_review": "The paper addresses an important problem however, the novelty and experimental setup are limited. ", "strength_and_weaknesses": "Strengths: \n+ The paper presents an important scientific application. \n+ The paper presents some promising experimental results, though I have some reservations about the robustness of the results\n+ The paper is easy to follow \n+ The paper addresses the multi-objective problem, which is relatively less studied in the context of molecules, but it is worth noting that it has been recently extensively studied in the general Bayesian optimization problem\n\nWeaknesses\n\n+ The proposed technique is a direct combination of existing techniques with no new substantial addition. Therefore, the technical contribution and novelty are weak\n+ The paper uses the following statement “We assume that the oracle can be called as many times as necessary.” In Expensive settings, this is not usually true. \n+ The paper does not provide any time complexity analysis of the training. This is problematic because existing approaches are actually very fast, while gflownet is certainly much more expensive, so a time comparison and a discussion about complexity and tradeoffs are important.\n+ The paper discusses the sampling of the scalars extensively, but later in experiments, it is mentioned that 5 evenly-spaced preference vectors are used. It is not clear how this works exactly. \n+ State of the art \n    - State-of-the-art methods in molecular optimization are not stated or compared to.  The following is considered SOTA work and covers a wide range of relevant methods and benchmarks that should be discussed for fairness. [1]\n    - Most multi-objective BO papers are not mentioned nor compared to. In batch optimization the most efficient and high-performing methods are [2,3,4]. It is also misleading to state that no previous paper discussed diversity while [2] is a diversity-oriented method, and several single objective batch BO papers discussed diversity. \n    - The paper uses a scalarization technique where scalars are sampled from a distribution. This technique was previously proposed and used [5]. \n    - The Preference-based problem has been studied beyond the discussion that was mentioned in the paper. It is concerning to completely ignore principled existing work and claim it as a novelty. The following are some of the approaches, to name a few [6,7,8]\n\n+ The experimental setup is weak and surprising:\n- the paper states in the beginning that the evaluation was on several synthetic experiments and real-world experiments, but there are only two experiments. \n    - The paper uses three runs only to report the mean and standard error. BO papers report AT LEAST 10 runs usually, and most recent papers report 50 to 100 runs. I don’t think 3 runs can provide any statistical significance or deliver any conclusions about performance. In fact, even expensive deep learning models are usually tested with a higher number of runs. \n    - The diversity is not reported for some of the algorithms. If diversity is a metric applied to the Pareto front, why can’t it be applied to some of the approaches? \n    - The paper reports results for batch size 100 only. Batch BO papers usually evaluate several batch sizes. \n    - There is a total absence of many relevant baselines from molecular optimization, Bayesian optimization, and preference-based optimization. The paper is mainly experimental since the technical novelty is weak. Therefore, it needs to present a thorough experimental evaluation. \n\n[1] Maus, Natalie, et al. \"Local Latent Space Bayesian Optimization over Structured Inputs.\" arXiv preprint arXiv:2201.11872 (2022).\n\n[2] Konakovic Lukovic, Mina, Yunsheng Tian, and Wojciech Matusik. \"Diversity-guided multi-objective bayesian optimization with batch evaluations.\" Advances in Neural Information Processing Systems 33 (2020): 17708-17720.\n\n[3] Eric Bradford, Artur M Schweidtmann, and Alexei Lapkin. Efficient multiobjective optimization employing gaussian processes, spectral sampling and a genetic algorithm. Journal of global optimization, 71(2):407–438, 2018.\n\n[4] Syrine Belakaria and Aryan Deshwal. Uncertainty-aware search framework for multi-objective bayesian optimization. In AAAI Conference on Artificial Intelligence (AAAI), 2020.\n\n[5] Paria, Biswajit, Kirthevasan Kandasamy, and Barnabás Póczos. \"A flexible framework for multi-objective bayesian optimization using random scalarizations.\" Uncertainty in Artificial Intelligence. PMLR, 2020.\n\n[6] Abdolshah M, Shilton A, Rana S, Gupta S, Venkatesh S. Multi-objective Bayesian optimization with preferences over objectives. Advances in neural information processing systems. 2019;32.\n\n[7] Taylor, Kendall, et al. \"Bayesian preference learning for interactive multi-objective optimization.\" Proceedings of the Genetic and Evolutionary Computation Conference. 2021.\n\n[8] Lin, Zhiyuan Jerry, et al. \"Preference Exploration for Efficient Bayesian Optimization with Multiple Outcomes.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2022.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JNPFJET4R6", "reviewer": "Reviewer_ZgdY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for multi-objective Bayesian optimization for molecular optimization. The proposed method uses GFlowNets to optimize the acquisition function in BO, and uses a hypernetwork-based method to incorporate the preference vector into GFlowNets such that a diverse set of points can be sampled from the Pareto front.", "summary_of_the_review": "The paper solves an important problem for molecular optimization using GFlowNets, and I don't have major concerns about the paper. The concerns I listed under \"Weaknesses\" are mostly regarding the writing of the paper.", "strength_and_weaknesses": "Strengths:\n- The proposed method is intuitive and modifies GFlowNets in a reasonable way to facilitate multi-objective optimization.\n- The experiments are nicely done, and the experimental results look promising.\n- The paper is in general well written, and the proposed method is well motivated.\n\nWeaknesses:\n- I feel that some of the algorithmic details are not clearly explained, particularly the connection between Algorithm 1 (for training HN-GFN) and BO. For example, does the dataset $\\mathcal{D}$ correspond to the currently available observations from all previous iterations of BO? Does the reward function $R$ here correspond to the acquisition function calculated in BO? How is the set of target preference vectors built? More importantly, do you need to run Algorithm 1 after every iteration (or every batch) of BO? If yes, then the computational costs may become an issue and hence should discussed.\n- I think Section 4.3 needs to be revised to make it clearer, in the current form, it's not easy to understand.\n- Top paragraph of page 2: it's still unclear to me why limitation 1) makes \"existing discrete molecular optimization methods\" not applicable as \"acquisition function optimizer\". Can't you simply use those acquisition functions which directly take diversity into account? For example, if you use the GP-BUCB acquisition function from paper [1] below, to select an input in a batch, you can simply invoke an existing discrete molecular optimization method to maximize the acquisition function (whose GP posterior standard deviation is updated every time a new input is selected), which will naturally lead to a diverse set of inputs in a batch.       \n[1] Parallelizing Exploration-Exploitation Tradeoffs in Gaussian Process Bandit Optimization, JMLR 2014\n- Top paragraph of page 9: the number of rounds $N=8$ is in fact unusually small in BO, and the batch size $b=100$ is also unusually large for BO as well. Are these choices the common practice in molecular optimization using GFlowNet?\n- (minor) In the Related Work section, the previous works on multi-objective BO should also be discussed.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject