Sample-efficient Multi-objective Molecular Optimization with GFlowNets
Authors: Yiheng Zhu
Jialu Wu
Chaowen Hu
Jiahuan Yan
Chang-Yu Hsieh
Tingjun Hou
Jian Wu
Venue: NeurIPS 2023 poster (2023)
Keywords: drug discovery
multi-objective molecular optimization
Bayesian optimization
generative flow networks
TL;DR: A preference-conditioned GFlowNet-based Bayesian optimization algorithm for sample-efficient multi-objective molecular optimization

Abstract: Many crucial scientific problems involve designing novel molecules with desired properties, which can be formulated as a black-box optimization problem over the *discrete* chemical space. In practice, multiple conflicting objectives and costly evaluations (e.g., wet-lab experiments) make the *diversity* of candidates paramount. Computational methods have achieved initial success but still struggle with considering diversity in both objective and search space. To fill this gap, we propose a multi-objective Bayesian optimization (MOBO) algorithm leveraging the hypernetwork-based GFlowNets (HN-GFN) as an acquisition function optimizer, with the purpose of sampling a diverse batch of candidate molecular graphs from an approximate Pareto front. Using a single preference-conditioned hypernetwork, HN-GFN learns to explore various trade-offs between objectives. We further propose a hindsight-like off-policy strategy to share high-performing molecules among different preferences in order to speed up learning for HN-GFN. We empirically illustrate that HN-GFN has adequate capacity to generalize over preferences. Moreover, experiments in various real-world MOBO settings demonstrate that our framework predominantly outperforms existing methods in terms of candidate quality and sample efficiency. The code is available at https://github.com/violet-sto/HN-GFN.

Sample-efficient Multi-objective Molecular Optimization with GFlowNets
Yiheng Zhu'; Jialu Wu²; Chaowen Hu², Jiahuan Yan',
Chang-Yo Yu Hsieh²t Tingjun Hou? Jian W5..5.6f
College of Computer Science and Technology, Zhejiang University
"College Pharmaccutical Sciences, Zhejiang University
PPolttehniic Institute, Zhejiang University
Second Affiliated Hospital School of Medicine, Zhejiang University
Schooo of Public Health, Zhejiang University
IInttittee of Wenzhou, Zhejiang University
(zhuyiheng2020, jialuwu, ChaowenHu, jyansir, kimhsieh, tingjunhou, wujian2000) @ @zju.edu.cn
Abstract
Many crucial scientific problems involve designing novel molecules with desired properties, which can be formulated black-box optimization problem over the discrete chemical space. In practice, multiple confliciigg objectives and costly evaluations (e.g., wet-lab experiments) make the diversity of candidates paramount. Computational methods have achieved initial success but still struggle with considering diversity in both objective and search space. To fill this gap, we propose a multi-objective Bayesian optimization (MOBO) algorithm lever- GFlowNets (HN-GFN)
aging the hypernetwork-basse optimizer, with the purpose of sampling a diverse batch of candidate molecular graphs from an approximate Pareto front. Using single preferennee-oonditione hypernetwork, HN-GFN learns to explore various trade-off's between objectives. We further propose hindsight-like off-policy strategy share high-performing molecules among different preferences in order to speed up learning for HN- GFN. We empirically illustrate that HN-GFN has adequate capacity generalize over preferences. Morcover, experiments in various real-world MOBO settings demonstrate that our framework predominantly outperforms existing methods in terms of candidate quality and sample efficiency. The code is available at https://gtt wht..t.......... cww wtt.:/villetssoo///oo
acquisition function
1 Introduction
Designing novel molecular structures with desired properties, also referred as molecular optimiza- tion, is crucial task with great application potential in scientific fields ranging from drug discovery material design. Molecular optimization can be naturally formulated as a black-bo optimization problem over the discrete chemical space, which is combinatorially large [52]. Recent years have witnessed the trend to leverage computational methods, such as deep generative models (31] and combinatorial optimization algorithms [62 (30], to facilitate optimization. However, the applicability of most prior approaches real-world scenarios is hindered by two practical constraints: (i) Chemists commonly seek to optimize multiple properties of interest simultaneously [60] (3). For example, in addition to effectively inhibiting a lissass--assciatee target, an ideal drug is desired to be easily syn- thesizable and non-toxic. Unfortunately, objectives often conflict, in most cases, there is single
"Equal contribution. t Corresponning authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Training: : A-Dir(() (n) Sampling: Asorges
a. (Happe (read)
S
$2
Sp-1
S
Hyperswwook A0)
b
Surtogate Model Mr puo
R2 (x
Rs
Rx(x.)
Evalusted betch
Rx ((()))
represents the initial state and D represents the complete object
Figure 1: MOBO loop for molecular optimization using a surrogate model M for uncertainty estimation and HN-GFN for acquisition function optimization. In each round, the policy Ty is trained with reward function Rx, where À is sampled from Dir(a) per iteration. A new batch of candidates sampled from the approximate Pareto front according to target preference vectors Auaget E A.
optimal solution, but rather a set of Pareto optimal solutions defined with various trade-offs [8] 45
(ii) Realistic oracles (e.g., wet-lab experiments and high-fieelity simulations) require substantial costs to synthesize and evaluate molecules 22 Hence, the number of oracle evaluations notoriously limited. In such scenarios, the diversity of candidates is critical consideration.
provides a sample-efficient framework for globally optimizing expensive black-box functions. The core idea is to construct a cheap-to-evaluate surrogate model approximating the true objective function (also known the oracle) the observed dataset and optimize an acquisition function (built upon the surrogate model) to obtain informative candidates with high utility for the next round of evaluations. Due to the common large-batds and low-round settings in biochemical experiments B) batch BO is prioritized to shorten the entire cycle of optimization [25]. MOBO also received broad attention and achieved promising performance in continuous problems by effectively optimizing differentiable acquisition functions [12 13 Nevertheless, it is less prominent in discrete problems, where no gradients can be leveraged to navigate the search space.
Bayesian optimization (BO) 34 54]
Although existing discrete molecular optimization methods can be adopted as the ccquisition.function optimizer alleviate this issue, they can hardly consider diversity in both search and objective space: 1) many neglect the diversity of proposed candidates in search space 330 33]; 2) many multi-objective methods simply rely on a predefined scalarization function (c.g., mean) to turn the multi-objective optimization (MOO) problem into a singlc-objective one [61 20 [6] model cannot exactly reproduce the oracle's 's full behaviors and the optimal trade-off is unclear before optimization (even with domain knowledge), it is desired to not only propose candidates that bring additional information about the search space but also explore more potential trade-offs of interest. To achieve this goal, we explore how to extend the recently proposed generative flow networks (GFlowNets) 6 7 class of generative models that aim to learn stochastic policy for sequentially constructing objects with probability proportional to reward function, to facilitate multi-objective molecular optimization. Compared with traditional combinatorial optimization methods, GFlowNets possess merit in generating diverse and high-reward objects, which has been verified in single-objectio problems 66 (28).
As the surrogate
In this work, we present a MOBO algorithm based on GFlowNets for sample-efficient multi-objective molecular optimization. We propose hypernetwork-basd function optimizer within MOBO to sample a diverse batch of candidates from an approximate Pareto front. Instead of defining a fixed reward function as usual in past work [6], we train a unified GFlowNet on the distribution of reward functions (random scalarizations parameterized by preference vectors) and control the policy using single preference-condiiionel hypernetwork. While sampling candidates, HN-GFN explores various trade-offs between competing objectives flexibly by varying the input preference vector. Inspired by hindsight experience replay 22, we further introduce hindsight-likk off-policy strategy share high high-performing molecules among different preferences
GFlowNet (HN-GFN) the acquisition

---

and speed up learning for HN-GFN. As detailed in our reported experiments, we first empirically verify that HN-GFN capable of generalizing over preference vectors, then apply the proposed framework to real-world scenarios and demonstrate its effectiveness and efficiency over existing methods. Our key contributions are summarized below:
We introduce GFlowNNet-based MOBO algorithm to facilitate real-world molecular optimization. We propose HN-GFN, a conditional variant GFlowNet that can efficiently sample candidates from an approximate Pareto front.
We introduce a hindsight-like off-policy strategy to speed up learning in HN-GFN.
Experiments verify that our MOBO algorithm based on HN-GFN can find high-quality Pareto front more efficiently compared to state-of the-art baselines.
2
Related Work
Molecular optimization. Molecular optimization has been approached with wide variety of computational methods, which can be mainly grouped into three categories: 1) Latent space optimization (LSO) methods [24 31 58 66 perform the optimization over the low-dimensional continuous latent space learned by generative models such as variational autoencoders (VAEs) [36].
These methods require the latent representations to be discriminative, but the training of the generative model is decoupled from the optimization objectives, imposing challenges for optimization (58).
Instead of navigating the latent space, combinatorial optimization methods search for the desired molecular structures directly in the explicit discrete space with 2) evolutionary algorithms (30) and 3) reinforcement learning (RL) [62] guiding the search. However, most prior methods only focus on single property, from non-biological properties such drug- likelliselsness (QED) [ and
synthetic accessibility (SA) [19]. to biological properties that measure the binding energy to a protein target [6). Multi-objecctive molecular optimization has recently received wide attention B3] 61 20). For example, MARS [6 employs Markov chain Monte Carlo (MCMC) sampling to find novel molecules satisfying several properties. However, most approaches require notoriously large number In contrast, we tackle this problem in a
of oracle calls to evaluate molecules on the fly 33 61 sample-efficient manner.
GFlowNet. GFlowNets 6 7 aim to sample composite objects proportionally to a reward function, instead of maximizing it as usual in RI. [56). GFlowNets related the MCMC methods due to the same objective, while amortizing the high cost of sampling (mixing between modes) over training generative model. GFlowNets have made impressive progress in various applications, such as biological sequence design (28). discrete probabilistic modeling [63]. and Bayesian structure learning [15]. While the concept of conditional GFlowNet was originally discussed Bengio et al. M. we are the first to study and instantiate this concept for MOO, in parallel with Jain et al. [29]. Compared with them, we delicately design the conditioning mechanism and propose a hindsight- like off off-policy strategy that is rarely studied for MOO (in both the RL and GFlowNet literature). Appendix D provides detailed comparison.
Bayesian optimization for discrete spaces. The application of BO in discrete domains has pro- liferated in recent years. much more challenging to construct surrogate models and optimize acquisition functions in discrete spaces, compared to continuous spaces. One common approach is to convert discrete space into continuous space with generative models [4 16 Besides, one
can directly define a Gaussian Process (GP) with discrete kernels 46 53
and solve the acquisition
function optimization problem with evolutionary algorithms 35
57
Multi-objective Bayesian optimization. BO has been widely used in MOO problems for efficiently optimizing multiple competing expensive black-box functions. Most popular approaches are based
on hypervolume improvement [12]. random scalarizations [37 49). and entropy search (27
5
While there have been several approaches that take into account parallel evaluations [1 38 and
diversity (38), they are limited to continuous domains.
3

---

3 Background
3.1 Problem formulation
We address the problem of searching over discrete chemical space X' to find molecular graphs X that maximize a vector-valued objective f(x (z) f2(),..... fM
X + RM,, where Jm black-box function (also known as the oracle) evaluating certain property of molecules. Practically, realistic oracles are extremely expensive to evaluate with either high-fidelity simulations or wet-lab experiments. We thus propose to perform optimization in as few oracle evaluations as possible, since the sample efficiency is paramount in such scenario.
There is typically single optimal solution to the MOO problem, different objectives may contradict each other. Consequently, the goal is to recover the Pareto front - the set of Pareto optimal solutions which cannot be improved in any one objective without deteriorating another 18 45). In
the context of maximization, a solution f(x) is said to Pareto dominates another solution f(x') iff
(2) 2 fm (x') Vm = I...., M and 3m' such that fm'(x) f (z'), and denote f(x) > f(x'). A solution f(x*) is Pareto optimal if not Pareto dominated by any solution. The Pareto front
can be written as p {f(x*) {s(x)= (xx) > f(x*) } 0}.
The quality of a finite approximate Pareto front P is commonly measured by hypervolume 667 the M-dim Lebesgue measure AM of the space dominated by P and bounded from below by given reference point r E MR; (V(P,)) = Am (u [r [r, wil), where [r. yil denotes the hyper-rectangle bounded by r and y: (x;)).
3.2 Batch Bayesian optimization
Bayesian optimization (BO) [54] provides a model- -ased iterative framework for sample-efficient black-box optimization. Given an observed dataset D, BO relies on a Bayesian surrogate model M to estimate a posterior distribution over the true oracle evaluations. Equipped with the surrogate model, an acquisition function
- is induced to assign the utility values to candidate objects for deciding which to be evaluated next on the oracle. Compared with the costly oracle, the cheap-to-evaluar acquisition function can be efficiently optimized. We consider the scenario where the oracle is given an evaluation budget of N rounds with fixed batches of size b.
To be precise, we have access to a random initial dataset Dg {(>,,, b. ) where uf - (x.) is a true oracle evaluation. In each round i E f1...., N), the acquisition function is maximized to yield a batch of candidates B; = [z]} to be evaluated in parallel on the oracle v} = [(a)). The
observed dataset 1 is then augmented for the next round: D. = D:-1 U (z),,))))
Method
In this section, we present the proposed MOBO algorithm based hypernetwork-based GFlowNet (HN-GFN), shown in Figure [ Due to the space limitation, we present the detailed algorithm Appendix[A] Our key idea extend GFlowNets as the acquisition function optimizer for MOBO, with the objective of sampling diverse batch of candidates from the approximate Pareto front. To begin, we introduce GFlowNets in the context of molecule design, then describe how GFlowNet can be biased by prfference-onddiiooee hypernetwork to sample molecules according to various trade-offs between objectives. Next, we propose a hindsight-like off-policy strategy to speed up learning in HN-GFN. Finally, we introduce the surrogate model and acquisition function.
4.1 Preliminaries
GFlowNets [6] seek to learn stochastic policy for sequentially constructing discrete objects X' with a probability (xx) x R(x), where X' is a compositional space and R: X R2o is a non-negative reward function. The generation process of object E X can be represented by sequence of discrete actions a A that incrementally modify a partially constructed object, which denoted as state E S. Let the generation process begin at special initial state 8o and terminate with special action indicating that the object is complete (s = x E X)), the construction of an object Sn x).
can be defined as complete trajectory (so + 31

---

Following fragment-based molecule design 66 61], the molecular graphs are generated by sequentially attaching a fragment, which chosen from a predefined vocabulary of building blocks, to an atom of the partially constructed molecules. The maximum trajectory length is 8, with the number of actions varying between 100 and 2000 depending on the state, making | up to 10t6, There are multiple action sequences leading to the same state, and no fragment removal actions, the space of possible action sequences can thus be denoted by a directed acyclic graph (DAG) g = (S, E), where the edges s' from one state to another. To learn the aforementioned desired policy, Bengio et al. [6] propose to see the DAG structure as a flow network.
in & are transitions
Markovian flows. Bengio et al. 7 first define a trajectory flow : R20 the set of all complete trajectories T to measure the unnormalized density. The edge flow and state flow can then be defined as Fs 5') Es-s'er F(T) and F(s) = Eeer F(),, respectively. The trajectory flow determines a probability measure P(T) Σ.er F(F) flow is Markovian, the forward
F(T)
transition probabilities PF can be computed as Ps(s'(s) = F(a)
F(x->))
Flow matching.
A flow is consistent the following flow consistency equation is satisfied Vss E S:
F(s)
Σ
F(s' - s) R(s) +
Σ
F(s + s")
(I)
s'EPag(s)
s"ssPPagss)) (s")
where Pag(s) is a set of parents of in G. As proved in Bengio et al. (6. if the flow consistency
equation is satisfied with R(s) for non-terminal state s and F(x) = R(x)) 2 0 for terminal state
r, a policy π defined by the forward transition probability T(s'(s) = (r(s'ss) (sssss) samples object æ with
a probability x (zz) x R(x:). GFlowNets propose to approximate the edge flow F(s - +') using a neural network (o s') with enough capacity, such that the flow consistency equation is respected at convergence. To achieve this, Bengio et al. [6] define a temporal difference-like [56] learning objective, called flow-matching (FM):
EvePog(s) Fo('5,) Fo 8"))
2
Co(s, log R(s) +
(2)
One can any exploratory policy से with full support sample training trajectories and obtain the consistent flow Fo s') by minimizing the FM objective [6] Consequently, a policy defined by this approximate flow ( {ss) Pr, (8'1s) Fo(s-ts')
Fe(e) can also sample objects with a probability
Tx(x) proportionally to reward R(x). Practically, the training trajectories are sampled from a mixture between the current policy PF. and uniform distribution over allowed actions (6). We adopt the FM objective in this work because the alternative trajectory balance 433 was also examined but gave a worse performance in early experiments. Note that more advanced objectives such subtrajectory balance 42 can be employed in future work.
4.2 Hypernetwork- -based GFlowNets
Our proposed HN-GFN aims at sampling a diverse batch of candidates from the approximate Pareto front with a unified model. A common approach to MOO is to decompose it into a set of scalar optimization problems with scalarization functions and apply standard single-objective optimization methods to gradually approximate the Pareto front [3).
64). Here we consider the weighted sum (WS); (x Σ, (x) and Tehebycheff 45 a (z) max; A; (x), where X = (X,, .AM) is preference vector that defines the trade-off between the competing properties.
To support parallel evaluations, one can simultaneously obtain candidates according to different scalarizations. Practically, this approach hardly scales efficiently with the number of objectives for discrete problems. Taking GFlowNet as an example, we need to train multiple GFlowNets separately for each choice of the reward function RA (a) = (x)) to cover the objective space:
0x arg min Co Rx)
0
Our key motivation is to design a unified GFlowNet to sample candidates according to different reward functions, even ones not seen during training. Instead of defining the reward function with
5

---

fixed preference vector A, we propose to train preference-conditione GFlowNet on the distribution
of reward functions Rx. where À sampled from a simplex SM = {(> E FT
Σ d =
1, 2 0}:
l* arg min ExESa EsesLo(s, Rx)
4
9
Remarks. Assuming an infinite model capacity, it is easy to prove that the proposed optimization scheme (Equation (4) as powerful as the original one (Equation (3), since the solutions both loss functions coincide [77] Nevertheless, the assumption of infinite capacity is extremely strict and hardly holds, so how to design the conditioning mechanism in practice becomes crucial.
4.2.1 Hypernetwork--asser conditioning mechanism
We propose to condition the GFlowNets on the preference vectors via hypernetworks [26]. Hyper- networks are networks that generate the weights of target network based on inputs. In vanilla GFlowNets, the flow predictor Fg is parameterized with the message passing neural network (MPNN) [23] over the graph of molecular fragments, with two prediction heads approximating F(s.s) and F(s) based the node and graph representations, respectively. These two heads are parameterized with multi-layer perceptrons (MLPs). For brevity, we write 0 = (Ompan: (pred.
One can view the training of HN-GFN as learning an agent to carry out multiple policies that correspond to different goals (reward functions Rx) defined in the same environment (state space S and action space A). We thus propose only to condition the weights of prediction heads pred with hypernetworks, while sharing the weights of MPNN Ompen. leading to more generalizable state represenniie More precisely. the hypernetwork l(;: Φ) takes the preference vector À as inputs and outputs the weights 0pred = (A;; Φ) of prediction heads. The parameters e of the hypernetwork are optimized like normal parameters. Following Navon al. [47]. we parametrize h using an MLP with multiple heads, each generating weights for different layers of the target network.
4.2.2 As the acquisition function optimizer
Training. At each iteration, first sample preference vector À from a Dirichlet distribution Dir(a). Then the HN-GFN is trained with the reward function RA (2) = al (=)), (z);; where μ and are the posterior mean and standard deviation estimated by M. In principle, retrain HN-GFN after every round. We also tried only initializing the parameters of the hypernetwork and found it brings similar performance and is more efficient.
Sampling, At each round i, we use the trained HN-GFN to sample a diverse batch of b candidates. Let A' be the set of k target preference vectors Xunger We sample 4 molecules for each Munget E A and evaluate them on the oracle in parallel. We simply construct A' by sampling from Dir(a), but it worth noting that this prior distribution can also be defined adaptively based on the trade-off of interest. As the number of objectives increases, we choose a larger l to cover the objective space.
4.3 Hindsight-like off- off-policy strategy
Resorting to the conditioning mechanism, HN-GFN can learn family of policies achieve various goals, i.e., one can treat sampling high-reward molecules for particular preference vector as separate goal. As verified empirically in Jain al. (28), since the FM objective is off-policy and offline, we can use offline trajectories to train the target policy for better exploration, so long as the a assumption of full support holds. Our key insight is that each policy can learn from the valuable apperrience (high-reward molecules) of other similar policies.
Inspired by hindsight experience replay 2 in RL, we propose to share ligh-performing molecules among policies by re-examining them with different preference vectors. As there are infinite possible preference vectors, we focus on target preference vectors A', which are based on to sample candidates, and build a replay buffer for each Mingeet E As. After sampling some trajectories during training, we store in the replay buffers the complete objects x with the corresponding reward R (z).
complete
To be specific, at each iteration, we first sample preference vector from a mixture between Dir(a) and a uniform distribution over A (1 - Dir(a)) + Uniform. If A is chosen, we construct half D
of the training batch with offline trajectories from the corresponding replay buffer of molecules woaeeeetttetrrrrr

---

encountered with the highest rewards. Otherwise, we incorporate offline trajectories from the current observed dataset D; instead to ensure that HN-GFN samples correctly the vicinity of the observed Pareto set. Our strategy allows for flexible trade-offs between generalization and specialization. As we vary Y from 0 to I, the training distribution of preference vectors moves from Dir(a) to A'. Exclusively training the HN-GFN with the finite target preference vectors A can lead to poor generalization. In practice, although we only sample candidates based on A', we argue that it is vital to keep the generalization to leverage the trained HN-GFN to explore various preference vectors adaptively. The detailed algorithm can be found in Appendix
4.4 Surrogate model and acquisition function
While GPs are well-established in continuous spaces, they scale poorly with the number of observa- tions and do not perform well in discrete spaces [57]. There has been significant work in efficiently
training non- -an-ssa neural networks estimate epistemic uncertainty [2] 39]. We benchmark
widely used approaches (in Appendix C.3). and use a flexible and effective one: evidential deep learning M. As for the acquisition function, the uncertainty. To be precise, the objectives are modeled with single multi-task MPNN, and the acquisition function is applied to the scalarization.
Upper Confidence Bound [55] to incorporate
Experiments
We first verify that HN-GFN has adequate capacity to generalize over preference vectors in single- round synthetic scenario. Next, we evaluate the effectiveness of the proposed MOBO algorithm based on HN-GFN in multi-round practical scenarios, which are more in line with real-worl world molecular optimization, Besides, we conduct several ablation studies to empirically justify the design choices.
5.1 Single-round synthetic scenario
Here, our goal is to demonstrate that we can leverage the HN-GFN to sample molecules with preference-conditionn
property distributions. The HN-GFN is used as a stand-alone optimizer outside of MOBO to directly optimize the scalarizations of oracle scores. As the oracle cannot be called as many times as necessary practically, we refer to this scenario as a synthetic scenario. To better visualize the trend of the property distribution of the sampled molecules as function of the preference weight, we only consider two objectives: inhibition scores against glycogen synthase kinase-3 beta (GNK333 and c-Jun N-terminal kinase-3 (JNK3)
Compared methods. We compare HN-GFN against the following methods. Preference-speciff GFlowNets (PS-GFN) set of vanilla unconditional GFlowNets, each trained separately for a specific preference vector. Note that PS-GFN is treated as "gold standard" rather than baseline, as it is trained and evaluated using the same preference vector. Concat-GFN and FiLM-GFN are two alternative conditional variations of GelowNet based the concatenation and FiLM [5]), respectively. MOEA/D (64) and NSGA-III 14 are two multi-objective evolutionary algorithms that also incorporate preference information. We perform evolutionary algorithms over the 32-dim latent space learned by Hier [32), which gives better optimization performance than JT-VAE (II.
Metrics. All the above methods are evaluated over the same set of 5 evenly spaced preference vectors. For each GFlowNet- based method, we sample 1000 molecules per preference vector as solutions. We compare the aforementioned methods on the following metrics: Hypervolume indicator (HV) measures the volume of the space dominated by the Pareto front of the solutions and bounded from below by the preference point 0). Diversity (Div) is the average pairwise Tanimoto distance over Morgan fingerprints. Correlation (Cor) is the Spearman's 's rank correlation coefficient between the probability of sampling molecules from an external test set under the GFlowNet and their respective rewards in the logarithmic domain [48). See more details in Appendix B.2. In nutshell, HV and Div measure the quality of the solutions, while Cor measures how well the trained model is aligned with the given preference vector. Each experiment is repeated with 3 random seeds.
Experimental results. As shown in Table HN-GFN outperforms the baselines and achieves competitive performance to PS-GFN (gold standard) on all the metrics. Compared to balse
7

---

Table 1: Evaluation of different methods on the synthetic scenario.
Method
HV (t)
Div (t)
Cor (?)
MOEA/D
0.182 + 0.045
n/a
n/a
NSGA-III
0.364 + 0.041
n/a
n/a 0.653 0.003
PS-GFN
0.545 + 0.055
0.786 = 0.013
Concat GFN
0.534 + 0.069
0.786 + 0.004
0.646 + 0.008
FiLM-GFN
0.431 + 0.045
0.795 = 0.014
0.633 0.009
HN-GFN
0,550 + 0.074
0.797 = 0.015
0.666 0.010
0.66
0.50
Avarage Top-K Reward
To-100 JNK3 sconds
0.64
0.45
0.40
0.62
0.30
0.50
0.25
Preference weight for JNK3
Taining Steps
Figure 2: Left: The distribution of Top-100 JNK3 scores. Right: The progression of the average Top-20 rewards over the course of training of the HN-GFN in optimizing GSK33 and JNK3.
methods, evolutionary algorithms (MOEA/D and NSGA-III) fail to find high-scoring molecules, especially the MOEA/D. HN-GFN outperforms Concat-GFN and FiLM-GFNN, and is the only conditional variant that can match the performance of PS-GFN, implying the superiority the well- designed hypernetwork-based conditioning mechanism. In Figure (left), we visualize the empirical property (JNK3) distribution the molecules sampled by HN-GFN and PS-GFN conditioned on the set of evaluation preference vectors. We observe that the distributions are similar and the trends are consistent: the larger the preference weight, the higher the average score. The comparable performance and consistent sampling distributions illustrate that HN-GFN has adequate capacity to generalize over preference vectors. Since the runtime and storage space of PS PS-GFN scale linearly with the number of preference vectors, unified HN-GFN provides significantly efficient way explore various trade-offs between objectives.
5.2 Multi-objective Bayesian optimization
Next, we evaluate the effectiveness of HN-GFN as an acquisition function optimizer within MOBO practical scenarios. We consider the following objective combinations of varying sizes:
GNK33+JNK33 Jointly inhibiting Alzheimer-related targets GNK33 and JNK3.
GNK33+-NK3-QED+SA- Jointly inhibiting GNK33 and JNK3 while being drug-like and easy-to-synthesize.
We rescale the SA score (initially between 10 and 1) so that all the above properties have range of [0.1] and higher is better. For both combinations, we consider starting with D 200 random molecules and further querying the oracle N = 8 rounds with batch size h 100. Each experiment repeated with random seeds.
Baselines. We compare HN-GFN with three representative LSOO methods (gParEGO 3 gEHVI [2] and LaMOO 665). well variety of state-of-the-art combinatorial optimization

---

Reperduume
0.4
0.25
0.20
0.3
6.15
0.2
0.30
200
300
400
500
600
700
8:0
500
1000
200
300
000
500
600
700
800
900 1000
lotal Orace Evaluasions
1stal Orace Evaluasions
(a) GSK33 JNK3
(b) GSK38 + JNK3 + QED + SA
Figure 3: Optimization performance (hypervolume) over MOBO loops.
Table 2: Diversity for different methods in MOBO scenarios.
Div (t)
GSK33 + JNK3
GSK33 3 JNK3 + QED SA
Graph GA
0.347 # 0.059 0,653 # 0,072
0.562 0.031
MARS
0.754 = 0.027
P-MOCO
0.646 + 0.008
0.350 0.130
HN-GFN
0.810 = 0.003
0.744 0.008
HN-GFN w/ hindsight
0.793 + 0.007
0.738 0.009
MCMC sampling approach, and P-MOCO 41 is multi-objective RL method. We provide more details in Appendix B.2
methods: Graph GA 30 is a genetic algorithm, MARS 61
Experimental results. Figure illustrates that HN-GFN achieves significantly superior perfor- mance (HV) to baselines, especially when trained with the proposed hindsight-like off-policy strategy. HN-GFN outperforms the best baselines P-MOCO and Graph GA of the two objective combinations by large margin (0.67 vs. 0.50 and 0.42 vs. 0.34), respectively. Besides, HN-GFN is sample- efficient, matching the performance of baselines with only half the number of oracle evaluations. All combinatorial optimization methods result in more performance gains compared to the LSO methods, implying that is promising to optimize directly over the discrete space. In Table 2 we report the Div (computed among the batch of 100 candidates and averaged over rounds). For fair comparison, we omit the LSO methods as they only support 160 rounds with batch size due to memory constraints. Compared with Graph GA and P-MOCO, which sometimes propose quite similar candidates, the superior optimization performance of HN-GFN can be attributed to the ability sample diverse batch of candidates. Another interesting observation, in the more challenging setting (GNK3 3+JNK3+QED+SA) is that MARS generates diverse candidates via MCMC sampling but fails find high-quality Pareto front, indicating that HN-GFN can find high-reward modes better than MARS. The computational costs are discussed in Appendix B.4
5.3 Ablations
Effect of the hindsight-like off-policy strategy. In the first round of MOBO, for each Auaget we sample 100 molecules every 500 training steps and compute the average Top-20 reward over A. In Figure (right), we find that the hindsight-like off-policy strategy significantly boosts average rewards, demonstrating that sharing high-performing molecules among policies effectively speeds up training of HN-GFN. On the other hand, further increasing 7 leads slight improvement. Hence, choose 0.2 for the desired trade-off between generalization and specialization.

---

Effect of Dir(a). Here we consider the more challenging GN combination, where the difficulty of optimization varies widely for various properties. Table shows that the distribution skewed toward harder properties results in better optimization performance. In our early experiments, we found that if the distribution of A is fixed, HN-GFN is quite robust to changes in a.
Effect of scalarization functions 8A- In Table we observe that WS leads to better Pareto front than Tchebycheff. Although the simple WS is generally inappropriate when a non-convex Pareto front is encountered [TO]. we find that is effective when optimized with HN-GFN, which can sample diverse high-revs bigh-reward candidates and may reach the non-convex part of the Pareto front. In addition, we conjecture that the non-smooth reward landscapes induced by Tchebycheff are harder to optimize.
Table 3: Ablation study of the and scalarization functions on GNK3AHING++EE+SSA
a
scalarization function
(1.1,II)
13.3.1.1)
(3,4.2,1)
WS
Tchebycheff
HV
0.039
0.039
0.385 + 0.018
0.416 0.023
0.416 0.023
0.304 = 0.075
Div
0.815 0.015
0.758 # 0.018
0.738 0.009
0.738 0.009
0.732 0.014
Conclusion
We have introduced a MOBO algorithm for sample-efficient multi-objective molecular optimization. This algorithm leverages a hypernetwork- GFlowNet (HN-GFN) to sample diverse batch of candidates from the approximate Pareto front. In addition, we present a hindsight-like off-policy strategy to improve optimization performance. Our algorithm outperforms existing approaches in synthetic and practical scenarios. Future work includes extending this algorithm to other discrete optimization problems such as biological sequence design and neural architecture search. One limitation of the proposed HN-GFN is the higher computational costs than other training-free optimization methods. However, the costs resulting from model training are negligible compared to the costs of evaluating the candidates in real-world applications. We argue that the higher quality of the candidates is much more essential than lower costs.
Acknowledgments and Disclosure of Funding
This research was partially supported by National Key R&D Program of China under grant No.201 12201AAAA112210 102. National Natural Science Foundation of China under grants No.62176231, No.62106218 No.82202984. No. No.92259202, No.62132017 and U22B2034, Zhejiang Key R&D Program of China under grant No. 202330033 Tingjun Hou's 's research was supported in part by National Natural Science Foundation of China under grant N2.22220102001.
References
[1] Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. Advances in Neural Information Processing Systems, 33:14927-14937, 2020.
[2] Marcin Andrychowicz. Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong. Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[3] Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In International Conference on Learning Representations, 2020. URL htttps://openreete
net/forum? Hxbkkkr
[4] Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wil- son, and Eytan Bakshy. Botorch: a framework for efficient monte-carlo Bayesian optimization. Advances in neural information processing systems, 14:21524-21533 2020.
10

---

[5] Syrine Belakaria, Aryan Deshwal, and Janardhan Rao Doppa. multi-objective Bayesian optimization. Advances in Neural Information Processing Systems, 32, 2019.
entropy search for
[6] Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381-22 -223944,,,, 2021.
[7] Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J Hu, Mo Tiwari, and Emmanuel Bengio.
Gflownet foundations. Journal of Machine Learning Research, 24(210):1-55, 2023.
[8] G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L. Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90-98, 2012.
[9] Julian Blank and Kalyanmoy Deb. Pymoo: Multi-objective optimization in python. IEEE Access, 8.89497-89509, 2020.
[10] Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[11] Eric Bradford, Artur M Schweidtmann, and Alexei Lapkin. Efficient multiobjective optimization employing Gaussian processes, spectral sampling and genetic algorithm. Journal of global optimization, 71(2):407-433 2018.
[12] Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hypervol- ume improvement for parallel multi-objective Bayesian optimization. Advances in Neural Information Processing Systems, 33:9851-9864 2020.
[13] Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Parallel Bayesian optimization of multiple noisy objectives with expected hypervolume improvement. Advances in Neural Information Processing Systems, 34:2187-2200, 2021.
[14] Kalyanmoy Deb and Himanshu Jain, An evolutionary many-objective optimization algorithm using reference-point-boee based nondominated sorting approach, part i: solving problems with box constraints. IEEE transactions on evolutionary computation, 18(4):577-601, 2013.
[15] Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julies Stefan Bauer, and Yoshua Bengio. Bayesian Structure Learning with Generative Flow Networks. arXiv preprint, 2022.
[16] Aryan Deshwal and Jana Doppa. Combining latent space and structured kernels for Bayesian optimization over combinatorial spaces. Advances in Neural Information Processing Systems, 34:8185-822 8200, 2021.
[17] Alexey Dosovitskiy and Josip Djolonga. You only train once: Loss-conditional training of deep networks. In International conference on learning representations, 2019.
[18] Matthias Ehrgott. Multicriteria optimization, volume 491. Springer Science & Business Media, 2005.
[19] Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug- molecules based on molecular complexity and fragment contributions. Journal of cheminfor- matics, 1(1):1-11, 2009.
[20] Tianfan Fu, Wenhao Gao, Cao Xiao, Jacob Yasonik, Connor W Coley, and Jimeng Sun. Differentiable scaffolding tree for molecule optimization. In International Conference on Learning Representations, 2022.
[21] Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059. PMLR, 2016.
[22] Wenhao Gao, Tianfan Fu, Jimeng Sun, and Connor Coley. Sample efficiency matters: benchmark for practical molecular optimization. Advances in Neural Information Processing Systems, 35:21342-213577 2022.

---

[23] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. International conference on machine learning. pages 1263-1272. PMLR, 2017.
[24] Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparragunn Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven continuous representation of molecules. ACS central science, 4(2):268-21) 2018.
[25] Javier González, Zhenwen Dai, Philipp Hennig, and Neil Lawrence. Batch bayesian optimization via local penalization. In Artificial intelligence and statistics, pages 648 657. PMLR, 2016.
[26] David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint 160.00/000009000 2016.
[27] Daniel Hemáandez-Lobato, Jose Hermandez-Lobato, Amar Shah, and Ryan Adams. Predictive entropy search for multi-objective Bayesian optimization. In International conference on machine learning, pages 1492-1501. PMLR, 2016.
[28] Moksh Jain, Emmanuel Bengio, Alex Sramandez-Gariian Jarrid Beotor-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning. pages 1786-9801. PMLR, 2022.
[29] Moksh Jain, Sharath Chandra Raparthy, Alex Hernandez-Garcia. Jarrid Rector-Brooks, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. arXiv preprint arXiv:2210. 12765, 2022.
[30] Jan H Jensen. A graph- based genetic algorithm and generative model/monte carlo tree search for the exploration of chemical space. Chemical science, 10(12):3567-335 -372, 2019.
[31] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning, pages 2323-2332. PMLR, 2018
[32] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In International conference on machine learming, pages 4839- 4848. PMLR, 2020.
[33] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multi-objective molecule generation using interpretable substructures. In International conference on machine learning, pages 4849-4859, PMLR, 2020.
[34] Donald R Jones, Matthias Schonlau, and William Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455-492, 1998.
[35] Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Neural architecture search with Bayesian optimisation and optimal transport. Advances in neural information processing systems, 31, 2018.
[36] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.61 14, 2013.
[37] Joshua Knowles. Parego: A hybrid algorithm with on-line landscape approximation for expen- sive multiobjective optimization problems. IEEE Transactions on Evolutionary Computation, 10(1):50-66, 2006.
[38] Mina Konakovic Lukovic, Yunsheng Tian, and Wojeiech Matusik. Diversity-guidee multi- objective Bayesian optimization with batch evaluations. Advances in Neural Information Processing Systems, 33:17708-17720, 2020.
[39] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances neural information processing systems, 30, 2017.
12

---

[40] Yibo Li, Liangren Zhang, and Zhenming Liu. Multi-objective de novo drug design with conditional graph generative model. Journal of cheminformatics, 10(1):1-24, 2018.
[41] Xi Lin, Zhiyuan Yang, and Qingfu Zhang. Pareto set learning for neural multi-objective combinatorial optimization. In International Conference on Learning Representations, 2021.
[42] Kanika Madan, Jarrid Rector- Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, Andrei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability. In International Conference Machine Learning, pages 23467-23483 PMLR, 2023.
[43] Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. arXiv preprint arXiv:2201. 13X:52201.13259, 2022.
[44] Natalie Maus, Haydn T Jones, Juston S Moore, Matt J Kusner, John Bradshaw, and Jacob R Gardner. Local latent space Bayesian optimization over structured inputs. arXiv preprint 1181 2022.
[45] Kaisa Miettinen. Nonlinear multiobjectivv optimization, volume 12. Springer Science & Business Media, 2012.
[46] Henry Moss, David Leslie. Daniel Beck, Javier Gonzalez, and Paul Rayson. Boss: Bayesian optimization over string spaces. Advances in neural information processing systems, 33: 15476-15486, 2020.
[47] Aviv Navon, Aviv Shamsian, Ethan Fetaya, and Gal Chechik. Learning the pareto front with hypernetworks. In ICLR, 2021.
[48] Andrei Cristian Nica, Moksh Jain, Emmanuel Bengio, Cheng-Hao Liu, Maksym Korablyov, Michael M Bronstein, and Yoshua Bengio. Evaluating generalization in gflownets for molecule design. In ICLR2022 Machine Learning for Drug Discovery, 2022.
[49] Biswajit Paria, Kirthevasan Kandasamy, and Barnabás Póczos. A flexible framework for multi-objective Bayesian optimization using random scalarizations. In Uncertainty in Artificial Intelligence, pages 766-776. PMLR, 2020.
[50] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, nigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[51] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI Conference on Artificial Intelligence 2018.
[52] Pavel G Polishchuk, Timur I Madzhidoy, and Alexandre Varnek. Estimation of the size of drug-like chemical space based gdb-17 data. Journal of computer- aided molecular design, 27(8):675-66 679, 2013.
[53] Binxin Ru, Xingchen Wan, Xiaowen Dong, and Michael Osborne. Interpretable neural ar- chitecture search via bayesian optimisation with weisfeiler-lehman kemelss. arXiv preprint arXiv:2006.07556 2020.
[54] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104 (1):148-175, 2015.
[55] Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: no regret and experimental design. Proceedings of the 27th International Conference on International Conference on Machine Learning, pages 1015-1022, 2010.
[56] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
13

---

[57] Kevin Swersky, Yulia Rubanova, David Dohan, and Kevin Murphy. Amortized Bayesian optimization over discrete spaces. Conference on Uncertainty in Artificcal Intelligence, pages 769-778. PMLR, 2020.
[58] Austin Tripp. Erik Daxberger, and José Miguel Hernáneez-Lobato Sample-efficient optimiza- tion in the latent space deep generative models via weighted retraining. Advances Neural Information Processing Systems, 33: 1259-11277 2020.
[59] Austin Tripp, Gregor NC Simm, and José Miguel Hernández-Lobato A fresh look at de novo molecular design benchmarks. In NeurlPS 2021 Al Science Workshop, 2021.
[60] Robin Winter, Floriane Montanari, Andreas Steffen, Hans Briem, Frank Noé, and Djork-Arn Clevert. Efficient multi-objective molecular optimization in continuous latent space. Chemical science, 10(34):8016 8024, 2019.
[61] Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. Mars: Markov molecular sampling for multi-objective drug discovery. In International Conference on Learning Representations, 2021.
[62] Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. Advances in neural information processing systems, 31, 2018.
[63] Dinghuai Zhang, Nikolay Malkin, Zhen Liu, Alexandra Volokhova, Aaron Courville, and Yoshua Bengio. Generative flow networks for discrete probabilistic modeling. arXiv preprint arXiv:2202.01361. 2022.
[64] Qingfu Zhang and Hui Li. Moeald: multiobjective evolutionary algorithm based on decom- position. IEEE Transactions evolutionary compulation, 1 11(6):11--731 2007.
[65] Yiyang Zhao, Linnan Wang. Kevin Yang, Tianjun Zhang, Tian Guo, and Yuandong Tian. Multi- objective optimization by learning space partition. In International Conference on Learning Representations, 2022. URL https:///ppenrevee:
[66] Yiheng Zhu, Zhenqiu Ouyang, Ben Liao, Jialu Wu, Yixuan Wu, Chang- Yu Hsieh, Tingjun Hou, and Jian Wu. Molhf: A hierarchical normalizing flow for molecular graph generation. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 5002-5010, 2023.
[67] Eckart Zitzler and Lothar Thiele. Multiobjective evolutionary algorithms: comparative case study and the strength pareto approach. IEEE transactions Evolutionary Computation, 3(4): 257-271, 1999.
14

---

A Algorithms
Algorithm| [ describes the overall framework of the proposed MOBO algorithm, where HN-GFN leveraged as the acquisition function optimizer. Algorithm describes the training procedure for HN-GFN within MOBO.
Algorithm 1 MOBO based on HN-GFN
Input: oracle f = (f1..... fM). initial dataset Do = f fx
acquisition function a,
parameter of Dirichlet distribution a, number of rounds N, batch size b Initialization: surrogate model M. parameters of HN-GFN Th for i to N do
Fit surrogate model .M on dataset D
Sample the set of target preference weights A Dir(a) Train Ty with reward function (x) = (x)())xx)) (81(2))); Sample query batch B, = { based on Atarget E A
D Algorithm
Evaluate batch B; with f and augment the dataset Di+1 D; U (()))
end for
Algorithm 2 Training procedure for HN-GFN with the hindsight-like off-policy strategy
Input: available dataset D, reward function R, minibatch size m, set of target preference vectors A. proportion of hindsight-like strategy % replay buffers {Rx) IeE while not converged do Flag Bernoulli(y) if Flag = then
X ~ A
Sample m trajectories from replay buffer RA
else
A ~ Dir(a)
Sample m trajectories from the available dataset D;
end if
mmpan o))
Sample m trajectories from policy गे and store terminal states x in R, for all X E A 2
Compute reward Rx (1) on terminal states x from each trajectory the minibatch
Update parameters Ompon and ái with a stochastic gradient descent step w.r. Equation end while
B Implementation details
B.1
Experimental settings
B.1.1 Molecule domain
Following Bengio al. (6), the molecules are generated by sequentially attaching a fragment, which is chosen from predefined vocabulary of building blocks, to an atom of the partially constructed molecules. The maximum trajectory length and 2000 depending the state, making X up to 10 6 We adopt the property prediction models released by Xie et al. [61 to evaluate the inhibition ability of generated molecules against GSK3,3 and JNK3.
with the number actions varying between 100
B.1.2 Metrics
Diversity. Diversity (Div) is the average pairwise Tanimoto distance over Morgan fingerprints. In the synthetic scenario, for each preference vector, we sample 1000 molecules, calculate the Div among the Top-100 molecules, and report the averages over preferences. computed among the batch of 100 candidates round, as Graph GA and MARS are not preference- conditioned. And we believe this metric possibly more aligned with how these methods might be used in biology chemistry experiment.
MOBO, the DiV
15

---

Correlation. Correlation (Cor) is the Spearman's 's rank correlation coefficient between the probabil- ity of sampling molecules from an external
under the GFlowNet and their respective rewards in the logarithmic domain: Cor Spearman's (log(at())) ())) The external test set is obtained two steps: First, we generate random dataset containing 300K molecules uniformly based on the number of building blocks; Next, we sample the test sets with uniform property distribution
corresponding to GSK33 and JNK3, respectively, from the 300K molecules.
B.2 Baselines
All baselines are implemented using the publicly released source codes with adaptations for our
MOBO scenarios. The evolutionary algorithms (MOEA/D and NSGA-IIII implemented in PyMOO [9], and the LSO methods (gParEGOO gEHVI, and LaMOO) are implemented in BoTorch 4 In MOBO scenarios, LaMOO and Graph GA utilize EHVI as the acquisition function. For all GP- based methods, each objective is modeled by an independent GP.
B.3 HN-GFN
We implement the proposed HN-GFN in PyTorch [50). The values key illustrated in Table 4
are
Surrogate model: We use the 12-layer MPNN as the base architecture of the surrogate model in our experiments. In MOBO, a single multi-task MPNN is trained with batch size of 64 using the Adam optimizer with a dropout rate 0.1 and weight decay rate of le-6. We apply early stopping to improve generalization.
HN-GFN: HN-GFN contains vanilla GelowNet and preference-conditiones hypernetwork. The architecture GFlowNet is a 10-layer MPNN, and the hypernetwork is a 3-layer ML. with multiple heads, each generating weights for different layers of the target network. The HN-GFN is trained with Adam optimizer to optimize the Flow Matching objective.
Table 4: Hyper-paramette used in the real-world MOBO experiments.
Hyper-parameter
GSK33 + INK3
GSK33 + JNK3 + QED + SA
Surrogate model
Hidden size
64
64
Learning rate
2.5e-4
le-3
X for evidential regression
0.1
0.1
Number iterations
10000
10000
Early stop patience
500
500
Dropout
0.1
0.1
Weight decay
le-6
le-6
Acquisition function (UCB)
8
0.1
0.1
HN-GFN
Learning rate
5e-4
5e-4
Reward exponent
8
8
Reward norm
1.0
1.0
Trajectories minibatch size
8
8
Offline minibatch size
hindsight
0.2
0.2
Uniform policy coefficient
0.05
0.05
Hidden size for GelowNet
256
256
Hidden size for hypernetwork
100
100
Training steps
5000
5000
a
(1,1)
(3,4,2,1)
16

---

B.4 Empirical running time
The efficiency is compared on the same computing facilities using Tesla V100 GPU. In the context of MOBO, the running time of three LSO methods (i.e., gParEGO, gEHVI, and LaMOO) is around 3 hours, while Graph GA optimizes much faster and costs only 13 minutes. In contrast, the time complexity of hap-learning-basee discrete optimization methods is much larger. MARS costs 32 hours, while our proposed HN-GFN costs hours. With the hindsight-lite training strategy, the running time of HN-GFN will increase roughly by 33%.
However, if look at the problem in bigger picture, the time costs for model training are likely negligible compared to those of evaluating molecular candidates in real-world applications. Therefore, argue that the high quality of the candidates (the performance of the MOBO algorithm) is more essential than having a lower training cost.
C Additional results
C.1 Sampled molecules in MOBO experiments
We give some examples of sampled molecules from the Pareto front by HN-GFN in the GSK33 - JNK3 + QED + SA optimization setting (Figure 4 The numbers below each molecule refer to GSK33, JNK3, QED, and SA scores respectively.
HIN
0,98, 0. 34, 0.72, 0.83
0.79. 0.83, 0.59, 0.85
0.75, 0.42, 0.58, 0.94
0.64, 0.22, 0.83, 0.86
Figure 4: Sampled molecules from the approximate Pareto front by HN-GFN.
C.2 Synthetic scenario
As illustrated in Figure the distribution of Top-100 GSK33 scores shows a consistent trend in preference-specifie GFlowNet and our proposed HN-GFN, although the trend is not significant as the JNK3 property.
Petoranspeectt Chowie
HN-2F
0.0
0.5
0.50
0.55
0.1
Profoe
Figure 5: Comparison of the distribution of Top- 100 GSK3, scores sampled by different preference vectors using prefefence-speecific GelowNets and HN-GFN,

---

C.3 Effect of Surrogate models
We conduct ablation experiments to study the effectiveness of different surrogate models. We consider the following three surrogate models: evidential regression [I], Deep Ensembles [39], and GP based on the Tanimoto kernel [59]. As shown in Table we can observe that evidential regression leads to better optimization performance than Deep Ensembles. While the HV of evidential regression and GP is comparable, evidential regression can propose more diverse candidates. Furthermore, we argue that GP is less flexible over discrete spaces than evidential regression and Deep Ensembles, as different kernels need be designed according to the data structures.
Table 5: Evaluation of different surrogate models in MOBO scenarios
GSK38
JNK3
GSK33 + JNK3 HV
QED +
HV
Div
Div
HN-GFN (Evidential)
0.669 0.061
0.793 = 0,007
0.416 0,023
0.738 = 0,009
HN-GFN (Ensemble)
0.583 0.103
0.797 0.004
0.355 0.048
0.761 0.012
HN-GFN (GP)
0.662 0.054
0.739 0.008
0.421 0.037
0.683 0.018
D Comparison with parallel work
While the concept Pareto GelowNet was theoretically discussed Bengio et al. 77 we are among the first to study and instantiate this concept for MOO, and we address practical challenges that are not discussed thoroughly in the original theoretical exposition to potentially make sample-efficiee molecular optimization reality. We extensively study the impact of the conditioning mechanism (Section 5. ) and surrogate models (Appendix C.3p. Moreover, we delicately propose hindsight-like off-policy strategy (Section 4.3) which is rarely studied for MOO (in both the RL and GelowNet literature).
We note that there is parallel work introduced in Jain et al. 129 which shares the idea of using GFlowNets for MOO. Jain et al. [29], they propose two variants MOGFN-PC and MOGFN-AL for MOO in the single-round scenario and Bayesian optimization (BO) scenario, respectively. Indeed, when HN-GFN used as a stand-alone optimizer outside of MOBO (Section 5.I). it is similar to MOGFN-PC except for using different conditioning mechanisms. As for MOBO, MOGFN-AL a vanilla GFlowNet whose reward function defined as a multi-objective acquisition function (NEHVT [3).. In addition extending GelowNet, delicately propose hindsight-like off-policy strategy (Section 4.3 which rarely studied for MOO (in both the RL and GFlowNet literature).
18


Reviews:
[{"review_id": "yuWaD04R0w", "reviewer": "Reviewer_9YCN", "summary": "This work addresses molecular design with GFLOWNET, an algorithm learning a sampling policy $\\pi$ proportional to the reward function, i.e. where $\\pi(x) \\propto R(x)$. In particular, this work tackles the essential yet under-adressed multiobjective optimization setting. They do so using a hypernetwork (conditioned on a preference vector) providing weights for the sampler.", "questions": "1- A part of the protocole is unclear to me: are the generated molecules evaluated on the objective using an actual oracle or using the learned surrogate function ?\n\n2 - How are the objectives computed ? Is it “physics” based or does it rely on learning a surrogate model ? Then, how are the generated molecules evaluated ?\n\n3 - What is the length of the generated molecules ? Is it a hyperparameter of the model ?\n\n4 - How sensitive are the results to the original dataset  $D_0$?  In my personal experience, Gflownets tend to be rather unstable or difficult to train due to the sparse feedback (and depends on the sequence size) and 200 initial molecules seem low to be able to learn a robust policy.\n \n5 - Does the proposed Hindsight-like off policy strategy amounts to the proposition of [1], i.e. to hybrid the learning between generated molecules vs already generated molecules ?\n\n6 - What are the variable predicted by the MPNN ? \n\n7 -  Can the authors discuss their implementation choice regarding the gflow network. For instance [1] use a MLP as sampler. \n\n\n[1]: Jain et al, Biological sequence designs with Gflownets.", "limitations": "A limitation of the paper is the lack of diversity in the experimental settings. For instance, the paper Gflownet for biological sequence design presents optimization results for several benchmark datasets such as GFP. Can the authors comment on the ability of their method to adapt to such an ‘off-line’ setting (which is an essential in biological sequence design) ?", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In my opinion the presentation is clear and the motivation behind this work are nicely presented. In addition, the paper is well written and easy to follow. As a side note, its presentation of the GFN is interesting and worth the reminder to ease the understanding of the paper. \nThe approach is both original and sound, boosting its potential use in practical application. \nFinally, the presented results are conclusive and seem to validate the intuition behind the method.", "weaknesses": "The main weaknesses in the paper is in my opinion in the lack of diverse experiments or explanation about them. See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer for the insightful and constructive comments!\n\n> How are the objectives computed? Is it “physics” based or does it rely on learning a surrogate model? Then, how are the generated molecules evaluated? Are the generated molecules evaluated on the objective using an actual oracle or learned surrogate function? \n\nFor molecular optimization, the true objective values should be obtained by conducting expensive wet-lab experiments. However, real experimental evaluation makes the benchmarking of algorithms difficult and time-consuming. While physics-based simulation sounds rigorous, it consumes too many computing resources and may not be accurate enough to justify its extensive usage under many circumstances. Therefore, we follow the evaluation scheme adopted by prior related works (e.g., [1]) and use prediction models trained on an external dataset (not used for training HN-GFN) as our oracle functions.\n\nAt each round of BO, during the training of HN-GFN, the generated molecules (trajectories) are evaluated using the learned surrogate function. Then the trained HN-GFN is used to propose a diverse batch of candidates, which are evaluated using the oracle.\n\n> What is the length of the generated molecules? Is it a hyperparameter of the model?\n\nWe are sorry for our unclear description. The length is not a hyperparameter, we allow the agent to learn a special stop action and generate up to 8 fragments.\n\n> How sensitive are the results to the original dataset? In my personal experience, GFlowNets tend to be rather unstable or difficult to train due to the sparse feedback (and depends on the sequence size) and 200 initial molecules seem low to be able to learn a robust policy.\n\nWe argue that the robustness of the policy will not be affected by the number of initial molecules for the following two reasons. 1) The 200 initial molecules are mainly used for training the surrogate models which determine the reward. 2) The training of GFlowNets primarily relies on trajectories obtained through online sampling. Moreover, in BO, the initial molecules are commonly randomly sampled, and their evaluation is also included in the budget. Hence, initial molecules are relatively low, different from offline model-based optimization where the performance is greatly influenced by the quality of initial molecules. For a fair comparison, we use the same fixed set of random initial molecules. To confirm our point, we conducted experiments with different sets of random initial molecules and found that the results are not sensitive. \n\n| | |GSK3β + JNK3 + QED + SA | |\n|:-|:-|-:|-:|\n|  | Starting HV | Final HV | Improvement |\n| HN-GFN (fixed)   | 0.101 | 0.416 ± 0.023 | 0.315 |\n| HN-GFN (random)  | 0.082 ± 0.018 | 0.402 ± 0.018 | 0.320 |\n\n> Does the proposed Hindsight-like off-policy strategy amounts to the proposition of [1], i.e. to hybrid the learning between generated molecules vs already generated molecules?\n\nIndeed, [1] only incorporates the trajectories from the available observed dataset, rather than those generated during training. Inspired by the empirical observation that GFlowNets benefit from offline trajectories, we adopt the concept of replay buffer which is widely used in RL to store and share high-performing molecules among policies by re-examining them with different preferences. Our strategy is delicately proposed for MOO, and has not been actively explored (in both RL and GFlowNet literature) prior to our work. Furthermore, the proposition of [1] is a special case of our strategy (when $\\gamma=0$, the third paragraph in Sec 4.3). In Figure 2 (right), our strategy off-policy ($\\gamma>0$) consistently outperforms [1] off-policy ($\\gamma=0$).\n\n> What are the variables predicted by the MPNN?\n\nThere are two MPNNs in our algorithm. 1) The MPNN serving as the policy network takes as input the partially constructed molecular graphs and predicts edge flow $F(s,s')$ and state flow $F(s)$. Specifically, for each stem of the graph (an atom where the policy can attach a new block), MPNN predicts $F(s,s')$ representing the unnormalized probability of attaching each block to this stem. For the stop action, we perform a global pooling and predict $F(s)$. 2) The MPNN serving as the surrogate model takes as input the complete molecular graphs and predicts multiple objectives simultaneously.\n\n> Can the authors discuss the choice regarding the flow network? For instance [1] use a MLP as a sampler.\n\nFollowing [2,3], we used MPNN for a fair comparison. In general, MPNN performs well in representing molecules. Other GNNs fit our framework as well. MLP is used for a fair comparison too in [1]. In my personal experience, 1D CNN and Transformer are better for sequence design.\n\n> A limitation of the paper is the lack of diversity in the experimental settings. [1] presents results for several benchmark datasets such as GFP. Can the authors comment on the ability of their method to adapt to such an offline setting (which is essential in biological sequence design)?\n\nThanks for pointing this out. To our best knowledge, offline multi-objective benchmarks are rare. After reviewing related literature, we choose RFP from LaMBO [4] (optimize red-spectrum fluorescent protein sequences to maximize stability and SASA) and adapt the online setting to the offline setting. Following [1], we generate 128 candidates starting with the same $\\mathcal{D}$ = 512 examples as [4]. Our HN-GFN significantly outperforms LaMBO.\n\n| |Relative HV| \n| :- | -: | \n|LaMBO|1.086 ± 0.010|  \n|HN-GFN|**1.268 ± 0.089**| \n\n[1] Biological sequence designs with Gflownets.\n\n[2] Flow network based generative models for non-iterative diverse candidate generation.\n\n[3] Mars: Markov molecular sampling for multi-objective drug discovery.\n\n[4] Accelerating Bayesian optimization for biological sequence design with denoising autoencoders.\n\n**We hope our response can alleviate your concerns. Please let us know if you have any additional questions.**"}, {"review_id": "8ADkNHE7Nt", "reviewer": "Reviewer_BkDd", "summary": "This paper proposes the use of GFlotNets to address the problem of sample-efficient multi-objective molecular optimization, an important problem in various scientific discovery application - such as materials design and drug discovery.\nThe key idea proposed in this work is to leverage hypernetwork-based GFlowNets - referred to as HN-GFN - to optimize the acquisition function for multi-objective Bayesian optimization (MOBO).\nThe goal is to enable efficient sampling of a diverse high-quality batch of molecular candidates from an approximate Pareto front.\n", "questions": "Please see the questions and suggestions in the weaknesses section.\n", "limitations": "The conclusion section includes a very brief discussion of a limitation of the current method and suggests directions for future work.\nThe broader implication of the work is not explicitly discussed in the manuscript.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed hyper network-based GFlowNets as a MOBO acquisition function optimizer provides a novel and intuitive way of using GFlowNets for sampling novel molecular candidates, where a \"unified\" GFlowNet is trained that considers the distribution of different reward functions that correspond to different preference vectors instead of relying on a single GFlowNet for a fixed preference vector.\nThis may allow the resulting model, HN-GFN, to naturally explore the various trade-offs between multiple objectives that may compete with one another by adapting to varying the input preference vector.\n\nThe proposed method builds on the recently proposed and widely popular GFlowNets and extends its flexibility for multi-objective molecular optimization by incorporating a hypernetwork-based approach.\nAdditionally, the adoption of a hindsight-like off-policy strategy is proposed to improve the learning efficiency, and ultimately, the multi-objective molecular optimization performance.\n\nOverall, the paper is well-organized and written in a clear manner, the proposed method is well-motivated and novel, and the performance evaluation results demonstrate the potential advantages of the proposed HN-GFN.\n\n\n\n", "weaknesses": "Although the batch size may significantly affect the overall computational cost as well as the optimization performance, there is no discussion on the impact of selecting a specific batch size nor any empirical evaluation based on different batch sizes.\n\nWhile the evaluation results provide some preliminary evidence of the potential advantages of HN-GFN, the evaluations in the current study are limited to (virtually) a single problem: i.e., inhibition of GSK3β + JNK3 (with potential additional considerations for synthesizability and drug-likeness).\nAdditional examples are needed to more convincingly demonstrate the general applicability (and merits) of the proposed HN-GFN.\nI suggest providing further evaluation results based on other benchmark problems often used for evaluating other generative models (e.g., JT-VAE, HierVAE, etc.)\n\nIn Figure 2, only trends for JNK3 are shown, but it would be helpful to show the optimization trends for GNK3β as well for completeness.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer for the insightful and constructive comments!\n\n> Although the batch size may significantly affect the overall computational cost as well as the optimization performance, there is no discussion on the impact of selecting a specific batch size nor any empirical evaluation based on different batch sizes.\n\nThanks for pointing this out. We have conducted experiments to discuss the impact of different batch sizes. Just as expected, our HN-GFN benefits from lower batch size and more rounds. As GFlowNet amortizes the cost of optimization during training, the overall computational cost is proportional to the number of rounds. From the perspective of the trade-off between computational cost and optimization performance, we believe that 8*100 is suitable for the design and benchmarking of algorithms for molecular optimization.\n\n| # Round * BS | 16 * 50 | 8 * 100 | 4 * 200 | \n| :-----| ----: | ----: | ----: |\n| HV       | 0.442 ± 0.012 (0.026 $\\uparrow$)| 0.416 ± 0.023 | 0.373 ± 0.038 (0.043 $\\downarrow$)|\n| Run-time | 2 $\\times$    | 1 $\\times$    | 0.5 $\\times$  |\n\n> While the evaluation results provide some preliminary evidence of the potential advantages of HN-GFN, the evaluations in the current study are limited to (virtually) a single problem: i.e., inhibition of GSK3β + JNK3 (with potential additional considerations for synthesizability and drug-likeness). Additional examples are needed to more convincingly demonstrate the general applicability (and merits) of the proposed HN-GFN. I suggest providing further evaluation results based on other benchmark problems often used for evaluating other generative models (e.g., JT-VAE, HierVAE, etc.)\n\nThanks for pointing this out. GSK3β + JNK3 (with potential additional considerations for synthesizability and drug-likeness) is the most widely-used benchmark for multi-objective molecular optimization since proposed in [1]. For further diverse evaluation, after reviewing the related literature, we consider the following objective combinations: dopamine type 2 receptor (DRD2) + QED + SA [2] and soluble epoxide hydrolase (sEH) + QED + SA [3]. Our HN-GFN still achieves significantly superior performance than P-MOCO (the best baseline for GSK3β + JNK3).\n\n|  | DRD2 + QED + SA | | sEH + QED + SA | |\n| :-----| ----: | ----: | ----: | ----: |\n|  | HV | Div | HV | Div |\n| P-MOCO   | 0.710 ± 0.015 | 0.209 ± 0.056 | 0.649 ± 0.020 | 0.445 ± 0.005 |\n| HN-GFN   | **0.853 ± 0.001** | **0.735 ± 0.045** | **0.694 ± 0.010** | **0.828 ± 0.002** |\n\n> In Figure 2, only trends for JNK3 are shown, but it would be helpful to show the optimization trends for GNK3β as well for completeness.\n\nWe appreciate the reviewer's helpful reminder. We only show trends for JNK3 due to the space limitation. We will include the trends for GNK3β as well.\n\n[1] Multi-objective molecule generation using interpretable substructures.\n\n[2] Multi-constraint molecular generation based on conditional transformer, knowledge distillation and reinforcement learning.\n\n[3] Flow network based generative models for non-iterative diverse candidate generation.\n\n**We hope our response can alleviate your concerns. Please let us know if you have any additional questions.**"}, {"review_id": "mtzoI1L9TT", "reviewer": "Reviewer_hutP", "summary": "A GFlowNet for molecular optimization conditioned on the preference weights of multiple objectives is proposed. To be precise, the model is trained to sample molecules from a target space with probability proportional to some combination of reward functions (weighted sum or Chebyshev scalarization) and trained with varying weights of the rewards, with the GFlowNet conditioning achieved by a hypernetwork parametrization. The system is tested in both single-step and Bayesian optimization / active learning settings.", "questions": "Please see above.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The writing is generally clear, problem is introduced well and placed in context.\n- Good results on the tasks studied relative to the chosen baselines\n- Two interesting GFlowNet contributions not specific to molecules:\n  - Study of conditioning algorithms (FiLM and Concat variants), with the hypernetwork-based approach found to perform best.\n  - The use of a conditional replay buffer, which is, as far as I know, new in GFlowNet literature (a replay buffer was used with GFlowNets in [arXiv:2202.13903, UAI 2022] but in a vanilla RL manner).\n  - Both are illustrated well with ablation studies.\n- The code is a helpful addition that aids in understanding the details of what was done.", "weaknesses": "Meta-weakness: Due to my closely following the literature, I was aware of an earlier submission of this paper to ICLR 2023. The present submission is almost identical to ICLR version (post-rebuttal) and therefore cannot incorporate the feedback of the reviewers. That said, to avoid biasing myself here, I did not read the ICLR reviews and this assessment is entirely my own.\n\nComparison with prior work:\n- There is the paper \"Multi-objective GFlowNets\" [MO; arXiv:2210.12765, ICML 2023], which also studies GFlowNets for multiobjective Bayesian optimization by conditioning on scalarization weights. Can you comment on the differences in problem setting or method with that paper?\n- The attached code includes the implementations of two baselines, in `code/generator/gfn.py`, but they are not included in the paper:\n  - MOReinforce, which is a natural candidate for an RL-based baseline that can readily share the parametrization of the agent with the GFlowNet.\n  - GFlowNet with trajectory balance [TB; arXiv:2201.13259, NeurIPS 2022]. Trajectory balance is the objective used in nearly all work on GFlowNets for molecular optimization after [5], but for some reason it is not evaluated or mentioned in the text. In particular, it is evaluated on molecule synthesis in both [TB] and [MO].\n\nSmall issue in presentation/math: There are some inconsistencies in definitions related to GFlowNets. Equation (1) writes $R(s)$ for the reward received when terminating at $s$, as in [5]. However, the text immediately following it talks about terminal and nonterminal states (with nonterminals having reward 0), and the text above says there is a special termination action that leads to terminal states. This is more consistent with the convention from [TB], where terminal states have no children and are the only states with nonzero reward. This is not a major bug as the two conventions are equivalent, but it would be good to be careful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer for the insightful and constructive comments!\n\n> There is the paper \"Multi-objective GFlowNets\" [MO; arXiv:2210.12765, ICML 2023], which also studies GFlowNets for multiobjective Bayesian optimization by conditioning on scalarization weights. Can you comment on the differences in problem setting or method with that paper?\n\nOur work is actually concurrent with [MO]. They propose two variants MOGFN-PC and MOGFN-AL for MOO in the single-round scenario and Bayesian optimization (BO) scenario, respectively. Indeed, when HN-GFN is used as a stand-alone optimizer outside of MOBO (Section 5.1), it is similar to MOGFN-PC except for using different conditioning mechanisms. As for MOBO, MOGFN-AL is a vanilla GFlowNet whose reward function is defined as a multi-objective acquisition function (NEHVI). In our early experiments, we also tried to directly optimize this acquisition function. Unfortunately, we found that this approach is ineffective as the number of objectives increases. Because the value of NEHVI is so tiny that the policy cannot be effectively optimized. This phenomenon is tougher when the exponent of the reward is large. While in [MO], they only consider two objectives, and we consider setting up to four objectives which exposes the aforementioned challenges that we address in this work. In addition to extending GFlowNet, we delicately propose a hindsight-like off-policy strategy which is rarely studied for MOO (in both RL and GFlowNet literature).\n\n> The attached code includes the implementations of two baselines, in code/generator/gfn.py, but they are not included in the paper:\n> - MOReinforce, which is a natural candidate for an RL-based baseline that can readily share the parametrization of the agent with the GFlowNet.\n> - GFlowNet with trajectory balance [TB; arXiv:2201.13259, NeurIPS 2022]. Trajectory balance is the objective used in nearly all work on GFlowNets for molecular optimization after [5], but for some reason, it is not evaluated or mentioned in the text. In particular, it is evaluated on molecule synthesis in both [TB] and [MO].\n\nThanks for the careful review of our code. 1) Indeed, MOReinforce (arXiv: 2203.15386, ICLR 2022) was included in the paper (referred to as P-MOCO). We believe this misunderstanding is caused by inconsistent naming: [P-MOCO] is termed MOReinforce in [MO], while we use the original name P-MOCO. 2) In our early experiments, we found that TB performs worse than FM, and we encountered a numerical issue with NaN in the predicted logits. We contacted the author of [TB] and described the situation, and he later confirmed that 'I did also get the error you reported when I ran the code myself.' Hence, we have chosen to use FM. In [MO], they did evaluate it on molecule synthesis. However, in their ablation experiments about the fragment-based molecular generation task (Figure 3c in ICML version and Table 6 in 2210.12765v1), we found that only when an extremely large exponent (96) is set can MOGFN-PC outperforms baselines (Table 2 in ICML version and Table 3 in 2210.12765v1). To some extent, we argue that setting such a large exponent is unreasonable, especially for sampling algorithms like GFlowNets, because the probability of sampling the molecules with slightly lower rewards (before the exponential operation) will be very low. Put differently, the sampling distribution is concentrated close to the modes, and the diversity of the candidates is sacrificed.\n\n> There are some inconsistencies in definitions related to GFlowNets. Equation (1) writes $R(s)$ for the reward received when terminating at $s$ as in [5]. However, the text immediately following it talks about terminal and nonterminal states (with nonterminals having reward 0), and the text above says there is a special termination action that leads to terminal states. This is more consistent with the convention from [TB], where terminal states have no children and are the only states with nonzero rewards. This is not a major bug as the two conventions are equivalent, but it would be good to be careful.\n\nThanks again for your patience and helpful reminder. We will carefully revise these inconsistencies in definitions to ease the understanding of GFlowNets.\n\n**We hope our response can alleviate your concerns. Please let us know if you have any additional questions.**"}, {"review_id": "dDToAeawjp", "reviewer": "Reviewer_Dc63", "summary": "Molecule generation involves the optimization of multiple potentially competing objectives simultaneously. As evaluating these objectives can be a time-consuming and costly task, sample efficiency is paramount. This work proposes a multi-objective Bayesian optimization approach leveraging GFlowNets to tackle this problem. The authors propose a preference-based decomposition and a hyper-network-based parameterization for incorporating preferences and conditioning on those. They also employ a hindsight-experience replay strategy to utilize offline data during learning. The hyper-network-based GFlowNet is then used as an acquisition function optimizer to find a diverse set of molecules on the empirical Pareto front. ", "questions": "- It would be nice if the authors could dedicate some more information about the test problem. What is the search space of the molecules, what atom types, how are impossible molecular structures avoided, etc? Are the building blocks created in such a way that they always can be connected to each other without violations?\n\n- Why a batch size of 100 and 8 rounds? In what setting would a practitioner be able to perform 100 wet lab experiments simultaneously? Perhaps with a simulator, this would be possible. But wouldn't the surrogate models (at least from the BO qEHVI and qParEGO) benefit from lower batch sizes and more iterations? This could change the benchmarks significantly.", "limitations": "The authors have addressed limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper covers an important and impactful topic of multi-objective optimization in molecule design.\n-  The experimental results, in principle, are strong.\n- It is an original combination of existing techniques.\n- Many benchmarks are compared.\n- Ablation study", "weaknesses": "- It is not directly evident to me how this work is different from Pareto GFlowNet that was proposed in [GFlowNet Foundations](https://arxiv.org/pdf/2111.09266.pdf)\n\n- Some parts remain somewhat unclear to me. How is the hyper-network trained? Is it stable, does it do what it is expected to do? \n\n- I am somewhat missing the link between the surrogate model and HN-GFN from what is described in the main paper. Why do we actually need the surrogate model? Can't the GFlownet propose diverse candidates by itself?\n\n- Perhaps the work would benefit from a clearer description and/or toy tasks to clear this up.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer for the insightful and constructive comments!\n\n> It is not directly evident to me how this work is different from Pareto GFN proposed in GFlowNet Foundations.\n\nWhile the concept of Pareto GFN was theoretically discussed in GFlowNet Foundations, we are among the first to study and instantiate this concept for MOO, and we address practical challenges (not discussed thoroughly in the original theoretical exposition) to potentially make sample-efficient molecular optimization a reality. We extensively study the impact of the conditioning mechanism (Sec5.1) and surrogate models (Sec4.4). Moreover, we delicately propose a hindsight-like off-policy strategy (Sec4.3) which is rarely studied for MOO (in both RL and GFN literature).\n\n> How is the hypernetwork trained? Is it stable, does it do what it is expected to do?\n\nWe are sorry for our unclear description. Hypernetworks generate the weights of a target network based on inputs. In our implementations, the hypernetwork takes the preference vector $λ$ as inputs and outputs the weights of prediction heads upon the MPNN encoder. In short, the parameters of the hypernetwork are optimized like normal parameters, while the parameters of the prediction heads are the output of the hypernetwork. \n\nIn our experiments, the hypernetwork is stable and does what it is expected to do. We attribute this to our design where only the weights of prediction heads are conditioned with hypernetwork while the weights of the encoder are shared among all preference vectors (Sec4.2.1). We study various conditioning mechanisms and find that the hypernetwork-based approach performs best.\n\n> I am somewhat missing the link between the surrogate model and HN-GFN. Why do we actually need the surrogate model? Can't GFN propose diverse candidates by itself? Perhaps the work would benefit from a clearer description and/or toy tasks to clear this up.\n\nWe apologize for our unclear description. We propose to extend GFN for MOO. We first consider a synthetic single-round scenario (Sec5.1) where the oracle is assumed to be called as many times as necessary, without the need of a surrogate model. In this case, GFN can directly optimize the reward defined by the oracle. Practically, realistic oracles are extremely expensive to evaluate. We thus focus more on the BO scenario (Sec5.2) where the oracle is given an evaluation budget. Then we need a cheap statistic surrogate model to approximate the expensive oracle. As the surrogate model cannot exactly reproduce the oracle's full behaviors, an ideal one should provide a calibrated uncertainty (exploration) estimate. By incorporating uncertainty into the reward function, GFNs search not only the space where the model prediction is high (exploitation) but also the space where the uncertainty is large (exploration). This behavior is paramount for BO.\n\n> It would be nice if the authors could dedicate some more information about the test problem. What is the search space of the molecules, what atom types, how are impossible molecular structures avoided, etc? Are the building blocks created in such a way that they always can be connected to each other without violations?\n\nWe appreciate the reviewer's helpful reminder. We follow the molecule environment of [1], where the molecules are generated by sequentially attaching a fragment (from a predefined vocabulary of building blocks) to an atom of the partially constructed molecules. The maximum trajectory length is 8, with the number of actions varying between 100 and 2000 depending on the state (the larger a molecule, the more possible additions exist), making $|\\mathcal{X}|$ up to $10^{16}$. The blocks are chosen via the process proposed in JT-VAE [2]. To ensure valid molecular structures, we only predict action (which block to attach) for each stem of the graph (stems are atoms which new blocks can be attached to). We will revise our writing to dedicate experimental settings.\n    \n> Why a batch size of 100 and 8 rounds? In what setting would a practitioner be able to perform 100 wet lab experiments simultaneously? Perhaps with a simulator, this would be possible. But wouldn't the surrogate models (at least from qEHVI and qParEGO) benefit from lower batch sizes and more iterations? This could change the benchmarks significantly.\n\nWhile these choices are unusual for common problems in BO, they are common but challenging in molecular optimization and biological sequence design, where large-batch and low-round setting is desired, as candidates can be synthesized and evaluated in parallel in biochemical experiments. For example, DyNA PPO [3] considers 10 rounds with a batch size of 100. The original paper of GFlowNet [1] even considers a batch size of 200.\n\nI agree that the surrogate models benefit from lower batch sizes and more iterations. However, in practice, the time costs for proposing candidates are negligible compared to evaluation, especially when the candidates are proposed by generative models, these novel candidate molecules may not be available for purchase. Customized synthesis may take weeks if not more for each round; hence we consider such an experimental setting to be more practical in chemistry-related fields. As we noted in line 322, the latent space optimization (LSO) methods (qEHVI, qParEGO, and LaMOO) only support 160 rounds with batch size 5 due to memory constraints. Nevertheless, LSO methods still fall far short of combinatorial optimization methods. We believe that the reason is that the latent space learned by generative models (HierVAE and JT-VAE) is not sufficiently expressive and discriminative.\n\n[1] Flow network based generative models for non-iterative diverse candidate generation\n\n[2] Junction tree variational autoencoder for molecular graph generation\n\n[3] Model-based reinforcement learning for biological sequence design\n\n**We hope our response can alleviate your concerns. Please let us know if you have any additional questions.**"}]

Author rebuttal:
Dear Area Chairs and Reviewers,

We greatly appreciate the reviewers' time, valuable comments, and constructive suggestions. Overall, the reviewers deem our paper as "well-organized" (9YCN, BkDd, hutP), studying "an important problem" (9YCN, BkDd, Dc63), acknowledging our methodology novelty and soundness (9YCN, BkDd, hutP, Dc63), experiments with strong experimental results (hutP, Dc63) and ablation study (hutP, Dc63), demonstrating the potential use and advantages of the proposed method (9YCN, BkDd). 
    
In the author response period, we made diligent efforts to address reviewers' concerns and provided additional experimental results to further verify our contributions. The summary of our main efforts is presented as follows:

- We have provided a detailed explanation for the protocol, experimental details, and implementation details. (To 9YCN, BkDd, hutP, and Dc63)

- We have further conducted diverse experiments (offline biological sequence design and different molecular optimization benchmarks) to demonstrate the general applicability of the proposed method (To 9YCN and BkDd)

- We have further conducted experiments to analyze the impact of the initial dataset to demonstrate that the results are not sensitive to the initial dataset (To 9YCN)

- We have further discussed and conducted experiments to analyze the impact of batch size to demonstrate that our setting (a batch size of 100 and 8 rounds) is reasonable and suitable for the design and benchmarking of algorithms for molecular optimization (To BKDd and Dc63)

In our individual responses, we provide detailed answers to all the specific questions raised by the reviewers. Further discussions are welcomed to facilitate the reviewing process toward a comprehensive evaluation of our work.

Meta-review: This paper proposes a new GFlowNet framework for multi-objective optimization of molecules. The reviewers appreciate the importance of the problem, presentation, and the idea of using GFlowNets (which can produce diverse outputs) for multi-objective optimization. 

After reading the paper myself, I particularly enjoyed the idea of using hyper-networks to condition GFlowNets on the preference vectors. In terms of multi-objective molecular optimization, I think the empirical evaluation could be improved by adding more competitive baselines, e.g., Winter et al., 2019. However, despite the moderate empirical evaluation, I believe this work provides a useful contribution to the GFlowNet community, so I would recommend acceptance for this work. 

Finally, as the authors mentioned, there is a concurrent work that already appeared at ICML 2023 that shares the idea of using GFlowNets for multi-objective optimization. There also exists the Pareto GFlowNet that shares a similar idea with the paper. While I agree that this work provides new contributions even when compared with the ICML 2023 paper and the Pareto GFlowNet, I recommend the authors add a thorough discussion with respect to these papers, once the paper is accepted. An empirical comparison would be even more significant and beneficial for the community.

[Winter et al., 2019] Efficient multi-objective molecular optimization in a continuous latent space
Decision: Accept (poster)