Multi-Objective GFlowNets
Authors: Moksh Jain
Sharath Chandra Raparthy
Alex Hernández-García
Jarrid Rector-Brooks
Yoshua Bengio
Santiago Miret
Emmanuel Bengio
Venue: Submitted to ICLR 2023 (2023)
Keywords: generative flow networks
multi-objective optimization
drug discovery
material design

Abstract: In many applications of machine learning, like drug discovery and material design, the goal is to generate candidates that simultaneously maximize a set of objectives. As these objectives are often conflicting, there is no single candidate that simultaneously maximizes all objectives, but rather a set of Pareto-optimal candidates where one objective cannot be improved without worsening another. Moreover, these objectives, when considered in practice are often under-specified, making diversity of candidates a key consideration. The existing multi-objective optimization methods focus predominantly on covering the Pareto front, failing the capture diversity in the space of candidates. Motivated by the success of GFlowNets for generation of diverse candidates in a single objective setting, in this paper we consider Multi-Objective GFlowNets (MOGFNs). MOGFNs consist of a Conditional GFlowNet which models a family of single-objective sub-problems derived by decomposing the multi-objective optimization problem. Our work is the first to empirically demonstrate conditional GFlowNets. Through a series of experiments on synthetic tasks and real-world domains, we empirically demonstrate that MOGFNs outperform existing methods in terms of Hypervolume, R2-distance and candidate diversity. We also demonstrate the effectiveness of MOGFNs over existing methods in active learning settings. Finally, we supplement our empirical results with a careful analysis of each component of MOGFNs.

MULTI-OBIECTIVE GFLOWNETS
Anonymous authors Paper under double-blind review
ABSTRACT
In many applications of machine learning, like drug discovery and material design, the goal is to generate candidates that simultaneously maximize set of objectives. As these objectives are often conflicting, there is no single candidate that simul- taneously maximizes all objectives, but rather set of Pareto-optimal candidates where one objective cannot be improved without worsening another. Moreover, in practice, these objectives are often under- snder--pecified. making the diversity of candi- dates a key consideration. The existing multi-objective optimization methods focus predominantly on covering the Pareto front, failing to capture diversity in the space of candidates. Motivated by the success of GFlowNets for generation of diverse candidates single objective setting, in this paper we consider Multi-Objective GFlowNets (MOGFNs). MOGENs consist of novel Conditional GFlowNet which models a family of single-objective sub-problems derived by decomposing the multi-objective optimization problem. Our work is the first to empirically demon- strate conditional GFlowNets. Through a series of experiments on synthetic and benchmark tasks, we empirically demonstrate that MOGFNs outperform existing methods in terms of Hypervolume, R2-distance and candidate diversity. We also demonstrate the effectiveness of MOGFNs over existing methods in active learning settings. Finally, we supplement our empirical results with a careful analysis of each component of MOGFNs.
INTRODUCTION
Decision making in practical applications often involves reasoning about multiple, often conflicting. objectives (Keeney et al., 1993). For example, in drug discovery, the goal is to generate novel drug-like molecules that inhibit target, easy to synthesize and can safely be used by humans (Dara et al., 2021). Unfortunately, these objectives often conflict molecules effective against a target might also have adverse effects humans- so there is no single molecule which maximizes all the objectives simultaneouslly. Such problems fall under the umbrella of Multi Sollti-Objeetvvee Optimization (MOO; Ehrgott, 2005; Miettinen, 2012), wherein one is interested in identifying Pareto-optimal candidates. The set of Pareto-optimal candidates covers all the best tradeoff's among the objectives, i.e., the Pareto front, where each point on that front corresponds different set of weights associated with each of the objectives.
In-silico drug discovery and material design are typically driven by proxies trained with finite data, which only approximate the problem's 's true objectives, and therefore include intrinsic epistemic uncertainty associated with their predictions. In such problems, not only it is important to cover the Pareto front, but also to generate sets of diverse candidates at each solution of the front so as to increase the likelihood of success downstream evaluations (Jain et al., 2022).
Generative Flow Networks (GFlowNets; Bengio et al., 2021a;b) are recently proposed family of probabilistic models which tackle the problem of diverse candidate generation. Contrary to the reward maximization view of reinforcement learning (RL) and Bayesian optimization (BO), GFlowNets sample candidates with probability proportional to the reward. Sampling candidates, as opposed to greedily generating them, implicitly encourages diversity in the generated candidates. GFlowNets have shown promising results in single objective problems of molecule generation (Bengio et al., 2021a) and biological sequence design (Jain et al., 2022).
In this paper, we study Objtti-Obeectiv GFlowNets (MOGFNs). extensions of GFlowNets which tackle the multi-objective optimization problem. We consider two variants of MOGENs

---

(a) Preference- Coneriece-Co GFlowNets (MOGFN-PC) which combine Conditiaon GFlowNets (Bengio et al., 2021b) with Weighted Sum Scalarization (Ehrgott, 2005) and (b) MOGEN- AL, extension of GFlowNet-AL (Jain et al., 2022) for multi-objective active learning settings. We empirically demonstrate the advantage of MOGFNs over existing approaches on a variety of high- dimensional multi-objective optimization tasks: the generation of small molecules, DNA aptamer sequences and fluorescent proteins. Our contributions are as follows:
C1 We demonstrate how two variants of GFlowNets - MOGFN-PC and MOGFN-AL can be applied multi-objective optimization. Our work the first successful empirical validation of -wwarl-Conditionall GFlowNets (Bengio et al., 2021b).
C2 Through series of experiments molecule generation and sequence generation demonstrate that MOGFN-PC generates diverse Pareto-optimal candidates.
C3 In a challenging active learning task for designing fluorescent proteins, we show that MOGFN-AL results in significant improvements to sample-efficiens and diversity of generated candidates.
C4 We perform a thorough analysis of the main components of MOGENs to provide insights into design choices that affect performance.
BACKGROUND
2.1 MULTI-OBIECTIVE OPTIMIZATION
Multi-objective optimization (MO0) involves finding a set of feasible candidates x² E X' which all simultaneously maximize a set of objectives:
max (R1())..... RR(x))
=EX
In general, the objectives being optimized can be conflicting such that there is no single x' which simultaneously maximizes all objectives. Consequently, the concept of Pareto optimality is adopted in MOO, giving rise to set of solutions trading off the objectives in different ways.
Given F1, x2 E X, Ii is said to dominate X2, written (#1 > 12). iff (x12)) 2 (x(22) Vi {1...., d} and 3k E [1....,d) d} such that R& (x1) > Rs (x2). A candidate x' is Pareto-optimal if there exists no other solution x' E X' which dominates x². In other words, for a Pareto-optimal candidate it is impossible to improve one objective without sacrificing another. The Pareto set is the set of all Pareto-optimal candidates in X, and the Pureto front defined as the image of the Pareto set in objective-space It is important to note that since the objectives being optimized in general might not be injective, any point on the Pareto front can be the image of several candidates in the Pareto set. This introduces notion of diversity in the candidate space, capturing all the candidates corresponding to a point on the Pareto front, that is critical for applications such as drug discovery.
While there several paradigms for tackling the MOO problem (Ehrgott, 2005; Miettinen, 2012; Pardalos et al., 2017), we consider Scalarization, where the multi-objective problem is decomposed into simpler single-objective problems, it is well suited for the GFlowNet formulation introduced in Section 3. A set of weights (preferences) wi are assigned to the objectives R2 such that wi 2 0 n and > i= Wi 1. The MOO problem in Equation 1 is then decomposed into solving single-objective sub-problems of the form max,ex R[xlw), where
a scalarization function.
1=1 wi R2 (x) is a widely used scalarization function which results Pareto optimal candidates for problems with convex Pareto front (Ehrgott, 2005). Weighted Tchebycheff. R(rlw) min wil [x) 2 where 2 denotes some ideal value for objective R,
Weighted Sum Scalarization, R(x(w) = Σd=
i results in Pareto optimal solutions even for problems with a non-convex Pareto front (Pardalos et al., 2017). See Appendix B for more discussion on scalarization. In summary, using scalarization, the MOO problem can be viewed as solving a family of single-objective optimization problems.
2.2 GELOWNETS
Generative Flow Networks (Bengio al., 2021a;b) are family of probabilistic models which generate, through a sequence of steps.compositional objects x e X' with probability proportional to a given reward R: X R+ The sequential construction of x E X can be described trajectory
2

---

T E T in weighted directed acyclic graph (DAG)' g (S,E), starting from an empty object su and following actions a E A as building blocks. The nodes S of this graph (states) correspond to the set of all possible objects that can be constructed using sequences of actions in A. An edge s S s E indicates that action a at state s leads state s'.
The forward policy (-|))) is a distribution over the children of state 8. x can be generated by starting sy and sampling sequence of actions iteratively from Pp. Similarly, the backward policy (B(- is distribution over the parents of state s and can generate backward trajectories starting at any state z, e.g., iteratively sampling from PB starting at x shows way x' could have been constructed. Let I (I: be the marginal likelihood of sampling trajectories terminating in following Pr. and partition function Z = Ezex R(x:). The learning problem solved by GFlowNets to estimate Pp such that π (xx) x R(z). This achieved using learning objectives like trajectory balance (TB: Malkin et al.. 2022), to learn (- 0), Pb (s; 0), Zo which approximate the forward and backward policies and partition function. parameterized by 6. We refer the reader to Bengio et al. (2021b); Malkin et al. (2022) for more thorough introduction to GelowNets.
3 MULTI-OBJECTIVE GFLOWNETS
We broadly categorize Hulti-Objectivv GFlowNets (MOGFNs) as GFlowNets which solve family of sub-problems derived from Multi-OOjecttive Optimization (MOO) problem. We first consider solving a family of MOO sub-problems simultaneously with preference-conditional GFlowNets, followed by MÓGEN-AL, which solves sequence of MOO sub-problems.
3.1 PREFERENCE PREFRRENCE-CONDTTOONNL GELOOWNETS
Whereas a GFlowNet Iearns how to sample according to a single reward function, rewarr-conditionll GFlowNets (Bengio et al., 2021b) are a generalization of GFlowNets that simultaneously model a family of distributions associated with corresponding family of reward functions. Let C denote a set of values c, with each c E C inducing a unique reward function R(x c). We can define a family of weighted DAGs {G. = (Se, E), c E C) which describe the construction of æ E X, with conditioning information e available at all states in S...
We denote
Pr(- c) and Pn(- e) the conditional forward and backward policies, Z(c) Exex R(z[c) as the conditional partition function and x (I]ec) as the marginal likelihood of sampling trajectories 7 from PF terminating in z given c. The learning objective in reward-conditionn GelowNets is thus estimating (- c) such that T (2|c) x R(x]c). We refer the reader to Bengio et al. (2021b) for a more formal discussion on conditional GFlowNets.
Recall from Section 2.1 that MOO problems can be decomposed into a family of single-objective problems each defined by a preference w over the objectives. Thus, we can employ reward-conditional GFlowNets to model the family of reward functions by using as the conditioning set C the d-simplex A spanned by the preferences w over d objectives.
cenference-condiiionall GFlowNets (MOGFN-PC) are reward-conditional GFlowNets conditioned on the preferences w E A over a set of objectives [R, (z),..., R.) In other words, MOGFN-PC model the family of reward functions R((lww where R(zlw) itself corresponds to scalarization of the MOO problem. We consider three scalarization techniques, which are discussed in Appendix B:
Weighted-sum (WS) (Ehrgott, 2005): R(Ilw) = Eim wiR((x)
Weighted-log-sum (WI.): R(x]w) = I (x)
Weighted-Tchebbyhee (WT) (Choo & Atkins, 1983): R(zlw) = min W. R1(x)) 2/ |-- Isid
MOGEN-PC is not constrained to any scalarization function, and can incorporate any user-defined scalarization scheme that fits the desired optimization needs.
Training MOGEN-PC The procedure to train MOGEN-PC, or any reward-conditional GelowNet, closely follows that of standard GFlowNet and is described in Algorithm 1. The objective is to learn
If the object is constructed in canonical order (say a string constructed from left to right), is tree.
3

---

the parameters 8 of the forward and backward conditional policies Pp(-Is, w 0) and Py(-Is', w 0), and the log partttitiion function log Zo(w). To this end, we consider an extension of the trajectory balance objective for reward-conditions GFlowNets:
(T, 1og Zo(w) IL PF wi 0)
2
R(x II PB(s|s' w: 8)
One important component is the distribution p(w) used to sample preferences during training, p(w) influences the regions of the Pareto front that are captured by MOGFN-PC. In our experiments, we use a Dirichlet(a) to sample preferences which are encoded with thermometer encoding (Buckman et al., 2018) when input to the policy. Following prior work, we also use an exponent /3 for the reward R(z]w), i.e. (<r/w) R( This incentivizes the policy to focus on the modes of R(ælw), which is critical for generation of high reward and diverse candidates.
MOGFN-PC and MOReinforce MOGFN-PC closely related to MOReinforce (Lin et al., 2021) in that both learn preference-condition policy to sample Pareto-optimal candidates. The key differ- ence is the learning objective: MOReinforce uses a multi-objeccivve version of REINFORCE (Williams, 1992), whereas MOGFN-PC uses a preference-conditional GFlowNet objective as in Equation (2). As discussed in Section 2. each point on the Pareto front (corresponding to a unique w) can be the image of multiple candidates in the Pareto set. MOReinforce, given a preference will converge to sampling a single candidate that maximizes R(zlw). MOGFN-PC, on the other hand, samples from R(x[w), which enables generation of diverse candidates from the Pareto set for a given w. This is key feature of MOGFN-PC whose advantage we empirically demonstrate in Section 5.
3.2 MULTI-OBJECTIVE ACTIVE LEARNING WITH GFLOWNETS
In many practical scenarios, the objective functions of interest are computationally expensive. For instance, in the drug discovery scenario, evaluating objectives such as the binding energy to target even in simulations the objective functions, and diversity of candidates, thus become critical in such scenarios. Black-box optimization approaches involving active learning (Zuluaga et al., 2013), particularly multi-objective Bayesian optimization (MOBO) methods (Shah & Ghahramani, 2016; Gameet,, 2022) are powerful approaches in these settings.
take several hours. Sample-efficiency, in terms of number of evaluations of
MOBO uses a probabilistic model to approximate the objectives R = [R Ra} and leverages the epistemic uncertainty in the predictions of the model as a signal for prioritizing potentially useful candidates. The optimization is performed over M rounds, where each round consists of generating batch of candidates B given all the candidates D; proposed in the previous rounds. The batch B is then evaluated using the true objective functions. The candidates are generated in each round by maximizing an acquisition function a which combines the predictions with their epistemic uncertainty into a single scalar utility score. We note that each round is effectively scalarization of the MOO problem, and as such may be decomposed into each round's single objective problem.
We broadly define MOGFN-AL as approaches which use GFlowNets to generate candidates in each round of an active learning loop for multi-objective optimization. MOGFN-AI tackles MOO through sequence of single-objective sub-problems defined by acquisition function a. As such, MOGFN-AL can be viewed as a multi-objective extension of GFlowNet--L (Jain et al., 2022). In this work, we consider instantiation of MOGFN-AL for biological sequence design summarized in Algorithm 2 (Appendix A), building upon the framework proposed by Stanton et al. (2022).
We start with an initial dataset Do- = (st, 11) of candidates x4 E X and their evaluation with the true objectives y/ = R(x). D; is used train surrogate probabilistic model (proxy) of the true objectives f X + RRd. which we parameterize as a multi-task Gaussian process (Shah & Ghahramani, 2016) with deep kernel (DKL GP: Maddox et al., 2021a;b). Using this proxy, the acquisition function defines the utility to be maximized a a: X' x F + R, where F denotes the space of functions represented by DKL GPs. In our work we use acquisition function noisy expected hypervolume improvement (NEHVI; Daulton et al., 2020).
We use GFlowNets to propose candidates each round by generating mutations for candidates P: where P., is the set of non-dominated candidates in D, Given sequence x., the GFlowNet
4

---

generates set of mutations [(4, e,)) where E (l...., [.....,|[[] is the location be replaced and v E A is the token to replace x||] while T is the number of mutations. This set is generated sequentially such that each mutation is sampled from Pr conditioned on z and the mutations sampled so far {(1,, n.)]. Let x' m be the sequence resulting from mutations m on sequence x. The reward for set of sampled mutations for x: is the value of the acquisition function on z'm- R(m, 2)) = a(x' mllf). This approach of generating mutations to existing sequences provides an key advantage over generating sequences token-by by-token as done in prior work (Jain et al., 2022) better scaling for longer sequences. We show empirically Section 5.3 that generating mutations with GelowNets results in more diverse candidates and faster improvements to the Pareto front than LaMBO (Stanton et al., 2022).
4 RELATED WORK
Evolutionary Algorithms (EA) Traditionally, evolutionary algorithms such NSGA-II have been widely used in various multi-objective optimization problems (Ehrgott, 2005; Konak et al., 2006; Blank & Deb, 2020). More recently, Miret al. (2022) incorporated graph neural networks into evolutionary algorithms enabling them to tackle large combinatorial spaces. Unlike MOGFNs, evolutionary algorithms do not leverage any type of data, including past experiences, and therefore are required to solve each instance of a MOO from scratch rather than by amortizing computation during training order to quickly generate solutions at run-time. Evolutionary algorithms, however, can be augmented with MOGENs for generating mutations to improve efficiency, as in Section 3.2.
Multi-Ohjective Reinforcement Learning MOO problems have also received significant interest
in the reinforcement learning (RL) literature (Hayes et al., 2022). Traditional approaches broadly consist of learning sets of Pareto-dominant policies (Roijers al., 2013; Van Moffaert & Nowé, 2014; Reymond et al., 2022). Recent work has focused on extending Deep RL algorithms for multi-objective settings such as Envelope-MOO and MOReinforce (Lin et al., 2021). A general shortcoming of RL based approaches is that they only discover a single mode of the reward function, and thus cannot generate diverse candidates, which also persists in the multi-objective setting. In contrast, MOGENs sample candidates proportional to the reward, implicitly resulting in diverse candidates.
(Yang al., 2019), MO-MPO (Abdolmaleki et al., 2020; 2021).
Multi-Objective Bayesian Optimization (MOBO) Bayesian optimization (BO) has been used in the context of MOO when the objectives are expensive evaluate and sample-efficiency is key consideration. MOBO approaches consist of learning a surrogate model of the true objective functions, which is used to define an acquisition function such as expected hypervolume improve- ment (Emmerich et al., 201 I; Daulton et al., 2020; 2021) and max-value entropy search (Belakaria et al., 2019), as well as scalarization-based approaches (Paria et al., 2020; Zhang & Golovin, 2020). Stanton et al. (2022) proposed LaMBO, which uses language models in conjunction with BO for multi-objective sequence design problems. The key drawbacks of MOBO approaches are that they do not consider the need for diversity in generated candidates and that they mainly consider continuous state spaces. As we discuss in Section 3.2, MOBO approaches can be augmented with GelowNets for diverse candidate generation in discrete spaces.
Other Works Zhao et al. (2022) introduced LaMOO which tackles the MOO problem by iteratively splitting the candidate space into smaller regions, whereas Daulton et al. (2022) introduce MORBO, which performs BO in parallel on multiple local regions of the candidate space. Both these methods, however, are limited to continuous candidate spaces.
EMPIRICAL RESULTS
In this section, we present our empirical findings across wide range of tasks ranging from sequence design to molecule generation. The experiments cover two distinct classes of problems in the context of GFlowNets: where g is DAG and where it is tree. Through our experiments, we aim to answer the following questions:
Q1 Can MOGFNs model the preference-comdiiional reward distribution?
Q2 Can MOGFNs sample Pareto-optimal candidates?
Q3 Are candidates sampled by MOGENs diverse?
Q4 Do MOGENs scale to high-dimensional problems relevant in practice?
5

---

Unigrams ICorflicting!
Bigrams (Correlated)
20005
Frent
150020
00
2,013
50013
poppa Boardd
0.2 04 3.
(b)
(C)
Figure 1: (a) The distribution learned by MOGFN-PC (Top) almost exactly matches the ground truth distribution (Bottom), in particular capturing all the modes, on hypergrid size 32 x 32 with 3 objectives. (b) and (c) illustrate the Pareto front of candidates generated by MOGEN-PC with conflicting and correlated objectives respectively.
Metrics: We rely on standard metrics such as the Hypervolume (HV) and R2 indicators, well as the Generational Distance+ (GD+). To measure diversity we use the Top-K Diversity and Top-K Reward metries of Bengio et al. (2021a). We detail all metrics in Appendix D. For all our empirical evaluations we follow the same protocol. First, we sample a set of preferences which are fixed for all the methods. For each preference their scalarized reward and diversity, and report the averages over preferences. We then use these samples to compute the HV and R2 indicators. We pick the best hyperparameters for all methods based on the HV and report the mean and standard deviation over 3 seeds for all quantities.
sample 128 candidates from which pick the top 10, compute
We consider the closely related MOReinforce (Lin et al., 2021) as baseline. We also study its variants MOSoftQL and MOA2C which use Soft Q-Learning (Haarnoja et al., 2017) and A2C (Mnih al., 2016) place of REINFORCE. We also compare against Envelope-MOQ (Yang et al., 2019), another popular multi-objective reinforcement learning method. For segment--bseed molecule generation we consider an additional baseline MARS (Xie al., 2021), a relevant MCMC approach for this task. To keep comparisons fair, we omit baselines like LaMOO (Zhao et al., 2022) and MORBO (Daulton et al., 2022) as they are designed for continuous spaces and rely on latent representations from pre- pre-trained models for discrete tasks like molecule generation.
Baselines:
5.1 SYNTHETIC TASKS
5.1.1 HYPER-GRID
We first study the ability of MOGEN-PC capture the preference-condittonal reward distribution in multi-objective version of the HyperGrid task from Bengio et al. (2021a). The goal here is to navigate proportional to a reward within a HyperGrid. We consider the following objectives for our experiments: brannin (x), currin (x), shubert (x)
Since the state space is small, we can compute the distribution learned by MOGFN-PC in closed form. In Figure la, we visualize π (x]w), the distribution learned by MOGFN-PC conditioned on set of fixed preference vectors w and contrast it with the true distribution R(rlw) in a x 32 hypergrid with 3 objectives. We observe that T (-- and R(-Ww) are very similar. To quantify this, we compute E, (r/() R(x w)/Z(w)) averaged over a set of 64 preferences, and find difference of about 10-4 Note that MOGFN-PC. is able to capture all the modes in the distribution, which suggests the candidates sampled from ı would be diverse. Further, we compute the GD+ metric for the Pareto front of candidates generated with MOGFN-PC, which comes up to average value of 0.42. For more details about the task and the additional results, refer to Appendix E.
5.1.2 N-GRAMS TASK
We consider version of the synthetic sequence design task from Stanton et al. (2022). The task consists of generating strings with the objectives given by occurrences of a set of d n-grams.
In the results summarized in Table 1, we consider Bigrams (with common characters in the bigrams resulting in correlated objectives) and 3 Unigrams (conflicting objectives) the objectives. MOGEN- PC outperforms the baselines in terms of the MOO objectives while generating diverse candidates.
We present additional results with more objectives in Appendix F.1

---

Table 1: N-Grams Task:
Diversity and Pareto performance of various algorithms on for the Bigrams and 3 Unigrams tasks with MOGFN-PC achieving superior Pareto performance.
Algorithm
3 Bigrams
Menograms
Reward (t)
Diversity
Rs
Reward
Diversity (t)
<t>
Rs (4)
Envelope-MOQ
0.08
Dig
2.012
19.06_n0e
D.08_oois
0_e
g003
21 28.18
MORcinforee
0.12aam
Dgo
g.015
20.32
0.05
0.e
a.036
21.04
MOSoftQL
0.28 jam
21.09
0.093
15.79
0.36
23.131 1o
0.105 La
12,80 paos
MOGFN-PC
0.44
19.79
0.220
9.97
0.38
22.71
+.12
11.39
Since the objective counts occurrences of n-grams, the diversity is limited by the performance, Le. high scoring sequences will have lower diversity, explaining higher diversity of MOSofiQL. We note that the MOReinforce and Sivellpe-MOQ baselines struggle in this task potentially due to longer trajectories with sparse rewards. MOGFN-PC adequately models the trade-off between conflicting objectives in the 3 Monograms task as illustrated by the Pareto front of generated candidates in Figure 1b. For the Bigrams task with correlated objectives, Figure Ic demonstrates MOGEN-PC generates candidates which can simultaneously maximize multiple objectives. We refer the reader to Appendix E.2 for more task details and additional results with different number of objectives and varying sequence length.
5.2 BENCHMARK TASKS
5.2.1
QM9
We first consider a small-molccule generation task based on the QM9 dataset (Ramakrishnan et al., 2014). We generate molecules atom-by-atom and bond-hy-bond with up to 9 atoms and use 4 reward signals. The main reward is obtained via MXMNet (Zhang et al., 2020) proxy trained on QM9 to predict the HOMO-LUMO gap. The other rewards are Synthetic Accessibility (SA), molecular weight target, and molecular logP target. Rewards are normalized to be between and 1, but the gap proxy can exceed 1, and so is clipped at 2. We train the models with 1M molecules and present the results in Table 2, showing that MOGEN-PC outperforms all baselines in terms of Pareto performance and diverse candidate generation.
Table Atom-based QM9 task: MOGFN-PC exceeds Diversity and Pareto performance on QM9 task with HUMO-LUMO gap, SA, QED and molecular weight objectives compared to baselines.
Algorithm
Reward (f)
Diversity (1)
(1)
Rgg (
MOA2C (Mnih al., 2016)
0.61 2005
4.39
1.16
6,28
Envelope QL (Yang et al., 2019)
0.65 2006
0,85
1.26
5,80
MOReinforce (1 al., 2021)
0.57
0.53
1.35
4.65 20.8
MOGEN-PC
0.76
0.93
1.40
2.44
5.2.2 FRAGMENT-BASER MOLECULE GENERATION
We evaluate our method on the fragment- based (Kumar et al., 2012) molecular generation task of Bengio et al. (2021a), where the task to generate molecules by linking fragments to form junction tree (Jin et al., 2020). The main reward function is obtained via pretrained proxy, available from Bengio et al. (2021a), trained on molecules docked with Autodock Vina (Trott & Olson, 2010) for the sEH target. The other rewards are based on Synthetic Accessibility (SA), drug likeness (QED), and a molecular weight target. We detail the reward construction Appendix E.4, Similarly to QM9, we train MOGFN-PC to generate 1M molecules and report the results in Table 3. We observe that MOGEN-PO is consistently outperforming baselines not only in terms of HV and Rz, but also candidate diversity score. Note that do not report reward and diversity scores for MARS, since the lack of preference conditioning would make it an unfair comparison.
5.2.3 DNA SEQUENCE GENERATION
As practical domain where the GFlowNet graph is a tree, we consider the generation of DNA aptamers, single-stranded nucleotide sequences that are popular in biological polymer design due to their specificity and affinity sensors in crowded biochemical environments (Zhou al., 2017; Corey et al., 2022; Yesselman et al., 2019; Kilgour et al., 2021). We generate sequences by adding one nucleobase (A, C, T or G) time, with maximum length of 60 bases. We consider three objectives:
7

---

Table Fragment-base Molecule Generation Task: Diversity and Pareto performance on the Fragment-base based drug design task with sEH, QED, SA and molecular weight objectives.
Algorithm
Reward (f) Diversity (1)
HV (f)
R2 (L)
MOReinforee (Lin al., 2021)
0.41 +0,7 +0.77
0.01 2000
0zo
9.88
MARS (Xie et al., 2021)
0.85_go08
1.94
MOA2C (Mnih al., 2016)
0.7 0.76
0.48 00339
0.75 =000
3.35
Envelope QL (Yang et al., 2019)
0.70
0.15 =ous
0.74
3,51
MOGFN-PC
0.89
0.75
0.90=
1.86
the free energy of the secondary structured calculated with the software NUPACK (Zadeh et al., 2011), the number of base pairs and the inverse of the sequence length to favour shorter sequences.
We report the results in Table In this case, the best Pareto performance is obtained by the multi-objective RL algorithm MOReinforce (Lin et al., 2021). However, it achieves by finding quasi-trivial solution with the pattern GCGCGC.. for most lengths, yielding very low diversity. In contrast, MOGFN-PC obtains much higher diversity and Top-K rewards but worse Pareto performance. An extended discussion, ablation study and further details are provided in Appendix E.5.
Table 4: DNA Sequence Design Task:
Diversity and Pareto performance of various algorithms DNA sequence generation task with free energy, number of base pairs and inverse sequence length objectives.
Algorithm
Reward (1)
Diversity (1)
HV (f)
R2 (J)
Envelope-MOQ (Yang et al., 2019)
0.238
0.0
0.16310013
5.657 5.657_10.73
MOReinforce (Lin al., 2021)
0.105
0.61782ax0
0.6292nmm
1.925
MOSofIQL
0.446 a0.10
32. g0.442
0.163
5.565 g0.17
MOGEN-PC
0.682
18. 131 90
0.517
2.432.20000
5.3 ACTIVE LEARNING
Finally, evaluate MOGEN-AL, consider the Proxy RFP task from Stanton et al. (2022), with the aim of discovering novel proteins with red fluorescence properties, optimizing for fold- ing stability and solvent-accessible surface area. We adopt all the experimental details (described in Appendix E.6) from Stanton et al. (2022), using MOGFN-AL for candidate generation. In addition to LaMBO, we use a model-free (NSGA-2) and model-based EA from Stanton et al. (2022) as baselines. We observe in Fig- 2a that MOGEN-AL results in significant gains to the improvement in Hypervolume rel- ative to the initial dataset, in a given budget of black-box evaluations. In fact, MOGFN-AL is able match the performance of LaMBO within about half the number of black-bor box evaluations.
Rel. Hypervalume
1.8
60
Stability
40
ROCTMAL
-20
300 450 660 053 1800
Black Box Evaluations
(a)
(b)
Algorithm
Diversity
NSGA-2
MTGP NEHVI GA
0. 2.
LaMBO
0. 1.
MOGEN
0.25 am
(c)
Figure 2: (a) MOGFN-AL demonstrates substan- tial advantage in terms of Relative Hypervolume and (b) Pareto frontier of candidates generated by MOGFN-AL dominates the Pareto front of the ini- tial dataset. (c) MOGFN-AL is particularly strong in terms of diversity of candidates.
Figure 2b illustrates that the Pareto frontier of candidates generated with MOGFN-AL, which dominates the Pareto frontier of the initial dataset. As we the candidates are generated by mutating sequences in the existing Pareto front, we also highlight the sequences that are mutations of each seqeunce in the initial dataset with the same color. To quantify the diversity of the generated candidates we measure the average value from DIAMOND (Buchfink et al., 2021) between the initial Pareto front and the Pareto frontier of generated candidates. Table 2c shows that MOGEN-AL generates candidates that are more diverse than the baselines.
8

---

6 ANALYSIS
In this section, we isolate the important components of MOGFN-PC: the distribution p(w) for sampling preferences during training, the reward exponent (3 and the reward scalarization R(alw) to understand the impact of each component on Pareto performance and diversity. We consider the 3 Bigrams task discussed in Section 5.1.2 and the fragment-based molecule generation task from Section 5.2.1 for this analysis and provide further results in the Appendix.
Impact of p(w) To examine the effect of p(w), which controls the coverage of the Pareto front, we set it to Dirichlet(a) and vary E 100). This results in w being sampled from different regions of Ad. Specifically, a = 1 corresponds to a uniform distribution over A a > 1 is skewed towards the center of A whereas < 1 is skewed towards the corners of A In Table 5 and Table 6 we observe that a = 1 results in the best performance. Despite the skewed distribution with a = 0.1 and a 10, still achieve performance close that of = 1 indicating that MOGFN-PO is able to interpolate to preferences not sampled during training. Note that diversity is not affected significantly by p(w).
Table 5: N-grams: Analysing the impact of o, f and R()lw) on the performace of MOGEN-PC
Metrics
ElTect of p(w)
Eflect of
Choice fi(z (w)
Dir(O.))
Dir(t)
DirtO)
16
48
WS
Reward (1)
g.38
0.44
0.37
0.22ge
0.36
0.44.mm
Diversity (1)
19.79 1848
18,56
29 2305
23.86
19.79
19.79
24.87
22.5)
Hypervolume (1) 0.17 aeoss
g.22
0.06 g.08
0.22 0.14
0.22 ar
2.097
H4 (1) 10.95 s021
9.97
10.52
.7.00.ozz
9.97 12.46.223
9.97
21.54
18.17
Impact of 3 During training 8, controls the concentration of the reward density around modes of the distribution. For large values of / the reward density around the modes become more peaky and vire-versa. Table 5 and Table 6 we present the results obtained by varying 3 E {16, 32, 48). As / increases, MOGFN-PC is incentivized to generate samples closer to the modes of R(*(w), resulting better Pareto performance. However, with high 3 values, the reward density is concentrated close to the modes and there is a negative impact on the diversity of the candidates.
Choice of scalarization R(*lw) Next, we analyse the effect of the scalarization defining R(x[w) used for training. The set of R(zlw) for different specifies the family of MOO sub-problemo and thus has a critical impact on the Pareto performance. Table 5 and Table 6 include results for the Weighted Sum (WS), Weighted-log-sum (WL) and Weighted Tchebycheff (WT) scalarizations. Note that we do not compare the Top-K Reward as different scalarizations cannot be compared directly. WS scalarization results in the best performance. WL scalarization on the other hand is not formally guaranteed to cover the Pareto front and consequently results in poor Pareto performance. We suspect the poor performance of WT and WL. are in part also due to the harder reward landscapes they induce.
CONCLUSION
In this work, we have empirically demonstrated the generalization of GFlowNets to conditional GFlowNets for multi-objective optimization problems (MOGFN) to promote the generation of diverse optimal candidates. We presented two instantiations of MOGEN: MOGFN-PC, which leverages reward-conditional GFlowNets (Bengio et al., 2021b) to model family of single-objective sub- problems, and MOGFN-AL, which sequentially solves a set of single-objective problems defined by multi-objective acquisition functions. Finally, we empirically demonstrated the efficacy of MOGENS for generating diverse Pareto-optimal candidates on sequence and graph generation tasks.
As limitation, we identify that in certain domains, such as DNA sequence generation, MOGEN generates diverse candidates but currently does not match RL algorithms in terms of Pareto perfor- mance. The analysis in Section 6 hints that the distribution of sampling preferences p(w) affects the Pareto performance. Since for certain practical applications only a specific region of the Pareto front is of interest, future work may explore gradient based techniques learn preferences for more structured exploration of the preference space. Within the context of MOFGN-AL AL, an interesting research avenue is the development of preference-conditional acquisition functions.

---

Reproducibility Statement We include the code necessary to replicate experiments with submission and provide detailed description of experimental setups in the Appendix. All datasets and pretrained models used are publicly available or included in the supplementary materials.
Ethics Statement We acknowledge that as with all machine learning algorithms, there is potential dual use of multi-objective GFlowNets by nefarious agents. This work was motivated by the application of machine learning accelerate scientific discovery in areas that benefit humanity. We explicitly discourage the use of multi-objective GFlowNets applications that may be harmful to others.
REFERENCES
Abbas Abdolmaleki, Sandy Huang, Leonard Hasenclever, Michael Neunert, Francis Song, Martina Zambelli, Murilo Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view on multi-objective policy optimization. In International Conference on Machine Learning, pp. 11-22. PMLR, 2020.
Abbas Abdolmaleki, Sandy Huang, Giulia Vezzani, Bobak Shahriari, Jost Tobias Springenberg, Shruti Mishra, Dhruva TB, Arunkumar Byravan, Konstantinos Bousmalis, Andras Gyorgy, et al. On multi-objective policy optimization as a tool for reinforcement learning. arXiv preprint arXiv:2/06.08199 2021.
Syrine Belakaria, Aryan Deshwal, and Janardhan Rao Doppa. Max-value entropy search for multi- objective bayesian optimization. Advances in Neural Information Processing Systems, 32, 2019.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021a. URL https: http:///ppeeee/eeeeeeeeeee
net forum? d=Arn2E4TRReE F
Yoshua Bengio, Tristan Deleu, Edward J. Hu, Salem Lahlou, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations, 2021b.
J. Blank and K. Deb. pymoo: Multi-objective optimization in python. IEEE Access, 8:89497-89509, 2020.
Benjamin Buchfink, Klaus Reuter, and Hajk-Georg Georg Drost. Sensitive protein alignments at tree-of-life scale using diamond. Nature methods, 1864)3366-368, 2021.
Jacob Buckman, Aurko Roy, Colin Raffel, and Ian Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. In International Conference on Learning Representations, 2018.
Eng Ung Choo and Derek Atkins. Proper efficiency in nonconvex multicriteria programming. Mathematics of Operations Research, 8()):467-470, 1983.
Peter JA Cock, Tiago Antao, Jeffrey T Chang, Brad A Chapman, Cymon J Cox, Andrew Dalke, Iddo Friedberg, Thomas Hamelryck, Frank Kauff, Bartek Wilezynski, al. Biopython: freely available python tools for computational molecular biology and bioinformatics. Bioinformatics, 25(11): 1422-1423, 2009.
David R Corey, Masad J Damha, and Muthiah Manoharan. Challenges and opportunities for nucleic acid therapeutics. nucleic acid therapeutics, 32(1):8-13, 2022.
Amber Dance et al. The hunt for red fluorescent proteins. Nature, 596(7870) 2021.
Suresh Dara, Swetha Dhamercherla, Surender Singh Jadav, CH Babu, and Mohamed Jawed Ahsan. Machine learning in drug discovery: a review. Artificial Intelligence Review, pp. 1-53, 2021.
Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hypervolume improvement for parallel multi-objective bayesian optimization. Advances in Neural Information Processing Systems, 33:9851 1-6644, 2020.
10

---

Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Parallel bayesian optimization of multiple noisy objectives with expected hypervolume improvement. Advances in Neural Information Processing Systems, 34:2187-2200, 2021.
Samuel Daulton, David Eriksson, Maximilian Balandat, and Eytan Bakshy. Multi-objective bayesian optimization over high-dimensional search spaces. In The 38th Conference Uncertainty in Arti- ficial Intelligence, 2022. URL https: : ooppnreviivi ew net/ forum? 5 9xq.
Matthias Ehrgott. Multicriteria optimization, volume 491. Springer Science & Business Media, 2005.
Michael TM Emmerich, André Deutz, and Jan Willem Klinkenberg. Hypervolume-based ex- pected improvement: Monotonicity properties and exact computation. In 2011 IEEE Congress of Evolutionary Computation (CEC), pp. 2147-2154 IEEE, 2011.
C.M. Fonseca, L. Paquete, and M. Lopez-Ibanez. An improved dimension-sweep algorithm for the hypervolume indicator. 2006 IEEE International Conference on Evolutionary Computation, pp. 1157-1163, 2006.
doi: 10.1109/CEC.2206.11588
Roman Garnett. Bayesian Optimization. Cambridge University Press, 2022. in preparation.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp. 1352-1361. PMLR, 2017.
Michael Pilegaard Hansen and Andrzej Jaszkiewicz. Evaluating the quality of upproximations the non-dominated set. Citeseer, 1994.
Conor Hayes, Roxana Rădulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. practical guide multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36(1):1-59) 2022.
Hisao Ishibuchi, Hiroyuki Masuda, Yuki Tanigaki, and Yusuke Nojima. Modified distance calculation in generational distance and inverted generational distance. In António Gaspar-Cunha, Carlos Henggeler Antunes, and Carlos Coello Coello (eds.), Evolutionary Multi-Criterion Optimization, pp. 120,125, Cham, 2015. Springer International Publishing. ISBN 978-3-319-15892-1
Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garcia. Jarrid Rector-Brooks, Bonaventure FP
Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning, pp. 1786-98001 PMLR, 2022.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Chapter 11. junction tree variational autoencoder for molecular graph generation. Drug Discovery, pp. 228-2499 2020. ISSN 2041-3211.
R.L. Keeney, H. Raiffa, K.R. L, and R.F. Meyer. Decisions with Multiple Objectives: Preferences and Value Trade-Offs. Wiley series in probability and mathematical statistics. Applied probability and statistics. Cambridge University Press, 1993. ISBN 9780521438834, URL https: /books.
google ca/books wtt.bboookadddPPeeee P1d=GPE2AAGCInCC..
Michael Kilgour, Tao Liu, Brandon Walker, Pengyu Ren, and Lena Simine, E2edna: Simulation protocol for dna aptamers with ligands. Journal of Chemical Information and Modeling, 61(9):
4139-4144, 4144, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http: //arxiv.org/ abs 2.... 6980.
Abdullah Konak, David W. Coit, and Alice E. Smith. Multi-objective optimization using genetic algorithms: tutorial. Reliability Engineering and System Safery, 91(9))992- 1007, 2006. ISSN 09518320. doi: 10.1016/j.ress 2005.
11

---

Ashutosh Kumar, Amout Voet, and Kam YJJ. Zhang. Fragment based drug design: from experimental to computational approaches. Current medicinal chemistry, 19(30):51285541 2012.
Greg Landrum. Rdkit: Open-source cheminformatics. URL http: httptt:::/wwww www.t..k.......
Xi Lin, Zhiyuan Yang, and Qingfu 7hang. Pareto set learning for neural multi-objeecivee combinatorial optimization. In International Conference on Learning Representations, 2021.
Wesley J Maddox, Maximilian Balandat, Andrew G Wilson, and Eytan Bakshy. Bayesian optimization with high-ding dingnniiensional outputs. Advances in Neural Information Processing Systems, 34:19274- 19287, 2021a.
Wesley J Maddox, Samuel Stanton, and Andrew G Wilson. Conditioning sparse variational gaussian processes for online decision-making. Advances in Neural Information Processing Systems, 34: 2021b.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. Neural Information Processing Systems (NeurIPS), 2022.
To appear.
Kaisa Miettinen. Nonlinear multiobjective optimization, volume 12. Springer Science & Business Media, 2012.
Santiago Miret, Vui Seng Chua, Mattias Marder, Mariano Phiellip, Nilesh Jain, and Somdeb Majum- dar. cnurrevolution-enhaneee multi-objective optimization for prixci-preciioon quantization. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1057-1065, 2022.
Volodymyr Mnih, Adria Puigdomenech Badia, Mchdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937. PMLR, 2016.
Panos M Pardalos, Antanas Zilinskas, Julius Zilinskas, et al. Non-convex oultcobee optimization. Springer, 2017.
Biswajit Paria, Kirthevasan Kandasamy, and Barnabás Póczos. flexible framework for multi- objective bayesian optimization using random scalarizations. In Uncertainty in Artificial Intelli- gence, pp. 766-776. PMLR, 2020.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7,
Mathieu Reymond, Eugenio Bargiacchi, and Ann Nowe. Pareto conditioned networks. In 2 International Conference on Autonomous Agents and Multi-agent System. IFAAMAS, 2022.
Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley, A survey of multi- objective sequential decision-making. Journal of Artificial Intelligence Research, 48:67-113, 2013.
Joost Schymkowitz, Jesper Borg, Francois Stricher, Robby Nys, Frederic Rousseau, and Luis Serrano. The foldx web server: an online force field. Nucleic acids research, 33(suppl 2):W322-W388 2005.
Amar Shah and Zoubin Ghahramani. Pareto frontier learning with expensive correlated objectives. In International conference on machine learning, pp. 1919-1927. PMLR, 2016.
Andrew Shrake and John A Rupley. Environment and exposure to solvent of protein atoms. lysozyme and insulin. Journal of molecular biology, 1922)3351-371, 1973.
Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Accelerating Bayesian optimization for biological sequence design with denoising autoencoders. Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning. volume 162 Proceedings of Machine Learning Research, pp. 20459 20478. PMLR, 17-23 Jul 2022. URL https: /proceed ngs ml press / v1 62. stanton22a html.
12

---

Oleg Trott and Arthur Olson. Autodock vina: improving the speed and accuracy of docking with a new scoring function, efficient optimization, and multithreading. Journal of computational chemistry, (1(2):4554411 2010.
Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of pareto dominating policies. The Journal of Machine Learning Research, 15(1):3483-351, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Ronald J Williams. Simple statistical moddeent-folloingg algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-2555 256, 1992.
Yutong Xie. Chence Shi, Hao Zhou, Yuwei Yang. Weinan Zhang, Yong Yu, and Lei Li. (MARS): Markov molecular sampling for multi-objective drug discovery. In International Confer- ence on Learning Representations, 2021. URL /ttpps://operreview wwt.forutttoooooooeoooooooo forum? - kHSu4ebxFXX.
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. generalized algorithm for multi-ohjective reinforcement learning and policy adaptation. Advances in Neural Information Processing Systems, 32, 2019.
Joseph D Yesselman, Daniel Filer, Erik D Carlson, Michael R Gotrik, Anne E d' Aquino, Alexandra N Ooms, Wipapat Kladwang, Paul D Carlson, Xuesong Shi, David A Costantino, et al. Computational design of three-dimensional ma structure and function. Nature nanotechnology. 14(9):866-873, 2019.
Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo Kim. Graph transformer networks. Advances in neural information processing systems, 32, 2019.
Joseph N Zadeh, Conrad D Steenberg, Justin S Bois, Brian R Wolfe, Marshall B Pierce, Asif R Khan, Robert M Dirks, and Niles A Pierce. Nupack: Analysis and design of nucleic acid systems. Journal of computational chemistry, (((1):7701177 2011.
Richard Zhang and Daniel Golovin. Random hypervolume scalarizations for provable multi-objective black box optimization. In International Conference on Machine Learning, pp. 11096-11105. PMLR, 2020.
Shuo Zhang, Yang Liu, and Lei Xie. Molecular mechanics-driven graph neural network with multiplex graph for molecular structures, 2020. URL
Yiyang Zhao, Linnan Wang, Kevin Yang, Tianjun Zhang, Tian Guo, and Yuandong Tian. Multi- objective optimization by learning space partition. In International Conference on Learning Representations, 2022. URL https: cectps://opentee //tp///wepi.....c/////e/e. n wwee:/ooummndd-w2225/ee
Wenhu Zhou, Runjhun Saran, and Juewen Liu. Metal sensing by dna. Chemical reviews, 117(12): 2017.
Marcela Zuluaga, Guillaume Sergent, Andreas Krause, and Markus Püschel. Active learning for multi-objective optimization. In International Conference on Machine Learning, pp. 462-470. PMLR, 2013.
13

---

A ALGORITHMS
We summarize the algorithms for MOGFN-PC and MOGFN-AL here.
Algorithm 1: Training preference-conditiona GFlowNets
Input:
p(w): Distribution for sampling preferences;
3: Reward Exponent;
6: Mixing Coefficient for uniform actions in sampling policy;
N: Number of training steps:
Initialize:
(8)(,)),,,), Pn(s)s', w), log Zww)): Conditional GFlowNet with parameters 0:
for i
= to N do
Sample preference w p(w):
Sample trajectory 7 following policy म = (1 Pr + OUniform ;
Compute reward [( for generated samples and corresponding loss L(T, w 0) in
Equation 2;
Update parameters 0 with gradients from the loss, VoL(T,w);
end
Algorithm 2: Training MOGEN-AL
Input:
R = [R,,... Raj: Oracles to evaluate candidates x and return true objectives
(Ri(z).... Rx)=)):
Do {(.,, }: Initial dataset with yı R(x;);
f: Probabilistic surrogate model to model posterior over R given a dataset D;
a(xlf): Acquisition function computing scalar utility for given f:
no: Learnable GFlowNet policy;
b: Size of candidate batch to be generated;
N: Number of active learning rounds;
Initialize:
f. mp:
for i == to N do
Fit f on dataset D;-1;
Extract the set of non-dominated candidates P:-1 from Di-1;
Train my with to generate mutations for P: using a(-lf) as the reward;
Generate batch
{ai.m........ Th by sampling xf from P._.. and applying to mutations m, sampled from we:
Evaluate batch B with R generate D. ((x1, R(x))) ...,, (m,, R(x.));
Update dataset D; = D, U D
end
Result:
Approximate Pareto set PN
B SCALARIZATION
Scalarization a popular approach for tackling multi-objective optimization problems. MOGFN- PC can build upon any scalarization approach. We consider three choices. Weighted-sum (WS) scalarization has been widely used in literature. WS finds candidates on the convex hull of the Pareto front (Ehrgott, 2005). Under the assumption that the Pareto front is convex, every Pareto optimal solution is solution to a weighted sum problem and the solution to every weighted sum problem Pareto optimal. Weigthed Tehebycheff (WT), proposed by Choo & Atkins (1983) is
14

---

an alternative designed for non-convex Pareto fronts. Any Pareto optimal solution be found by solving the weighted Tchebycheff problem with appropriate weights, and the solutions for any weights correspond to weakly Pareto optimal solution of the original problem (Pardalos et al., 2017). Lin et al. (2021) deomstrated through their empirical results that WT can be used with neural network based policies. The third scheme consider, Weighted-log-sun (WL.) has not been considered in prior work. We hypothesized that in some practical scenarios, we might want to ensure that all objectives are optimized, since, for instance, WS the scalarized reward can be dominated by single reward. WL, which considers the weigthed sum in log space can potentially help with this drawback. However, as discussed in Section in practice WL can be hard to optimize, and lead to poor performance.
C ADDITIONAL ANALYSIS
Table 6: Fragment-based molecule generation: Analysing the impact of a, and R(xlww) on the performace of MOGFN-PC
Metrics
Effect of p(w)
Effect of
Choice of R(z w)
Dir(0.1)
Dir(1)
Dir(10)
16
32
48
96
WS
WL
WT
Reward (t)
0,58 0.89
0.51 0.44
0,55
0.89
Diversity (1)
2.75
0,75
2006
0.86
0.85
0.75
0.75
0.82
0.10
Hypervolume (1)
0.08
0.82 0.90
0.59
0.55
0.60
0.90
0.90
0.53
0.71
12 (L) n.4
0.86
1.93
n.7
4.46
3.64
1.86
1.86
6.92
11.51
Can MOGFN-PC match Single Objective GFNs? To evaluate how well MOGFN-PC models the family of rewards R(x]w), we consider comparison with single objective GFlowNets. More specifically, we first sample a set of 10 preferences WI, W10, and train standard single objective GFlowNet using the weighted sum scalar reward for each preference. We then generate N 128 candidates from each GFlowNet, throughout training, and compute the mean reward for the top 10 candidates for each preference. We average this top 10 reward across fwi,..., w10), and call R.o- We then train MOGFN-PC, and apply the sample procedure with the preferences (wi...., W10 and call the resulting mean of top 10 rewards Rmo. We plot the value of the ratio Rmo/ R2o in Figure We observe that the ratio stays close to I, indicating that MOGFN-PC can indeed model the entire family of rewards simultaneously least fast single objective GFlowNet could.
1.2
Average Reward Ratio
Average Reward Ratio
1.4
1.1
1.2
1.0
1.0
0.9
0.8
0.8
0.6
0.0
2.5
5.0
7.5
10.0
0.00
0.25
0.50
0.75
1.00
Points seen (x1000)
Points seen (xM)
(a) 3 Bigrams task
(b) Fragment-based Molecule Generation Task
Figure 3: We plot the ratio of rewards Rma/ R.. for candidates generated with MOGFN-PC (Rm.) and single-objective GFlowNets Rso) for a set of preferences in the (a) Bigrams and (b) Fragment- based molecule generation tasks. We observe that MOGFN-PC matches and occasionally surpasses single objective GFlowNets
15

---

Efffec of Model Capacity and Architecture Finally we look at the effect of model size in training MOGFN-PC. As MOGFN-PC models a conditional distribution, an entire family of functions as we' ve described before, expect capacity play a crucial role since the amount of information to be learned higher than for a single-objective GFN. We increase model size in the Bigrams task to see that effect, and see in Table 7 that larger models do help with performance-althen although the performance plateaus after a point. We suspect that in order to fully utilize the model capacity we might need better training objectives.
Table 7: Analysing the impact model size on the performance of MOGFN-PC. Each architecture choice for the policy is denoted as A-B-C where A is number of layers, B is the number of hidden units in each layer, and C is the number of attention heads.
Metrics
Effect of model size
3-64-8
3-256-8
4-64-8
4-256-8
Reward (1)
0.44_gn.01
0.47
0.51 1000
Diversity (1)
19.79220.00
17.13
17.53
16. 16.12.0004
Hypervolume (1)
0.22
0.205.20.008
0.262
0.270 =0.011
R2 (L)
9.97
9.22
8.95
8.91 :0.0
D METRICS
In this section we discuss the various metrics that we used to report the results in Section 5.
1. Generational Distance Plus (GD +) (Ishibuchi et al., 2015): This metric measures the euclidean distance between the solutions of the Pareto approximation and the true Pareto front by taking the dominance relation into account. To calculate GD+ we require the knowledge of the true Pareto front and hence we only report this metric for Hypergrid experiments (Section 5.1.1)
2. Hypervolume (HV) Indicator (Fonseca et al., 2006): This is standard metric reported in MOO works which measures the volume in the objective space with respect to a reference point spanned by set of non-dominated solutions in Pareto front approximation.
3. R2 Indicator (Hansen & Jaszkiewicz, 1994): R2 provides monotonic metric comparing two Pareto front approximations using a set of uniform reference vectors and utopian point
z representing the ideal solution of the MOO.
This metric provides monotonic reference compare different Pareto front approxi- mations relative to a utopian point. Specifically, we define a set of uniform reference vectors À A that cover the space of the MOO and then calculate:R, (T A, 2*)
[A] ΣxEA min.err ma. (A,l2 - -11 where y E I corresponds the set of
1
solutions in given Pareto front approximations and =" is the utopian point corresponding the ideal solution of the MOO. Generally, Rg metric calculations are performed with =" equal to the origin and all objectives transformed to minimization setting. which serves to preserve the monotonic nature of the metrie. This holds true for our experiments as well.
4. Top-K Reward This metric was originally used (Bengio et al., 2021a), which we extend for our multi-objective setting. For MOGFN-PC. we sample N candidates per test preference and then pick the top-k candidates (A: < N) with highest scalarized rewards and calculate the mean. We repeat this for all test preferences enumerated from the simplex and report the average top-k reward score.
5. Top-K Diversity This metric was also originally used in (Bengio et al., 2021a), which we again extend for diversity of the generated candidates. Given a distance metric d(æ, 3) between candidates x and y we calculate the diversity of candidates as those who have d(x y) greater than
setting. We this metric quantify the notion of
16

---

Here we elaborate on the Hyper-Grid experimental setup which we discussed in Section 5.1.1. Consider n-dimensional hypercube gridworld where each cell in the grid corresponds state. The agent starts at the top left coordinate marked as (0.0,...) and is allowed to move only towards the right, down, or stop. When the agent performs the stop action, the trajectory terminates and the agent receives a non-zero reward. In this work, we consider the following reward functions threshold For MOGFN-PC, sample N candidates per test preference and then pick the top-k candidates based on the diversity scores and take the mean. We repeat this for all test preferences sampled from simplex and report the average top-k diversity score. We the edit distance for sequences, and minus the Tanimoto similarity for molecules.
E ADDITIONAL EXPERIMENTAL DETAILS
E.1 HYPER-GRID
brannin (x), currin (x), sphere (x), shubert (x). beale (x). Figure 4, we show the heatmap for each reward function. Note that we normalize all the reward functions between and
Branin
0
0.8
10
10
0.6
X
20
20
20
0.4
20
20
20
20
02
X1
0.0
Figure Reward Functions Different reward function considered for HyperGrid experiments presented in Section 5.1.1. Here the grid dimension is H 32
Additional Results To verify the efficacy of MOGFNs across different objectives sizes, we perform some additional experiments and measure the L, loss and the GD+ metric. In Figure 5, that as the reward dimension increases, the loss and GD + increases. This is expected because the number of rewards is indicative of the difficulty of the problem. We also present extended qualitative visualizations across more preferences in Figure 6.
0,00015
onn Distamce
0.20
0.25
0.5.
0,000005
0.08
0,00000
0.00
2
No. of Objectives
No. Objertivas
Figure 5: (Left) Average test loss between the MOGFN-PC distribution and the true distribution for increasing number objectives. (Right) GD + metries of MOGFN-PC across objectives.
Model Details and Hyperparameters For MOGFN-PC policies we use an MLP with two hidden layers each consisting 64 units. We use LeakyReLU as our activation function as Bengio et al. (2021a). All models are trained with learning rate=0. 01 with the Adam optimizer Kingma Ba (2015) and batch size=128. We sample preferences w from Dirichlet(a) where a = 1.5. We try two encoding techniques for encoding preferences 1) Vanilla encoding where just use the raw values the preference vectors and 2) Thermometer encoding (Buckman et al., 2018). In our experiments have observed significant difference in performance difference.
17

---

Truah Dist
[..n.0.o.11
0.0025
00.0000
0.0015
00010
000005
0.0000
Figure 6: Extended Qualitative Visualizations for Hypergrid epxeriments
E.2 N-GRAMS TASK
Task Details The task is to generate sequences of some maximum length L, which we set to 36 for the experiments in Section 5.1.2. We consider vocabulary (actions) of size 21, with 20 characters ["A", "R", "N", "D", "C", "E." "Q", "G", "H" "I", "I,", "K" "M", "F", "p", "5", "T", "8", "Y", "y"] and special token to in- given
dicate the end of sequence. The rewards { are defined by the number of occurrences set of n-grams in sequence z. For instance, consider ("AB", "BA"] as the n-grams. The rewards for a sequence r = ABABC would be [2, 1]. We consider two choices of n-grams: (a) Unigrams: the number of occurrences of a set of unigrams induces conflicting objectives since the number of occurrences of a monogram without replacing another a string of a particular length, (b) Bigrams: given common characters within the bigrams, the occurrences of multiple bigrams can be increased simultaneously within string of a fixed length. We also consider different sizes for the set n-grams considered, i.c. different number of objectives. This allows us evaluate the behaviour of MOGFN-PC on variety of objective spaces. We summarize the specific objectives used in our experiments in Table We normalize the rewards to [0, 1] in our experiments.
cannot increase
Table 8: Objectives considered for the N-grams task
Objectives
n-grams
2 Unigrams
["A", "C"]
Bigrams
["AC", "CV"] ["A", "C", "V"]
Unigrams
Bigrams
["AC", "CV", "VA"]
4 Unigrams
["A", "C" "V", "W"]
Bigrams
["AC", "CV", "VA", "AW"]
Model Details and Hyperparamettrs We build upon the implementation from Stanton et al. (2022) for the task: https / git hithuuu..................... com. amuel st ant on / ambo. For the string generation task, the backward policy Pg is trivial (as there is only one parent for each node s S), so we only have to Conditional Transformer encoder as the architecture. This consists of a Transformer encoder ( (Vaswani et al., 2017) with hidden layers of dimension 64 and 8 attention heads embed the current state (string generated so far) We have an MLP which embeds the preferences w which are encoded using thermometer encoding with 50 bins. The embeddings of the state and preferences MLP which generates categorical distribution over the actions (vocabulary token). We use the same architecture for the baselines using a conditional policy MOReinforce and MOSoftQL. For Envelope- MOQ, which does not condition on the preferences, we use a standard Transformer-encoderr with similar architecture. We present the hyperparameters we used in Table Fach method is trained for 10,000 iterations with minibatch size of 128. For the baselines we adopt the official implementations released by the authors for MOReinforce - https: //tp://gttttt com Xi- L/PMOCO and Envelope- MOQ https:tppp....... / thub. com Rtt////uuneeeanngaee /MORL.
parameterize Pr and log Z. As (p(-1s,w) is conditional policy,
concatenated and passed to a final
We present some additional results for the n n-grams task. We consider different number of objectives d E {2, 4} in Table 10 and Table 11 respectively. As with the experiments in Section 5.1.2 we observe that MOGFN-PC outperforms the baselines in Pareto performance while achieving high diversity scores. In Table 12, we consider the case of shorter sequences I. 24.
Additional Results
18

---

Table 9: Hyperparamelers for N-grams Task
Hyperparameter
Values
Learning Rate (Pp)
{0.01, 0.05, 0.001, 0.005, 0.0001} (0.01, 0.05, 0.0013
Learning Rate (Z)
Reward Exponent: /
(16, 32, 48) (0.01, 0.05, 0.13
Uniform Policy Mix: 5
MOGFN-PC continues to provide significant improvements over the baselines. There are two trends we can observe considering the N-grams task holistically:
1. As the sequence size increases the advantage of MOGFN-PC becomes more significant.
2. The advantage of MOGEN-PC increases with the number objectives.
Table 10: N-grams Task. 2 Objectives
Algorithm
Bigrams
Unigrams
Reward (1)
Diversity (t)
(t)
(1)
Reward (f)
Diversity (1)
(t)
Ra (
Envelope-MOQ
0.05 z0001
0gn
0.0go0
Ogan
0.09
0.o
0.014
5.73
MOReinforee
0.12
0g
0.151 20023
0.031
0.43
00
0.222
2.54
MOSofiQL
19.40
0.247
2.92
0.46
22.05
0.253
2.54_oo
MOGEN-TB
0.51 g.04
20.65 20.88
0.321 20011
2.31
0.48.2001
22.15
0.267 20:00
2.24
Table 11: N-grams Task. 4 Objectives
Algorithm
Bigrams
Unigrams
Reward (1) Diversity
HV
Rg (L)
Reward (1) Diversity
HV (f)
(1)
Envelope-MOQ
Ogo
0go
0.o
85.23
0go
0ge
0.o
80.36
MOReinforce
20.00
0go
#00001
60.42.1.52
0.00
0gn
0.e
79.12
MOSoftQL
2.12
24.32 1:21
0.013 go00
39.31
0.22
24. 21.4
0.019 2000
31.46g232
MOGEN-TB
0.23
20.31
0.055
24.42
0.33
23.24
0.063:a0022
23.31
19

---

Table 12: N-grams Task. Shorter Sequences
Algorithm
Bigrams
Unigrams
Reward (1)
Diversity (1)
HV (1)
R2 (4)
Reward (1)
Diversity (f)
HV (t)
Ra (U)
Envelope-MOQ
0.07
0.
0.027 100000
0.08 16.21
0go
20001
20.13
MOReinforce
0.18
0g
g.053
g3.35
0002
0go
0.041
19.25
MOSoftQL
0.31 go00
20. g281
0.143
12.79.0.1
0.38 g002
21.13 20:3
g0om
12.12
MOGEN-PC
g.45
19.62
0.225 40,009
9.82
0.39
21.94.021
0.125
10
E.3
QM9
Reward Details As mentioned in Section 5.2.1, 1, consider four reward functions for our experiments. The first reward function is the HUMO-LUMO gap, for which we rely on the predictions of a pretrained MXMNet (Zhang et al., 2020) model trained on the QM9 dataset (Ramakrishnan al., 2014). The second reward is the standard Synthetic Accessibility score which we calculate using the RDKit library (Landrum), to get the reward we compute (10 SA)/9. The third reward function molecular weight target. Here we first calculate the molecular weight of a molecule using RDKit, and then construct a reward function of the form e (mmmlm==------- 1000- /500 which is maximized at 105, Our final reward function is a logP target, e (lopp-2.5) which is again calculated with RDKit and is maximized at 2.5.
Model Details and Hyperparameters We sample new preferences for every episode from a Dirichlet(a). and encode the desired sampling temperature using thermometer encoding (Buck- man et al., 2018). We use a graph neural network based on a graph transformer architecture (Yun al., 2019). We transform this conditional encoding to embedding using an MLP. The embedding is then fed to the GNN as a virtual node, as well as concatenated with the node embeddings in the graph. The model's s action space is add new node the graph, a new bond, or set node or bond properties (like making bond double bond). It also has a st stop action. For more details please refer the code provided in the supplementary material. We summarize the hyperparameters used in Table 13.
Hyperparameter
Value
Learning Rate (Pp)
0.0005
Learning Rate (Z)
0.0005
Reward Exponent: 3
32
Batch Size:
64
Number of Embeddings 64
Uniform Policy Mix:
0.001
Number of layers
4
Table 13: Hyperparameters for QM9 Task
E.4 FRAGMENTS
More Details As mentioned Section 5.2.2, we consider four reward functions for our experiments. The first reward function is proxy trained on molecules docked with Autodock (Trott & Olson, 2010) for the sEH target; we use the weights provided by Bengio et al. (2021a). We also use synthetic accessibility, for QM9, and a weight target region (instead of the specific target weight used for QM9), ( (300 molwt) / 700 1) clip (0, 1) which favors molecules with a weight of under 300. Our final reward function is QED which is again calculated with RDKit.
Model Details and Hyperparameters We again use a graph neural network based on a graph transformer architecture (Yun et al., 2019). The experimental protocol is similar QM9 experiments discussed in Appendix E.3. We additionally sample from a lagged model whose parameters are updated 0' 70' + (1 7)0. The model's action space is to add node, by choosing from
20

---

list of fragments and an attachment point the current molecular graph. We list all hyperparameters used Table 14.
Hyperparameter
Value
Learning Rate (Pr)
0.0005
Learning Rate (Z)
0.0005
Reward Exponent
96
Batch Size:
256
Sampling model
0.95
Number of Embeddings
128
Number of layers
Table 14: Hyperparameters for Fragments
Additional Results We also present Figure view of the reward distribution produced by MOGFN-PC. Generally, the model is able to find good caa-PPartt-optiiaall samples, but is also able to spend time exploring. The figure also shows that the model able to respect the preference conditioning, and remains capable of generating diverse distribution rather than single point.
proference
54 SA
sih
sah
Rcd
qed
qed
preference
sah
5A
sah
wt
wt
wt
Figure 7: Fragment-based molecule generation: See Appendix E.4.
In the off-diagonal plots of Figure 7, show pairwise scatter plots for each objective pair; the Pareto front is depicted with a red line; each point corresponds to a molecule generated by the model it explores the state space; color is density (linear viridis palette). The diagonal plots show two overlaid informations: blue histogram for each objective, and an orange scatter plot showing the relationship between preference conditioning and generated molecules. The effect of this conditioning is particularly visible for seh (top left) and wt. (bottom right). As the preference for the sEH binding reward gets closer I, the generated molecules' reward for sEH gets closer to I as well. Indeed, the expected shape for such scatter plot is triangular-ish shape: when the preference close to 1, the model is expected to generate objects with a high reward for R,; as the preference wi gets further away from the model can generate anything, including objects with high -hathat is, unless there is trade off between objectives, in which case in cannot; this the case for the seh objective, but not for the wt objective, which has more triangular shape.
for reward R;
21

---

E.5 DNA SEQUENCE DESIGN
Task Details The set of building blocks here consists of the bases ["A", "C", "T", "G"] in addition to special end of sequence token. In order compute the free energy and number of base with the software NUPACK (Zadeh et al., 201 1), we used 310 K as the temperature. The inverse of 30 the length L objective was calculated as I as 30 was the minimum length for sampled sequences. The rewards are normalized to [0, 1] for our experiments.
Model Details and Hyperparameters We use the same implementation as the N-grams task, detailed in Appendix E.2. Here we consider 4-layer Transformer architecture, with 256 units per layer and 16 attention head instead. We detail the most relevant hyperparameters Table 15.
Table 15: Hyperparameters tuned for DNA-Aptamers Task.
Hyperparameter
Values
Learning Rate (Pp)
{0.001, 0.0001, 0.00001, 0.000001]
Learning Rate (Z)
0.001
Reward Exponent: 3
{40, 60, 80}
Batch Size:
16
Training iterations:
10,000
Dirichlet
(0.1, 1.0, 1.5)
Discussion of Results Contrary to the other tasks which we evaluated MOGEN-PC, for the generation of DNA aptamer sequences, our proposed model did not match the best baseline, multi- objective reinforcement learning (Lin et al., 2021), in terms of Pareto performance. Nonetheless, it is worth delving into the details in order to better understand the different solutions found by the two methods. First, as indicated in section 5, despite the better Pareto performance, the best sequences generated by the RL method have extremely low diversity (0.62), compared to MOGEN, which generates optimal sequences with diversity of 19.6 or higher. As a matter of fact, MOReinforce mostly samples sequences with the well- pattern GCGC for all possible lengths. Sequences with this pattern have indeed low (negative) energy and many number of pairs, but they offer little new insights and poor diversity if the model is not able to generate sequences with other distinct patterns. On the contrary, GFlowNets are able to generate sequences with patterns other than repeating the pair of bases G and C. Interestingly, we observed that GFlowNets were able to generate sequences with even lower energy than the best sequences generated by MOReinforce by inserting bases A and T into chains of GCGC
Finally, observed that one reason why MOGFN does not match the Pareto performance of MOReinforce is because for short lengths (one of the objectives) the energy and number of pairs are not successfully optimised. Nonetheless, the optimisation of energy and number of pairs is very good for the longest sequences. Given these observations, we conjecture that there is room for improving the set hyperparameters or certain aspects of the algorithm.
Additional Results In order to better understand the impact of the main hyperparameters of MOGEN-PC the Pareto performance and diversity of the optimal candidates, we train multiple instances by sweeping over several values of the hyperparameters, present the results in Table 16. One key observation is that there seems to be a tradcoff between the Pareto performance and the diversity of the Top-K sequences. Nonetheless, even the models with the lowest diversity are able to generate much more diverse sequences than MOReinforce. Furthermore, we also observe as well as higher 3 (reward exponent), both yield better metrics of Pareto performance but slightly worse diversity. In the case of {3, this observation consistent with the results in the Bigrams task (Table 5), but with Bigrams, best performance was obtained with a = 1. This is indicative of degree of dependence on the task and the nature of the objectives.
indicated in Table 15. We
1 as the parameter of the Dirichlet distribution to sample the weight preferences,
E.6 ACTIVE LEARNING
Task Details We consider the Proxy RFP task from Stanton et al. (2022). an in silico benchmark task designed to simulate searching for improved red fluorescent protein (RFP) variants (Dance et al., 2021). The objectives considered stability (-dG or negative change in Gibbs free energy) and
22

---

Table 16: Analysis of the impact of a, 8 and the learning rate on the performance of MOGFN-PC for DNA sequence design. We observe trade-off between the Top-K diversity and the Pareto performance.
Metrics
Efffet p(u)
Effect of
Effect of the learning
Dirto 0.1) Din(o 1) Dir(a
1.5)
40
60
80
10-3
10-4
10-3
10
Reward (f)
0.687_0o)
0.652
g.50
g.560
0.652
0.587 2001
0.652g001
0.654gom
0.604.00)
Divensity (1)
17.65 2m
19
20.18
28.49 30
g019
en 15
50
19
19.51 g114
g3
Hypersolume (1)
0.506
0,467
0.277
0.363
0.467
0.333
0.467
0.496
0.336
R₁ (
2.462
2.576
2.688 Loic
4.225
2,905
2.576
3.855
2.576
2,488
3:422
solvent-accessible surface area (SASA) (Shrake & Rupley, 1973) in simulation, computed using the FoldX suite (Schymkowitz al., 2005) and BioPython (Cock et al., 2009). We use the dataset introduced in Stanton et al. (2022) as the initial pool of candidates Do with [Dol 512.
Method Details and Hyperparameters Our implementation builds upon the publicly released code from (Stanton et al., 2022): https: pettps://gtthlte com/ samuelstant lambo. We follow the exact experimental setup used in Stanton et al. (2022). The surrogate model f consists of an encoder with ID convolutions (masking positions corresponding to padding tokens). We used standard pre-activation residual blocks with two convolution layers, layer norm, and swish activations, with a kernel size of 5, 64 intermediate channels and 16 latent channels. A multi-task GP with an ICM kernel is defined in the latent space of this encoder, which outputs the predictions for each objective. We also use the training tricks detailed Stanton et al. (2022) for the surrogate model. The hyperparameters, taken from Stanton al. (2022) shown in Table 17. The acquisiton function used is NEHVI (Daulton et al., 2021) defined as
N Σ
1 N + N HUNC (3)
a((r;))
Σ
N
where Fe,t = I....N are independent draws from the surrogate model (which is posterior over functions), and Pe denotes the Pareto frontier in the current dataset under fc.
Table 17: Hyperparameters for training the surrogate model f
Hyperparameter
Value
Shared enc. depth (# residual blocks)
3
Disc. enc. depth residual blocks) Decoder depth (# residual blocks)
3
Conv. kernel width (# tokens)
# conv. channels
64
Latent dimension
16
GP likelihood variance init
0.25
GP lengthscale prior
N(0.7, 0.01)
# inducing points (SVGP head)
64
DAE corruption ratio (training)
0.125
DAE learning rate (MTGP head)
5.00E-03
DAE learning rate (SVGP head)
1.00E-03
DAE weight decay
1.00E-04
Adam EMA params
0., le-2
Early stopping holdout ratio
0.1
Early stopping relative tolerance
1.00E-03
Early stopping patience (# epochs)
32
Max # training epochs
256
We replace the LaMBO candidate generation with GFlowNets. We generate set of mutations m [11,, v:) for sequences from the current approximation of the Pareto front P.. Note
23

---

that, opposed to the sequence generation experiments, Py here is not trivial as there mul- tiple ways (orders) of generating the set. For our experiments, we use a uniform random Pg- Pr takes input the sequence with the mutations generated former encoder with 3 layers, with hidden dimension 64 and 8 attention heads as the architec- ture for the policy. The policy outputs distribution over the locations in x, {1...., |a]] }. and a distribution over tokens for each location. The vocabulary of actions here is the same as the N-grams task ["A", "R", "N", "D", "c", "E", "Q", "G", "H", "I" "L", "x", "M", "F", "P", "S", "T", "w", "Y", "V"]. The logits of the locations of the mutations generated so far are set to 10000, to prevent generating the same sequence. The acquisition function(NEHV] value for the mutated sequence is used as the reward. We also use reward exponent 3. To make optimization easier (as the acquisition function becomes harder to optimize with growing (3), we reduce 3 linearly by a factor 6, at each round. We train the GFlowNet for 750 iterations in each round. Table 18 shows the MOGFN-A hyperparamcters. The active learning batch size is 16, and we run 64 rounds of optimization. Table 18 presents the hyperparameters used for MOGEN-AL.
far applied. We use Trans-
Table 18: Hyperparameters for MOGFN-AL
Hyperparameter
Values
Learning Rate (Pp)
(0.01, 0.001, 0.0001
Learning Rate (Z)
(0.01, 0.001)
Reward Exponent |
116 24]
Uniform Policy Mix: 5
(0.01, 0.05)
Maximum number of mutations
10, 15, 20}
83
(0.5, 1,
24


Reviews:
[{"review_id": "lyaiX7Q2tB", "reviewer": "Reviewer_sp4N", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper leverages GFlowNet to solve the Multi-Objective Optimization problem. The authors derive two versions of the algorithm, regular and active learning variants. The authors empirically demonstrate that the proposed algorithms outperform existing methods almost on every considered benchmark and perform reasonably on DNA sequence design tasks. The method as expected performs very well in terms of generating a diverse set of candidates.", "summary_of_the_review": "In general, I believe the novelty of the paper is incremental as it is a straightforward adaptation of  GFlowNets to MOO settings. However, this adaptation may possess a solid technical contribution. \n", "strength_and_weaknesses": "Strength\n\n* The authors successfully applied conditional GFlowNet (for the first time) to the Multi-Objective Optimization problem.\n* The author generalizes the GFlowNet active learning algorithm for the Multi-Objective Optimization problem case. \n* The results seem promising for both algorithms in terms of key metrics and especially the diversity of the generated candidate.\n* The results and comparisons with other methods are well explained. (e.g. why in some cases other baselines have higher scores on some of the metrics)\n\nWeaknesses\n* I believe the novelty is limited in terms of new ideas as the main technical contribution of the paper is the adaptation of the previously derived method to MOO setting. \n\nMinor issues and questions\n* To make the paper self-consistent can you add how \\alpha function in the Active Learning section combines a reward with an epistemic uncertainty?\n* It is unclear to me from the paper how \\alpha function map a set of rewards and their epistemic uncertainties to a single objective in each round of the Active Learning pipeline. Or is the \\alpha function defined externally for each round and can be considered as a hyperparameter? \n* Do you retrain GFlowNet from scratch for each round of Active Learning?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "klPBTdTLre", "reviewer": "Reviewer_Qeid", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends the GFlowNets proposed for single-objective optimization to multi-objective optimization, namely MOGFNs. Experiments are conducted on molecule generation and sequence generation to study the effectiveness of the proposed method.", "summary_of_the_review": "This paper proposes a new method for multi-objective optimization. However, the advantages and effectiveness of the proposed method are not well described and evaluated. ", "strength_and_weaknesses": "Strengths\n1.\tThis paper extends the GFlowNets proposed for single-objective optimization to multi-objective optimization. \n2.\tAnalyses are conducted to study the main components of MOGFNs. \n\nWeaknesses\n1.\tThe goal of the research in this paper is not clear. How is diversity defined if it is not to cover the Pareto front of a multi-objective optimization problem?\n2.\tThe proposed algorithm is mainly based on the assumption that a multi-objective optimization problem can be viewed as a family of single-objective problems defined by non-negative weighted vectors. This might be inappropriate when the Pareto front is not convex. \n3.\tSimilar to this paper, decomposition-based multi-objective evolutionary algorithms such as MOEA/D solve multi-objective optimization problems by considering a set of single-objective sub-problems. The advantages of the proposed method over such methods need to be discussed and evaluated.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NJKap7xYDM", "reviewer": "Reviewer_MZeU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends the simple linear scalarization method in multi-objective optimization to the GflowNets setting. Two variants of GflowNets are proposed and extensive empirical experiments are provided, which verifies the effectiveness of them.", "summary_of_the_review": "More technical depth of MOO should be incorporated. The technical contribution of extending MOO to GlowNets seems to be limited. The motivation of the study should be more precisely described.", "strength_and_weaknesses": "Strengths:\n+ The idea of extending gflow nets to the multi-objective setting is novel and interesting.\n+ Multi-objective gflow nets are useful in drug discovery.\n\nWeaknesses:\n+ The extension of weighted sum scalarization to glow nets seems incremental, which lacks novelty.\n+ The understanding of multi-objective of this paper is limited. Some of the assertions are overclaimed. For example, weighted sum scalarization is only one simple way to do multi-objective optimization (MOO). It has many drawbacks, especially when non-convex Pareto front is encountered. But in Section 2.1, the authors claim solving weighted sum scalarization equals solving MOO. More precise background knowledge on MOO should be incorporated.\n+ The study of the active version of gflow nets is not very clearly motivated.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject