Title: Multi-Fidelity Active Learning with GFlowNets
Venue: NeurIPS 2023
Accepted: no
Keywords: gflownets

multi-fidelity

active learning

bayesian optimization

scientific discovery

biological sequence design

molecular modelling

material discovery
Abstract: In the last decades, the capacity to generate large amounts of data in science and engineering applications has been growing steadily. Meanwhile, the progress in machine learning has turned it into a suitable tool to process and utilise the available data. Nonetheless, many relevant scientific and engineering problems present challenges where current machine learning methods cannot yet efficiently leverage the available data and resources. For example, in scientific discovery, we are often faced with the problem of exploring very large, high-dimensional spaces, where querying a high fidelity, black-box objective function is very expensive. Progress in machine learning methods that can efficiently tackle such problems would help accelerate currently crucial areas such as drug and materials discovery. In this paper, we propose the use of GFlowNets for multi-fidelity active learning, where multiple approximations of the black-box function are available at lower fidelity and cost. GFlowNets are recently proposed methods for amortised probabilistic inference that have proven efficient for exploring large, high-dimensional spaces and can hence be practical in the multi-fidelity setting too. Here, we describe our algorithm for multi-fidelity active learning with GFlowNets and evaluate its performance in both well-studied synthetic tasks and practically relevant applications of molecular discovery. Our results show that multi-fidelity active learning with GFlowNets can efficiently leverage the availability of multiple oracles with different costs and fidelities to accelerate scientific discovery and engineering design.

Abstract

In the last decades, the capacity to generate large amounts of data in science and engineering applications has been growing steadily. Meanwhile, the progress in machine learning has turned it into a suitable tool to process and utilise the available data. Nonetheless, many relevant scientific and engineering problems present challenges where current machine learning methods cannot yet efficiently leverage the available data and resources. For example, in scientific discovery, we are often faced with the problem of exploring very large, high-dimensional spaces, where querying a high fidelity, black-box objective function is very expensive. Progress in machine learning methods that can efficiently tackle such problems would help accelerate currently crucial areas such as drug and materials discovery. In this paper, we propose the use of GFlowNets for multi-fidelity active learning, where multiple approximations of the black-box function are available at lower fidelity and cost. GFlowNets are recently proposed methods for amortised probabilistic inference that have proven efficient for exploring large, high-dimensional spaces and can hence be practical in the multi-fidelity setting too. Here, we describe our algorithm for multi-fidelity active learning with GFlowNets and evaluate its performance in both well-studied synthetic tasks and practically relevant applications of molecular discovery. Our results show that multi-fidelity active learning with GFlowNets can efficiently leverage the availability of multiple oracles with different costs and fidelities to accelerate scientific discovery and engineering design.

1 Introduction

The current most pressing challenges for humanity, such as the climate crisis and the threat of 22 pandemics or antibiotic resistance could be tackled, at least in part, with new scientific discoveries. 23 By way of illustration, materials discovery can play an important role in improving the energy 24 efficiency of energy production and storage; and reducing the costs and duration for drug discovery 25 has the potential to more effectively and rapidly mitigate the consequences of new diseases. In 26 recent years, researchers in materials science, biochemistry and other fields have increasingly adopted 27 machine learning as a tool as it holds the promise to drastically accelerate scientific discovery 28 [7, 67, 3, 12]. 29

Although machine learning has already made a positive impact in scientific discovery applications 30 [55, 27], unleashing its full potential will require improving the current algorithms [1]. For example, 31 typical tasks in potentially impactful applications in materials and drug discovery require exploring 32 combinatorially large, high-dimensional spaces [46, 6], where only small, noisy data sets are available, 33 and obtaining new annotations computationally or experimentally is very expensive. Such scenarios 34 present serious challenges even for the most advanced current machine learning methods. 35

In the search for a useful discovery, we typically define a quantitative proxy for usefulness, which we 36 can view as a black-box function. One promising avenue for improvement is developing methods 37 that more efficiently leverage the availability of multiple approximations of the target black-box 38

function at lower fidelity but much lower cost than the highest fidelity oracle [10, 14]. For example, 39 the most accurate estimation of the properties of materials and molecules is only typically obtained 40 via synthesis and characterisation in a laboratory. However, this is only feasible for a small number of 41 promising candidates. Approximate quantum mechanics simulations of a larger amount of chemical 42 compounds can be performed via Density Functional Theory (DFT) [41, 51]. However, DFT is 43 still computationally too expensive for high-throughput exploration of large search spaces. Thus, 44 large-scale exploration can only be achieved through cheaper but less accurate oracles. Nonetheless, 45 solely relying on low-fidelity approximations is clearly suboptimal. Ideally, such tasks would be best 46 tackled by methods that can efficiently and adaptively distribute the available computational budget 47 between the multiple oracles depending on the already acquired information. 48

49

50

51

52

53

54

55

56

The past decade has seen significant progress in multi-fidelity Bayesian optimisation (BO) [19, 53], including methods that leverage the potential of deep neural networks [36]. Although highly relevant for scientific discovery, standard BO is not perfectly suited for some of the challenges in materials and drug discovery tasks. First and foremost, BO's ultimate goal is to find the optimum of an expensive black-box function. However, even the highest fidelity oracles in such problems are underspecified with respect to the actual, relevant, downstream applications. Therefore, it is imperative to develop methods that, instead of 'simply' finding the optimum, discover a set of diverse, high-scoring candidates.

57

58

59

60

61

62

63

64

65

66

67

Recently, generative flow networks (GFlowNets) [4] have demonstrated their capacity to find diverse candidates through discrete probabilistic modelling, with particularly promising results when embedded in an active learning loop [22]. Here, we propose to extend the applicability of GFlowNets for multi-fidelity active learning.

In this paper, we present an algorithm for multi-fidelity active learning with GFlowNets. We provide empirical results in two synthetic benchmark tasks and four practically relevant tasks for biological sequence design and molecular modelling. As a main result, we demonstrate that multi-fidelity active learning with GFlowNets discovers diverse, high-scoring samples when multiple oracles with different fidelities and costs are available, with lower computational cost than its single-fidelity counterpart.

2 Related Work

Our work can be framed within the broad field of active learning (AL), a class of machine learning 68 methods whose goal is to learn an efficient data sampling scheme to accelerate training [50]. For the 69 bulk of the literature in AL, the goal is to train an accurate model h ( x ) of an unknown target function 70 f ( x ) , as in classical supervised learning. However, in certain scientific discovery problems, which is 71 the motivation of our work, a desirable goal is often to discover multiple, diverse candidates x with 72 high values of f ( x ) . The reason is that the ultimate usefulness of a discovery is extremely expensive 73 to quantify and we always rely on more or less accurate approximations. Since we generally have 74 the option to consider more than one candidate solution, it is safer to generate a set of diverse and 75 apparently good solutions, instead of focusing on the single global optimum of the wrong function. 76

77

78

79

80

81

82

83

84

This distinctive goal is closely connected to related research areas such as Bayesian optimisation [19] and active search [20]. Bayesian optimisation (BO) is an approach grounded in Bayesian inference for the problem of optimising a black-box objective function f ( x ) that is expensive to evaluate. In contrast to the problem we address in this paper, standard BO typically considers continuous domains and works best in relatively low-dimensional spaces [18]. Nonetheless, in recent years, approaches for BO with structured data [13] and high-dimensional domains [21] have been proposed in the literature. The main difference between BO and the problem we tackle in this paper is that we are interested in finding multiple, diverse samples with high value of f and not only the optimum.

This goal, as well as the discrete nature of the search space, is shared with active search, a variant of 85 active learning in which the task is to efficiently find multiple samples of a valuable (binary) class 86 from a discrete domain X [20]. This objective was already considered in the early 2000s by Warmuth 87 et al. for drug discovery [59], and more formally analysed in later work [26, 25]. A recent branch of 88 research in stochastic optimisation that considers diversity is so-called Quality-Diversity [9], which 89 typically uses evolutionary algorithms that perform search in the latent space. All these and other 90 problems such as multi-armed bandits [48] and the general framework of experimental design [8] all 91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

share the objective of optimising or exploring an expensive black-box function. Formal connections between some of these areas have been established in the literature [54, 17, 23, 15].

Multi-fidelity methods have been proposed in most of these related areas of research. An early survey on multi-fidelity methods for Bayesian optimisation was compiled by Peherstorfer et al. [42], and research on the subject has continued since [44, 53], with the proposal of specific acquisition functions [56] and the use of deep neural networks to improve the modelling [36]. Interestingly, the literature on multi-fidelity active learning [35] is scarcer than on Bayesian optimisation. Recently, works on multi-fidelity active search have also appeared in the literature [40]. Finally, multi-fidelity methods have recently started to be applied in scientific discovery problems [10, 14]. However, the literature is still scarce probably because most approaches do not tackle the specific needs in scientific discovery, such as the need for diverse samples. Here, we aim addressing this need with the use of GFlowNets [4, 24] for multi-fidelity active learning.

3 Method

3.1 Background

GFlowNets Generative Flow Networks [GFlowNets; 4, 5] are amortised samplers designed for sampling from discrete high-dimensional distributions. Given a space of compositional objects X and a non-negative reward function R ( x ) , GFlowNets are designed to learn a stochastic policy π that generates x ∈ X with a probability proportional to the reward, that is π ( x ) ∝ R ( x ) . This distinctive property induces sampling diverse, high-reward objects, which is a desirable property for scientific discovery, among other applications [23].

The objects x ∈ X are constructed sequentially by sampling transitions s t → s t +1 ∈ A between partially constructed objects (states) s ∈ S , which includes a unique empty state s 0 . The stochastic forward policy is typically parameterised by a neural network P F ( s t +1 | s t ; θ ) , where θ denotes the learnable parameters, which models the distribution over transitions s t → s t +1 from the current state s t to the next state s t +1 . The backward transitions are parameterised too and denoted P B ( s t | s t +1 ; θ ) . The probability π ( x ) of generating an object x is given by P F and its sequential application:

which sums over all trajectories τ with terminating state x , where τ = ( s 0 → s 1 . . . → s | τ | ) is a complete trajectory. To learn the parameters θ such that π ( x ) ∝ R ( x ) we use the trajectory balance learning objective [37]

where Z θ is an approximation of the partition function ∑ x ∈X R ( x ) that is learned. The GFlowNet learning objective supports training from off-policy trajectories, so for training the trajectories are typically sampled from a mixture of the current policy with a uniform random policy. The reward is also tempered to make the policy focus on the modes.

Active Learning In its simplest formulation, the active learning problem that we consider is as follows: we start with an initial data set D = { ( x i , f ( x i )) } of samples x ∈ X and their evaluations by an expensive, black-box objective function (oracle) f : X → R , which we use to train a surrogate model h ( x ) . A GFlowNet can then be trained to learn a generative policy π θ ( x ) using h ( x ) as reward function, that is R ( x ) = h ( x ) . Optionally, we can instead train a probabilistic proxy p ( f |D ) and use as reward the output of an acquisition function α ( x, p ( f |D )) that considers the epistemic uncertainty of the surrogate model, as typically done in Bayesian optimisation. Finally, we use the policy π ( x ) to generate a batch of samples to be evaluated by the oracle f , we add them to our data set and repeat the process a number of active learning rounds.

While much of the active learning literature [50] has focused on so-called pool-based active learning, where the learner selects samples from a pool of unlabelled data, we here consider the scenario of de novo query synthesis , where samples are selected from the entire object space X . This scenario

is particularly suited for scientific discovery [30, 62, 64, 33]. The ultimate goal pursued in active 137 learning applications is also heterogeneous. Often, the goal is the same as in classical supervised 138 machine learning: to train an accurate (proxy) model h ( x ) of the unknown target function f ( x ) . For 139 some problems in scientific discovery, we are usually not interested in the accuracy in the entire input 140 space X , but rather in discovering new, diverse objects with high values of f . This is connected to 141 other related problems such as Bayesian optimisation [19], active search [20] or experimental design 142 [8], as reviewed in Section 2. 143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

3.2 Multi-Fidelity Active Learning

We now consider the following active learning problem with multiple oracles of different fidelities. Our ultimate goal is to generate a batch of K samples x ∈ X according to the following desiderata:

The samples obtain a high value when evaluated by the objective function f : X → R + .

The samples in the batch should be distinct and diverse, that is cover distinct high-valued regions of f .

Furthermore, we are constrained by a computational budget Λ that limits our capacity to evaluate f . While f is extremely expensive to evaluate, we have access to a discrete set of surrogate functions (oracles) { f m } 1 ≤ m ≤ M : X → R + , where m represents the fidelity index and each oracle has an associated cost λ m . We assume f M = f because there may be even more accurate oracles for the true usefulness but we do not have access to them, which means that even when measured by f = f M , diversity remains an important objective. We also assume, without loss of generality, that the larger m , the higher the fidelity and that λ 1 < λ 2 < . . . < λ M . This scenario resembles many practically relevant problems in scientific discovery, where the objective function f M is indicative but not a perfect proxy of the true usefulness of objects x -hence we want diversity-yet it is extremely expensive to evaluate-hence cheaper, approximate models are used in practice.

In multi-fidelity active learning-as well as in multi-fidelity Bayesian optimisation-the iterative sampling scheme consists of not only selecting the next object x (or batch of objects) to evaluate, but also the level of fidelity m , such that the procedure is cost-effective.

Our algorithm, MF-GFN, detailed in Algorithm 1, proceeds as follows: An active learning round j starts with a data set of annotated samples D j = { ( x i , f m ( x i ) , m i ) } 1 ≤ m ≤ M . The data set is used to fit a probabilistic multi-fidelity surrogate model h ( x, m ) of the posterior p ( f m ( x ) | x, m, D ) . We use Gaussian Processes [47], as is common in Bayesian optimisation, to model the posterior, such that the model h predicts the conditional Gaussian distribution of f m ( x ) given ( x, m ) and the existing data set D . We implement a multi-fidelity GP kernel by combining a Matern kernel evaluated on x with a linear downsampling kernel over m [61]. In the higher dimensional problems, we use Deep Kernel Learning [60] to increase the expressivity of the surrogate models. The candidate x is modelled with the deep kernel while the fidelity m is modelled with the same linear downsamling kernel. The output of the proxy model is then used to compute the value of a multi-fidelity acquisition function α ( x, m ) . In our experiments, we use the multi-fidelity version [56] of Max-Value Entropy Search (MES) [58], which is an information-theoretic acquisition function widely used in Bayesian optimization. MES aims to maximize the mutual information between the value of the queried x and the maximum value attained by the objective function, f ⋆ . The multi-fidelity variant is designed to select the candidate x and the fidelity m that maximize the mutual information between f ⋆ M and the oracle at fidelity m , f m , weighted by the cost of the oracle:

objects are annotated by the corresponding oracles and incorporated into the data set, such that 186 D j +1 = D j ∪ { ( x 1 , f m ( x 1 ) , m 1 ) , . . . ( x B , f m ( x B ) , m B ) } . 187

Algorithm 1: MF-GFN: Multi-fidelity active learning with GFlowNets. See Section 4.1 for quality (TopK ( D ) ) and diversity metrics.

Input: { ( f , λ ) } : M oracles and their corresponding costs;

m m D 0 = { ( x i , f m ( x i ) , m i ) } : Initial dataset; h ( x, m ) : Multi-fidelity Gaussian Process proxy model; α ( x, m ) : Multi-fidelity acquisition function; R ( α ( x ) , β ) : reward function to train the GFlowNet; B : Batch size of oracles queries; Λ : Maximum available budget; K : Number of top-scoring candidates to be evaluated at termination; Result: TopK ( D ) , Diversity Initialization: Λ j = 0 , D = D 0 while Λ j < Λ do · Fit h on dataset D ; · Train GFlowNet with reward R ( α ( x ) , β ) to obtain policy π θ ( x ) ; · Sample B tuples ( x i , m i ) ∼ π θ ; · Evaluate each tuple with the corresponding oracle to form batch B = { ( x 1 , f m ( x 1 ) , m 1 ) , . . . , ( x B , f m ( x B ) , m B ) } ; · Update dataset D = D∪B ; end

3.3 Multi-Fidelity GFlowNets

In order to use GFlowNets in the multi-fidelity active learning loop described above, we propose to make the GFlowNet sample the fidelity m for each object x ∈ X in addition to x itself. Formally, given a baseline GFlowNet with state and transition spaces S and A , we augment the state space with a new dimension for the fidelity M ′ = { 0 , 1 , 2 , . . . , M } (including m = 0 , which corresponds to unset fidelity), such that the augmented, multi-fidelity space is S M ′ = S ∪ M ′ . The set of allowed transitions A M is augmented such that a fidelity m > 0 of a trajectory must be selected once, and only once, from any intermediate state.

Intuitively, allowing the selection of the fidelity at any step in the trajectory should give flexibility for better generalisation. At the end, finished trajectories are the concatenation of an object x and the fidelity m , that is ( x, m ) ∈ X M = X ∪ M . In summary, the proposed approach enables to jointly learn the policy that samples objects in a potentially very large, high-dimensional space, together with the level of fidelity, that maximise a given multi-fidelity acquisition function as reward.

4 Empirical Evaluation

In this section, we describe the evaluation metrics and experiments performed to assess the validity and performance of our proposed approach of multi-fidelity active learning with GFlowNets. Overall, the purpose of this empirical evaluation is to answer the following questions:

Question 1 : Is our multi-fidelity active learning approach able to find high-scoring, diverse samples at lower cost than active learning with a single oracle?

Question 2 : Does our proposed multi-fidelity GFlowNet, which learns to sample fidelities together with objects ( x, m ) , provide any advantage over sampling only objects x ?

In Section 4.1 we describe the metrics proposed to evaluate the performance our proposed method, as well as the baselines, which we describe in Section 4.2. In Section 4.3, we present results on synthetic tasks typically used in the multi-fidelity BO and active learning literature. In Section 4.4, we present results on more practically relevant tasks for scientific discovery, such as the design of DNA sequences and anti-microbial peptides.

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

4.1 Metrics

One core motivation in the conception of GFlowNets, as reported in the original paper [4], was the goal of sampling diverse objects with high-score, according to a reward function.

Mean score, as per the highest fidelity oracle f M , of the topK samples.

Mean pairwise similarity within the topK samples.

Furthermore, since here we are interested in the cost effectiveness of the active learning process, in this section we will evaluate the above metrics as a function of the cost accumulated in querying the multi-fidelity oracles. It is important to note that the multi-fidelity approach is not aimed at achieving better mean topK scores than a single-fidelity active learning counterpart, but rather the same mean topK scores with a smaller budget .

4.2 Baselines

In order to evaluate our approach, and to shed light on the questions stated above, we consider the following baselines:

GFlowNet with highest fidelity (SF-GFN): GFlowNet based active learning approach from [22] with the highest fidelity oracle to establish a benchmark for performance without considering the cost-accuracy trade-offs.

GFlowNet with random fidelities (Random fid. GFlowNet): Variant of SF-GFN where the candidates are generated with the GFlowNet but the fidelities are picked randomly and a multi-fidelity acquisition function is used, to investigate the benefit of deciding the fidelity with GFlowNets.

Random candidates and fidelities (Random): Quasi-random approach where the candidates and fidelities are picked randomly and the top ( x, m ) pairs scored by the acquisition function are queried.

Multi-fidelity PPO (MF-PPO): Instantiation of multi-fidelity Bayesian optimization where the acquisition function is optimized using proximal policy optimization [PPO 49].

4.3 Synthetic Tasks

As an initial assessment of MF-GFNs, we consider two synthetic functions-Branin and Hartmannwidely used in the single- and multi-fidelity Bayesian optimisation literature [44, 53, 28, 36, 16].

Branin We consider an active learning problem in a two-dimensional space where the target function f M is the Branin function, as modified in [52] and implemented in botorch [2]. We simulate three levels of fidelity, including the true function. The lower-fidelity oracles, the costs of the oracles (0.01, 0.1, 1.0) as well as the number of points queried in the initial training set were adopted from [36]. We provide further details about the task in Appendix B.2. In order to consider a discrete design space, we map the domain to a discrete 100 × 100 grid. We model this grid with a GFlowNet as in [4, 37]: starting from the origin (0 , 0) , for any state s = ( x 1 , x 2 ) , the action space consists of the choice between the exit action or the dimension to increment by 1 , provided the next state is in the limits of the grid. Fig. 1a illustrates the results for this task. We observe that MF-GFN is able to reach the minimum of the Branin function with a smaller budget than the single-fidelity counterpart and the baselines.

Hartmann Next, we consider the 6-dimensional Hartmann function as objective f M on a hyper-grid domain. As with Branin, we consider three oracles, adopting the lower-fidelity oracles and the set of costs (0.125, 0.25, 1.0) from [53]. We discretize the domain into a six-dimensional hyper-grid of length 10, yielding 10 6 possible candidate points. The results for the task are illustrated in Fig. 1b, which indicate that multi-fidelity active learning with GFlowNets (MF-GFN) offers an advantage over single-fidelity active learning (SF-GFN) as well as some of the other baselines in this higherdimensional synthetic problem as well. Note that while MF-PPO performs better in this task, as shown in the next experiments, MF-PPO tends to collapse to single modes of the function in more complex high-dimensional scenarios.

262

263

264

265

266

267

10

2

10

1

10 0

Fraction of total SF-GFN budget (log)

50

40

30

20

10

0

Mean Top-50 score

SF-GFN

Rand. fid. GFN

10

2

10

1

10 0

Fraction of total SF-GFN budget (log)

1.5

2.0

2.5

3.0

Mean Top-10 score

SF-GFN

Rand. fid. GFN

MF-PPO

(b) Hartmann task

4.4 Benchmark Tasks

While the synthetic tasks are insightful and convenient for analysis, to obtain a more solid assessment of the performance of MF-GFN, we evaluate it, together with the other baselines, on more complex, structured design spaces of practical relevance. We present results on a variety of tasks including DNA aptamers (Section 4.4.1), anti-microbial peptides (Section 4.4.2) and small molecules (Section 4.4.3).

0.2

0.4

0.6

0.8

1.0

Fraction of total SF-GFN budget

4

6

8

10

12

14

16

Mean Top-100 energy

SF-GFN

Random

Rand. fid. GFN

MF-PPO

(-) Diversity (+)

10

3

10

2

10

1

10 0

Fraction of total SF-GFlowNet budget (log)

0.5

0.6

0.7

0.8

Mean Top-50 energy

(-) Diversity (+)

(a) DNA aptamers task

(b) Anti-microbial peptides (AMP) task

4.4.1 DNA Aptamers

DNA aptamers are single-stranded nucleotide sequences with multiple applications in polymer design 268 due to their specificity and affinity as sensors in crowded biochemical environments [66, 11, 63, 29]. 269 DNA sequences are represented as strings of nucleobases A, C, T or G. In our experiments, we 270 consider fixed-length sequences of 30 bases and design a GFlowNet environment where the action 271 space A consists of the choice of base to append to the sequence, starting from an empty sequence. 272 This yields a design space of size |X| = 4 30 (ignoring the selection of fidelity in MF-GFN). As 273 the optimisation objective f M (highest fidelity) we used the free energy of the secondary structure 274 as calculated by NUPACK [65]. As a lower fidelity oracle, we trained a transformer model on 1 275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

million randomly sampled sequences annotated with f M , and assigned it a cost 100 × smaller than the highest-fidelity oracle. Further details about the task are discussed in Appendix B.4.

The main results on the DNA aptamers task are presented in Fig. 2a. We observe that on this task MF-GFN outperforms all other baselines in terms cost efficiency. For instance, MF-GFN achieves the best mean topK energy achieved by its single-fidelity counterpart with just about 20% of the budget. It is also more efficient than GFlowNet with random fidelities and MF-PPO. Crucially, we also see that MF-GFN maintains a high level of diversity, even after converging to topK reward. On the contrary, MF-PPO is not able to discover diverse samples, as is expected based on prior work [22].

4.4.2 Antimicrobial Peptides

Antimicrobial peptides are short protein sequences which possess antimicrobial properties. As proteins, these are sequences of amino-acids-a vocabulary of 20 along with a special stop token. We consider variable length proteins sequences with up to 50 residues. We use data from DBAASP [45] containing antimicrobial activity labels, which is split into two sets - one used for training the oracle and one as the initial dataset in the active learning loop, following [22]. To establish the multi-fidelity setting, we train different models with different capacities and with different subsets of the data. The details about these oracles along with additional details about the task are discussed in Appendix B.5.

The results in Fig. 2b inidicate that even in this task MF-GFN outperforms all other baselines in terms of cost-efficiency. It reaches the same maximum mean topK score as the random baselines with 10 × less budget and almost 100 × less budget than SF-GFN. In this task, MF-PPO did not achieve comparable results. Crucially, the diversity of the final batch found by MF-GFN stayed high, satisfying this important criterion in the motivation of this method.

4.4.3 Small Molecules

Molecules are clouds of interacting electrons (and nuclei) described by a set of quantum mechanical descriptions, or properties. These properties dictate their chemical behaviours and applications. Numerous approximations of these quantum mechanical properties have been developed with different methods at different fidelities, with the famous example of Jacob's ladder in density functional theory [43]. To demonstrate the capability of MF-GFlowNet to function in the setting of quantum chemistry, we consMF-GFNoof-of-concept tasks in molecular electronic potentials: maximization of adiabatic electron affinity (EA) and (negative) adiabatic ionisation potential (IP). These electronic potentials dictate the molecular redox chemistry, and are key quantities in organic semiconductors, photoredox catalysis, or organometallic synthesis. We employed three oracles that correlate with experimental results as approximations of the scoring function, by uses of varying levels of geometry optimisation to obtain approximations to the adiabatic geometries, followed by the calculation of IP or EA with semi-empirical quantum chemistry XTB (see Appendix) [39]. These three oracles had costs of 1, 3 and 7 (respectively), proportional to their computational running demands. We designed the GFlowNet state space by using sequences of SELFIES tokens [32] (maximum of 64) to represent molecules, starting from an empty sequence; every action consists of appending a new token to the sequence.

The realistic configuration and practical relevance of these tasks allow us to draw stronger conclusions about the usefulness of multi-fidelity active learning with GFlowNets in scientific discovery applications. As in the other tasks evaluated, we here also found MF-GFN to achieve better cost efficiency at finding high-score topK molecules (Fig. 3), especially for ionization potentials (Fig. 3a). By clustering the generated molecules, we find that MF-GFN captures as many modes as random generation, far exceeding that of MF-PPO. Indeed, while MF-PPO seems to outperform MF-GFN in the task of electron affinity (Fig. 3b), all generated molecules were from a few clusters, which is of much less utility for chemists.

5 Conclusions, Limitations and Future Work

In this paper, we present MF-GFN, the first application of GFlowNets for multi-fidelity active learning. Inspired by the encouraging results of GFlowNets in (single-fidelity) active learning for biological sequence design [22] as a method to discover diverse, high-scoring candidates, we propose MF-GFN

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

0.2

0.4

0.6

0.8

1.0

Fraction of total SF-GFN budget

9

8

7

6

5

4

Mean Top-100 energy [eV]

SF-GFN

Random

Rand. fid. GFN

MF-PPO

(-) Diversity (+)

(a) Molecules ionisation potential (IP) task

0.2

0.4

0.6

0.8

1.0

Fraction of total SF-GFN budget

2

3

4

5

Mean Top-100 energy [eV]

SF-GFN

Random

Rand. fid. GFN

MF-PPO

(-) Diversity (+)

(b) Molecules electron affinity (EA) task

to sample the candidates as well as the fidelity at which the candidate is to be evaluated, when multiple oracles are available with different fidelities and costs.

We evaluate the proposed MF-GFN approach in both synthetic tasks commonly used in the multifidelity Bayesian optimization literature and benchmark tasks of practical relevance, such as DNA aptamer generation, antimicrobial peptide design and molecular modelling. Through comparisons with previously proposed methods as well as with variants of our method designed to understand the contributions of different components, we conclude that multi-fidelity active learning with GFlowNets not only outperforms its single-fidelity active learning counterpart in terms of cost effectiveness and diversity of sampled candidates, but it also offers an advantage over other multi-fidelity methods due to its ability to learn a stochastic policy to jointly sample objects and the fidelity of the oracle to be used to evaluate them.

Broader Impact Our work is motivated by pressing challenges to sustainability and public health, and we envision applications of our approach to drug discovery and materials discovery. However, as with all work on these topics, there is a potential risk of dual use of the technology by nefarious actors [57].

Limitations and Future Work Aside from the molecular modelling tasks, our empirical evaluations in this paper involved simulated oracles with relatively arbitrary costs. Therefore, future work should evaluate MF-GFN with practical oracles and sets of costs that reflect their computational or financial demands. Furthermore, we believe a promising avenue that we have not explored in this paper is the application of MF-GFN in more complex, structured design spaces, such as hybrid (discrete and continuous) domains [34], as well as multi-fidelity, multi-objective problems [24].

References

Ankit Agrawal and Alok Choudhary. Perspective: Materials informatics and big data: Realization of the 'fourth paradigm' of science in materials science. Apl Materials , 4(5):053208, 2016.

Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33 , 2020.

Ali Bashir, Qin Yang, Jinpeng Wang, Stephan Hoyer, Wenchuan Chou, Cory McLean, Geoff Davis, Qiang Gong, Zan Armstrong, Junghoon Jang, et al. Machine learning guided aptamer refinement and discovery. Nature Communications , 12(1):2366, 2021.

Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow 357 network based generative models for non-iterative diverse candidate generation. Advances in 358 Neural Information Processing Systems , 34:27381-27394, 2021. 359

Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J. Hu, Mo Tiwari, and Emmanuel Bengio. 360 Gflownet foundations. arXiv preprint arXiv: Arxiv-2111.09266 , 2021. 361

362

363

364

Regine S Bohacek, Colin McMartin, and Wayne C Guida. The art and practice of structurebased drug design: a molecular modeling perspective. Medicinal research reviews , 16(1):3-50, 1996.

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

Keith T Butler, Daniel W Davies, Hugh Cartwright, Olexandr Isayev, and Aron Walsh. Machine learning for molecular and materials science. Nature , 559(7715):547-555, 2018.

Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: A review. Statistical science , pages 273-304, 1995.

Konstantinos Chatzilygeroudis, Antoine Cully, Vassilis Vassiliades, and Jean-Baptiste Mouret. Quality-diversity optimization: a novel branch of stochastic optimization. In Black Box Optimization, Machine Learning, and No-Free Lunch Theorems , pages 109-135. Springer, 2021.

Chi Chen, Yunxing Zuo, Weike Ye, Xiangguo Li, and Shyue Ping Ong. Learning properties of ordered and disordered materials from multi-fidelity data. Nature Computational Science , 1(1):46-53, 2021.

David R Corey, Masad J Damha, and Muthiah Manoharan. Challenges and opportunities for nucleic acid therapeutics. nucleic acid therapeutics , 32(1):8-13, 2022.

Payel Das, Tom Sercu, Kahini Wadhawan, Inkit Padhi, Sebastian Gehrmann, Flaviu Cipcigan, Vijil Chenthamarakshan, Hendrik Strobelt, Cicero Dos Santos, Pin-Yu Chen, et al. Accelerated antimicrobial discovery via deep generative models and molecular dynamics simulations. Nature Biomedical Engineering , 5(6):613-623, 2021.

Aryan Deshwal and Janardhan Rao Doppa. Combining latent space and structured kernels for bayesian optimization over combinatorial spaces. In Neural Information Processing Systems , 2021.

Clyde Fare, Peter Fenner, Matthew Benatan, Alessandro Varsi, and Edward O Pyzer-Knapp. A multi-fidelity machine learning approach to high throughput materials screening. npj Computational Materials , 8(1):257, 2022.

Francesco Di Fiore, Michela Nardelli, and Laura Mainini. Active learning and bayesian optimization: a unified perspective to learn with a goal. arXiv preprint arXiv: Arxiv-2303.01560 , 2023.

Jose Pablo Folch, Robert M Lee, Behrang Shafei, David Walz, Calvin Tsay, Mark van der Wilk, and Ruth Misener. Combining multi-fidelity modelling and asynchronous batch bayesian optimization. Computers & Chemical Engineering , 172:108194, 2023.

Adam Evan Foster. Variational, Monte Carlo and policy-based approaches to Bayesian 393 experimental design . PhD thesis, University of Oxford, 2021. 394

Peter I. Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv: Arxiv-1807.02811 , 395 2018. 396

Roman Garnett. Bayesian optimization . Cambridge University Press, 2023. 397

Roman Garnett, Yamuna Krishnamurthy, Xuehan Xiong, Jeff Schneider, and Richard Mann. 398 Bayesian optimal active search and surveying. arXiv preprint arXiv:1206.6406 , 2012. 399

Antoine Grosnit, Rasul Tutunov, Alexandre Max Maraval, Ryan-Rhys Griffiths, Alexander I. 400 Cowen-Rivers, Lin Yang, Lin Zhu, Wenlong Lyu, Zhitang Chen, Jun Wang, Jan Peters, and 401 Haitham Bou-Ammar. High-dimensional bayesian optimisation with variational autoencoders 402 and deep metric learning. arXiv preprint arXiv: Arxiv-2106.03609 , 2021. 403

Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garcia, Jarrid Rector-Brooks, Bonaventure FP 404 Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. 405 Biological sequence design with gflownets. In International Conference on Machine Learning , 406 pages 9786-9801. PMLR, 2022. 407

408

409

Moksh Jain, Tristan Deleu, Jason Hartford, Cheng-Hao Liu, Alex Hernandez-Garcia, and Yoshua Bengio. Gflownets for ai-driven scientific discovery. Digital Discovery , 2023.

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

Moksh Jain, Sharath Chandra Raparthy, Alex Hernandez-Garcia, Jarrid Rector-Brooks, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. arXiv preprint arXiv:2210.12765 , 2022.

Shali Jiang, Roman Garnett, and Benjamin Moseley. Cost effective active search. Advances in Neural Information Processing Systems , 32, 2019.

Shali Jiang, Gustavo Malkomes, Geoff Converse, Alyssa Shofner, Benjamin Moseley, and Roman Garnett. Efficient nonmyopic active search. In International Conference on Machine Learning , pages 1714-1723. PMLR, 2017.

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature , 596(7873):583-589, 2021.

Kirthevasan Kandasamy, Gautam Dasarathy, Junier B. Oliva, Jeff Schneider, and Barnabas Poczos. Multi-fidelity gaussian process bandit optimisation, 2019.

Michael Kilgour, Tao Liu, Brandon D Walker, Pengyu Ren, and Lena Simine. E2edna: Simulation protocol for dna aptamers with ligands. Journal of Chemical Information and Modeling , 61(9):4139-4144, 2021.

Ross D King, Kenneth E Whelan, Ffion M Jones, Philip GK Reiser, Christopher H Bryant, Stephen H Muggleton, Douglas B Kell, and Stephen G Oliver. Functional genomic hypothesis generation and experimentation by a robot scientist. Nature , 427(6971):247-252, 2004.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.

Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alan Aspuru-Guzik. Self-referencing embedded strings (selfies): A 100% robust molecular string representation. Machine Learning: Science and Technology , 1(4):045024, 2020.

A Gilad Kusne, Heshan Yu, Changming Wu, Huairuo Zhang, Jason Hattrick-Simpers, Brian DeCost, Suchismita Sarker, Corey Oses, Cormac Toher, Stefano Curtarolo, et al. On-thefly closed-loop materials discovery via bayesian active learning. Nature communications , 11(1):5966, 2020.

Salem Lahlou, Tristan Deleu, Pablo Lemos, Dinghuai Zhang, Alexandra Volokhova, Alex Hernández-García, Léna Néhale Ezzine, Yoshua Bengio, and Nikolay Malkin. A theory of continuous generative flow networks. International Conference on Machine Learning , 2023.

Shibo Li, Robert M Kirby, and Shandian Zhe. Deep multi-fidelity active learning of highdimensional outputs. arXiv preprint arXiv:2012.00901 , 2020.

Shibo Li, Wei Xing, Mike Kirby, and Shandian Zhe. Multi-fidelity bayesian optimization via deep neural networks. Advances in Neural Information Processing Systems , 2020.

Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets, 2022.

Henry B. Moss, David S. Leslie, Javier Gonzalez, and Paul Rayson. Gibbon: General-purpose information-based bayesian optimisation, 2021.

Hagen Neugebauer, Fabian Bohle, Markus Bursch, Andreas Hansen, and Stefan Grimme. Benchmark study of electrochemical redox potentials calculated with semiempirical and dft methods. The Journal of Physical Chemistry A , 124(35):7166-7176, 2020.

Quan Nguyen, Arghavan Modiri, and Roman Garnett. Nonmyopic multifidelity acitve search. 451 In International Conference on Machine Learning , pages 8109-8118. PMLR, 2021. 452

Robert G Parr. Density functional theory of atoms and molecules. In Horizons of Quantum 453 Chemistry: Proceedings of the Third International Congress of Quantum Chemistry Held at 454 Kyoto, Japan, October 29-November 3, 1979 , pages 5-15. Springer, 1980. 455

456

457

Benjamin Peherstorfer, Karen Willcox, and Max Gunzburger. Survey of multifidelity methods in uncertainty propagation, inference, and optimization. Siam Review , 60(3):550-591, 2018.

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

John P Perdew and Karla Schmidt. Jacob's ladder of density functional approximations for the exchange-correlation energy. In AIP Conference Proceedings , volume 577, pages 1-20. American Institute of Physics, 2001.

Paris Perdikaris, M. Raissi, Andreas C. Damianou, ND Lawrence, and George Em Karniadakis. Nonlinear information fusion algorithms for data-efficient multi-fidelity modelling. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , 473, 2017.

Malak Pirtskhalava, Anthony A Amstrong, Maia Grigolava, Mindia Chubinidze, Evgenia Alimbarashvili, Boris Vishnepolsky, Andrei Gabrielian, Alex Rosenthal, Darrell E Hurt, and Michael Tartakovsky. Dbaasp v3: database of antimicrobial/cytotoxic activity and structure of peptides as a resource for development of new therapeutics. Nucleic acids research , 49(D1):D288-D297, 2021.

Pavel G Polishchuk, Timur I Madzhidov, and Alexandre Varnek. Estimation of the size of drug-like chemical space based on gdb-17 data. Journal of computer-aided molecular design , 27:675-679, 2013.

Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning . The MIT Press, 11 2005.

Herbert E. Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society , 58:527-535, 1952.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Burr Settles. Active learning literature survey. Independent Technical Report , 2009.

David S Sholl and Janice A Steckel. Density functional theory: a practical introduction . John Wiley & Sons, 2022.

András Sobester, Alexander Forrester, and Andy Keane. Appendix: Example Problems , pages 195-203. John Wiley & Sons, Ltd, 2008.

Jialin Song, Yuxin Chen, and Yisong Yue. A general framework for multi-fidelity bayesian optimization with gaussian processes. In International Conference on Artificial Intelligence and Statistics , 2018.

Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995 , 2009.

Jonathan M Stokes, Kevin Yang, Kyle Swanson, Wengong Jin, Andres Cubillos-Ruiz, Nina M Donghia, Craig R MacNair, Shawn French, Lindsey A Carfrae, Zohar Bloom-Ackermann, et al. A deep learning approach to antibiotic discovery. Cell , 180(4):688-702, 2020.

Shion Takeno, Hitoshi Fukuoka, Yuhki Tsukada, Toshiyuki Koyama, Motoki Shiga, Ichiro Takeuchi, and Masayuki Karasuyama. Multi-fidelity Bayesian optimization with max-value entropy search and its parallelization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 9334-9345. PMLR, 13-18 Jul 2020.

Fabio Urbina, Filippa Lentzos, Cédric Invernizzi, and Sean Ekins. Dual use of artificial497 intelligence-powered drug discovery. Nature Machine Intelligence , 4(3):189-191, 2022. 498

Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient bayesian optimization, 499 2018. 500

Manfred KK Warmuth, Gunnar Rätsch, Michael Mathieson, Jun Liao, and Christian Lemmen. 501 Active learning in the drug discovery process. Advances in Neural information processing 502 systems , 14, 2001. 503

Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel 504 learning. In Artificial intelligence and statistics , pages 370-378. PMLR, 2016. 505

506

507

Jian Wu, Saul Toscano-Palmerin, Peter I. Frazier, and Andrew Gordon Wilson. Practical multi-fidelity bayesian optimization for hyperparameter tuning, 2019.

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

Dezhen Xue, Prasanna V Balachandran, John Hogden, James Theiler, Deqing Xue, and Turab Lookman. Accelerated search for materials with targeted properties by adaptive design. Nature communications , 7(1):1-9, 2016.

Joseph D Yesselman, Daniel Eiler, Erik D Carlson, Michael R Gotrik, Anne E d'Aquino, Alexandra N Ooms, Wipapat Kladwang, Paul D Carlson, Xuesong Shi, David A Costantino, et al. Computational design of three-dimensional rna structure and function. Nature nanotechnology , 14(9):866-873, 2019.

Ruihao Yuan, Zhen Liu, Prasanna V Balachandran, Deqing Xue, Yumei Zhou, Xiangdong Ding, Jun Sun, Dezhen Xue, and Turab Lookman. Accelerated discovery of large electrostrains in batio3-based piezoelectrics using active learning. Advanced materials , 30(7):1702884, 2018.

Joseph N Zadeh, Conrad D Steenberg, Justin S Bois, Brian R Wolfe, Marshall B Pierce, Asif R Khan, Robert M Dirks, and Niles A Pierce. Nupack: Analysis and design of nucleic acid systems. Journal of computational chemistry , 32(1):170-173, 2011.

Wenhu Zhou, Runjhun Saran, and Juewen Liu. Metal sensing by dna. Chemical reviews , 117(12):8272-8325, 2017.

C Lawrence Zitnick, Lowik Chanussot, Abhishek Das, Siddharth Goyal, Javier Heras-Domingo, Caleb Ho, Weihua Hu, Thibaut Lavril, Aini Palizhati, Morgane Riviere, et al. An introduction to electrocatalyst design using machine learning for renewable energy storage. arXiv preprint arXiv:2010.09435 , 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this manuscript, the authors propose a multi-fidelity active learning scheme based on GFlowNets.\nThe work mainly aims to tackle scientific discovery problems, where one often faces exploring a huge high-dimensional space to identify novel, diverse, high-quality solutions.\nIn many scientific applications, accurately evaluating the quality of the potential solutions (or properties/characteristics of novel candidates) is expensive, hence lower-fidelity surrogate models are frequently adopted for efficient cost-effective evaluation.\nThe current work investigates how to carry out active learning - more specifically, in the context of de novo query synthesis instead of a pool-based active learning scenario - when multi-fidelity oracles/surrogates are available in order to efficiently identify a diverse set of high-quality candidates within a given budget.\n\nSTRENGTHS:\nThe multi-fidelity active learning scenario investigated in this work is of interest in various scientific discovery/design scenarios.\nThis work proposes how GFlowNet, a popular generative flow network model that can serve as an amortized sampler for drawing high-reward samples from a high-dimensional distribution, can be utilized for active learning under multi-fidelity setting.\nAccording to a relatively simple and intuitive procedure outlined in Algorithm 1, this work shows that the proposed MF-GFN has the potential to identify a diverse set of high-scoring candidates at a lower acquisition cost compared to active learning schemes that rely on a single-fidelity acquisition function.\n\nWEAKNESSES:\nAlthough the proposed approach MF-GFN is reasonable, there are several major concerns regarding the current study.\n\n1. While the authors claim that the proposed MF-GFN outperforms single-fidelity active learning as well as other multi-fidelity AL schemes, the evaluation results presented in the current manuscript (e.g., Figure 1, 2, 3) are not yet very convincing.\nIt appears that MF-GFN doesn't necessarily outperform other alternatives in a consistent manner, and when it does, the performance gain doesn't seem to be very significant.\n\n\n2. For single-fidelity AL, the authors only consider the use highest fidelity oracle, which quickly consumes the AL budget.\nUnless the high-fidelity samples lead to substantial learning improvement, it would be more desirable to use low-fidelity samples.\nOf course, the actual relative value of high-fidelity vs. low-fidelity samples (considering the acquisition cost) would be different case-by-case, hence it is unclear whether the current examples provide fair comparison between MF vs. SF active learning.\nTo be fair, single-fidelity active learning performance should be evaluated at each of the considered fidelities to provide a more comprehensive picture of how SF AL would work at different fidelities.\n\n\n3. On Hartmann function, MF-PPO clearly outperforms MF-GFN significantly. What are the characteristics of the Hartmann function that may lead to this discrepancy unlike some other examples considered in this work?\n\n\n4. Comparisons across different examples should be more consistent.\nCurrently, different sets of methods are evaluated in different examples, and different K values were used for evaluating the top K samples.\nThis looks quite arbitrary and unless there is a clear reason for these choices, the same set of methods should be evaluated based on the different examples using the same K value (or same set of K values).\n\n\n5. There should be further discussion on the computational cost of fitting h to D and retraining the GFlowNet in each iteration (of batch acquisition).\nConsidering that the multi-fidelity oracles may often be computational models with different computational cost, it may be sometimes (or often) more desirable to reduce training the GFlowNet multiple times and use this computational budget for a larger number of oracle evaluations.\nAs a result, this training cost should be considered in practice when designing and performing AL campaigns, and these practical aspects need to be discussed further.\n\n\n6. There is currently no discussion regarding the impact of the batch size on the performance of MF-GFN and its comparison to other alternatives.\n\n\n7.  Although (2) is a simple yet reasonable way of evaluating a potential sample considering both the acquisition cost and its value, it is not clear whether this would be a reliable estimate of the \"value\" of a given sample normalized by its acquisition cost.\nThere should be a better justification for this cost-adjusted utility function or at least some empirical evaluation.\n\nQUESTIONS:\nPlease see the comments above in \"Weaknesses\".\n\nLIMITATIONS:\nThe manuscript briefly discusses some potential limitations of the current study and its broader impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper introduce an algorithm for multi-fidelity active learning with GFlowNets and demonstrate that the proposed algorithm outperforms the baseline methods.

STRENGTHS:
The paper is well written and includes two synthetic benchmark tasks and four practically relevant tasks for extensive experiment analysis.

WEAKNESSES:
The novelty of the paper's contribution may be questioned as it appears to bear similarities to existing works such as BMFAL (Li et al., 2022) and D-MFDAL (Wu et al., 2021) with the exception of the GFlowNets component.

Regarding the experiments, several existing multi-fidelity active learning baselines are missing, including DMFAL (Li et al., 2020a), BMFAL_Random (Li et al., 2022a), BMFAL (Li et al., 2022), D-MFDAL (Wu et al., 2021), and MF-BALD (Gal et al., 2017). Furthermore, the variants of the proposed method, namely GFlowNet with random fidelities and GFlowNet with the highest fidelity, seem more akin to an ablation study rather than proper baseline comparisons.

Additionally, the evaluation metrics employed, such as the mean score and mean pairwise similarity, are specific to GFlowNet. To ensure fair comparisons, it is recommended that the author considers adopting the evaluation metrics used in previous multi-fidelity active learning papers.

QUESTIONS:
1. Where are the experiment results for mean pairwise similarity?

2. What is the number of samples selected at each fidelity level?

LIMITATIONS:
The limitations are included.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors adapt the standard GFlowNet framework to include a fidelity measurement for the oracle, and demonstrate on synthetic, biological and chemical datasets that, in almost all cases, MF-GFN outperforms relevant baselines in terms of achieving sampling performance within a fixed budget.\n\nSTRENGTHS:\n### Originality\n\nThe paper applies multi-fidelity ideas from Bayesian Optimization to the GFN framework. This is the first time such a thing has been done, and the standard GFN framework needed to be updated to sample a fidelity as well.\n\n### Quality\n\nThe quality of the paper and results is sufficient for publication. The synthetic benchmarks are standard for this area of research, and the biological examples for aptamers and peptides are biologically relevant. The QM results for small molecules are less relevant than other tasks (e.g. ADMET in drug discovery) could have been.\n\n### Clarity\n\nThe paper is very well written and easy to understand.\n\n### Significance\n\nThe paper is moderately significant since the fidelity configurations are contrived and simple and likely do not represent experimental drug discovery fidelities (see Weaknesses below).\n\nWEAKNESSES:\n- the multi-fidelity framework is rather simple and would apply in situations when the oracle is computational (e.g. running DFT) rather than experimental (e.g. running biochemical assay), since computation costs are easily assumed to be uniform and applicable per sample, whereas experimental costs are more complex, can require batch acquisition rather than single sample, and the results can be far noisier in general. While it is infeasible to perform such a study for this paper, a synthetic example could be constructed with such properties.\n- In the main paper, each task is only run with a single multi-fidelity configuration, but it would be interesting to run the same task but with different MF configurations in order to understand how the distribution of fidelities effects convergence per fraction of budget spent. Appendix D.1 does this once for the aptamer example, but a more comprehensive study, perhaps on synthetic data, would be enormously instructive.\n\nQUESTIONS:\n- In most examples, it is not clear how, at each fidelity level, the accuracy of the oracle relates to the cost. E.g. in the aptamer example, is the accuracy of an oracle trained on 1m aptamers 100x worse than the free energy calculation of the secondary structure? What is the relationship in the other examples?\n- Except for the PPO method, diversity seems to be pegged at its highest level in all examples. Is there a more nuanced discussion you could give about this diversity? Like number of modes?\n\nLIMITATIONS:\nThere is a sufficient discussion of limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this submission, the authors proposed to apply GFlowNets as a sampler to sample for active learning based on the selected acquisition functions, instead of directly optimizing them in the procedure of Multi-fidelity Bayesian Optimization (MFBO). Even though focusing on active learning applications, the authors presented the method more in the MFBO setting, which aims at optimizing a target function by iteratively querying it as well as several different low-fidelity low-cost surrogate functions. In this work, a multi-fidelity Gaussian Process was used as multi-fidelity surrogates and multi-fidelity MES was chosen as the acquisition function. The main focus of the submission is to adopt GFlowNets for MFBO to query according to the acquisition function and the authors claimed that it improves preferred diversity of queries samples. \n\nThe authors tested the performance of the proposed method with single fidelity BO with GFlowNet, random fidelity with GFlowNet, random selection, and Multi-fidelity PPO on synthetic Branin and Hartmann functions as well as real-world tasks on DNA Aptamers, protein design, and small molecule design. Although the proposed method does not always achieve the best performance, the authors claim that it has better sample efficiency with comparable score optimization performance.\n\nSTRENGTHS:\nGFlowNet was implemented for MFBO, specially for active learning tasks. The authors tested the performance of the proposed method with single fidelity BO with GFlowNet, random fidelity with GFlowNet, random selection, and Multi-fidelity PPO on synthetic Branin and Hartmann functions as well as real-world tasks on DNA Aptamers, protein design, and small molecule design. Although the proposed method does not always achieve the best performance, the authors claim that it has better sample efficiency with comparable score optimization performance.\n\nWEAKNESSES:\n1. The main concern is that the methodological contribution is limited. The authors are mostly using the existing acquisition functions as the reward function in GFlowNet to solve multi-fidelity active learning. There is not much theoretical analysis of this GFlowNet-based active learning strategy throughout the submission. A more serious concern is that the submission is very much similar as [1], by considering multi-fidelity settings but the fidelity was simply considered as an additional input variable. The whole pipeline and all the methods are very much the same. \n\n[1] Jain, Moksh, et al. \"Biological sequence design with gflownets.\" International Conference on Machine Learning. PMLR, 2022.\n\n2. The explanation of using GFlowNet can be improved. As described in the 194th line of the main text, the joint posterior distribution of the input $X$ and fidelity index $m$ was modeled but with the constraints that a fidelity $m>0$ of a trajectory must be selected once and only once, from any intermediate state. The authors may want to first define the DAG (Directed Acyclic graph) of this GFlowNet model, explicitly explain the allowable state transition, forward/backward policies, etc. \n\n3. The design of the multi-fidelity kernel may need further explanations. Especially, adding fidelity indices as additional input by adopting  $K_2(m_1, m_2)$ defined between lines 559 and 560 of Appendix does not seem to capture the difference of $m_1$ and $m_2$ and not really invariant when permuting the fidelity indices. How can this guarantee to select appropriate fidelity to query? The authors may want to discuss the intuition behind this design. Also the reference 68 does not exist in Appendix or main text.\n\n4. The provided code does not seem to have GFLowNet-based implementation but only has random and PPO implementations. \n\n5. Since the diversity of the queried data was advertised as a reason for utilizing GFlowNet, the authors may need to provide such validation results in the main text, for example by explicitly comparing the diversity at the end by different competing methods for the tasks. Also it may be better to provide more information on the PPO setup, for example whether it selects one sample or a batch of samples, because the `diversity' should be automatically taken care of by the acquisition function in the sequential setting and it may be tricky to select a batch of samples in this case.\n\n6. The author used the Top-K score of K candidates in each active learning round. The hyper-parameter K, and other hyper-parameters in GP kernels, could be influencing the results and conclusions. Sensitivity analysis should be provided.\n\n7. If GFlowNet-based sampling is the important contribution, then other MCMC sampling methods by acquisition functions besides PPO should be also benchmarked. \n\n8. There are language errors, for example, 1) there are typos in the 148th and 149th lines of the main text; 2) in the 610th line of the Appendix: ‘where C is the cost if the highest fidelity oracle’ should be corrected as ‘where C is the cost of the highest fidelity oracle’; and many others.\n\nQUESTIONS:\n1. Many tested scenarios appear to be MFBO instead of active learning. Shouldn't the author provide some actual active learning experimental results? \n\n2. How fidelity was taken care in this framework? \n\n3. How sensitive the MFBO performance will depend on different hyperparameters? \n\n4. How does the proposed method compare with either optimization-based and other MCMC sampling based methods?\n\nLIMITATIONS:\nBoth methodological limitations and societal impact were discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper offers a new framework for multi-fidelity active learning using Generative Flow Networks. Given the recent success of GFlowNets as models for sampling diverse candidates among terminating states in a DAG, the authors attempt to leverage this property to put a new spin on active learning, where instead of sampling from a pool of unlabeled candidates, new objects are sampled from the entire construction space. Furthermore, the GFlowNet is also responsible for sampling the fidelity at which to evaluate the object, which they later show can provide advantage over just sampling the objects themselves. Two important components of this framework are 1) the multi-fidelity GP proxy model, which is a surrogate for the true measure of 'goodness' of a sampled object, and 2) the multi-fidelity acquisition function, which is used as input to the reward function of the GFlowNet to encourage exploration of the construction space. The authors then justify this framework with results from a variety of domains showing that MF-GFN offers promising results in terms of its ability to effectively leverage the lower fidelity oracles to reduce the total cost of exploration compared with only using a single oracle.\n\nSTRENGTHS:\nThe paper provides an original way of injecting the desirable properties of the generative flow network into the hot field of active learning, which is especially important for guiding modern research in being able to know what experiments to run next. The paper communicates the main ideas relatively clearly and effectively, and is not limited by restrictive assumptions. The paper also backs up their claims with experimental evidence from a variety of domains.\n\nWEAKNESSES:\nI felt like the biggest weakness of the paper was probably the lack of thorough results and as noted in the Limitations section, the lack of practical oracles. To be an effective framework for active learning, I think it would have been much more compelling to tackle some real problems that domain specific scientists are working on, instead of the synthetic (Branin and Hartmann) tasks where MF-PPO seems to do just as well as the proposed model.\n\nAdditionally, I felt that the paper was a bit rushed, as there were some glaring typos (e.g. line 303) and some important aspects that were not entirely clear, such as the actual reward function that was used by the GFlowNet. This was not made explicit until the supplementary material and made it difficult for me to understand how the acquisition function precisely tied into the rest of the framework.\n\nThe last thing that I think would be helpful for the reader to understand is to give details on the budget; things like how long does it take the GFlowNet to sample one object, how long does it take for the oracle to evaluate the object, etc. would be helpful for the reader to put the timescales of things into perspective.\n\nQUESTIONS:\n* One thing that was odd was the presence of the active learning round in the reward function (in the exponent of rho). For rho != 1, it seems like the GFlowNet would have to have the active learning round as input, otherwise it wouldn't know how to appropriately match the flows. However, I could not find this detailed in the paper, and so I was wondering if this could be clarified.\n\n* What motivated the choice of using MES for the acquisition function? It felt like this ended up taking a lot of the paper to contextualize, and it seems like there are simpler choices (such as UCB or plain ES) that could have led to a more focused discussion of the MF-GFN itself.\n\n* A nice feature of GFlowNets is the ability to train from partial trajectories. Because the goal of the paper is to reduce the overall cost of exploring a design space, I would be interested to hear if any consideration has been made to try to \"early stop\" some of the trajectories from the GFlowNet, for example if the estimated flow has gone below some threshold. It seems like this might be able to further reduce the cost of sampling. Or if this was not considered, was this due to the relatively cheap cost of generating samples from the GFlowNet compared with the relatively expensive evaluation of the oracle?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-25"}