Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequences
Authors: Minsu Kim
Federico Berto
Sungsoo Ahn
Jinkyoo Park
Venue: NeurIPS 2023 poster (2023)
Keywords: Biological sequence design
offline model based optimization
conditional generation
bootstrapping
ensemble
TL;DR: This paper proposes a novel variations of classical ensemble and bootstrapping into conditional generation for offline biological sequence design.

Abstract: We study the problem of optimizing biological sequences, e.g., proteins, DNA, and RNA, to maximize a black-box score function that is only evaluated in an offline dataset. We propose a novel solution, bootstrapped training of score-conditioned generator (BootGen) algorithm. Our algorithm repeats a two-stage process. In the first stage, our algorithm trains the biological sequence generator with rank-based weights to enhance the accuracy of sequence generation based on high scores. The subsequent stage involves bootstrapping, which augments the training dataset with self-generated data labeled by a proxy score function. Our key idea is to align the score-based generation with a proxy score function, which distills the knowledge of the proxy score function to the generator. After training, we aggregate samples from multiple bootstrapped generators and proxies to produce a diverse design. Extensive experiments show that our method outperforms competitive baselines on biological sequential design tasks. We provide reproducible source code: https://github.com/kaist-silab/bootgen.

Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequences
Minsu Kim
Federico Berto
Sungsoo Ahn
Jinkyoo Park
Korea Advanced Institute of Science and Technology (KAIST)
2Pohang University of Science and Technology (POSTECH) {min-su, fberto, jinkyoo. paakk (sungsoo. ahn @hosteecc ac. kr
Abstract
We study the problem of optimizing biological sequences, e-g., proteins, DNA, and RNA, to maximize a black-box score function that is only evaluated in an offline dataset. We propose a novel solution, bootstrapped training of score- conditioned generator (BoOTGEN) algorithm. Our algorithm repeats two-stage process. In the first stage, our algorithm trains the biological sequence generator with rank-based weights to enhance the accuracy of sequence generation based on high scores. The subsequent stage involves bootstrapping, which augments the training dataset with self- gefe-aeeeetee data labeled by proxy score function. Our key idea to align the score-based generation with a proxy score function, which distills the knowledge of the proxy ing, we aggregate samples from multiple bootstrapped generators and proxies to produce diverse design. Extensive experiments show that method outper- forms competitive baselines on biological sequential design tasks. We provide reproducible source code: http:/ggiiiucoooomaaisisiissaaae
function to the generator. After train-
1 Introduction
The automatic design biological sequences, e.g., DNA, RNA, and proteins, with a specific prop- erty, e.g., high binding affinity, is vital task within the field of biotechnology [5, 54, 41, 33]. To solve this problem, researchers have developed algorithms to optimize a biological sequence to max- imize score function [40, 9, 10, 2, 29]. Here, the main challenge is the expensive evaluation of the score function that requires experiments in a laboratory setting or clinical trials.
To resolve this issue, recent works have investigated offline model-based optimization [32, 21, 44, 52, 12, 45, MBOJ. Given an offline dataset biological sequences paired with scores, offline MBO algorithms train proxy for the score function, e.g., deep neural network (DNN), and maximize the proxy function without querying the true score function. Therefore, such offline MBO algo- rithms bypass the expense of iteratively querying the true score function whenever is proposed. However, even optimizing such a proxy function is challenging due to the vast search space over the biological sequences.
solution
On the one hand, several works [21, 44, 52, 12) considered applying gradient-based maximization of the proxy function. However, when the proxy function is parameterized using DNN, these methods often generate solutions where the true score is low despite the high proxy score. This is due to the fragility of DNNs against adversarial optimization of inputs [52, 44, 21]. Furthermore, the gradient-basse methods additionally require reformulating biological sequence optimization as continuous optimization, e.g., continuous relaxation [21, 44, 12] or mapping discrete designs to continuous latent space [52].
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

On the other hand, one may consider training deep generative models to learn a distribution over high-scoring designs [32, 29]. They learn to generate solutions from scratch, which amortizes opti- mization over the design space.
To be specific, Kumar and Levine [32] suggests learning inverse map from a score to solu- tion with a focus on generating high-scoring solutions. Next, Jain et al. [29] proposed training a generative flow network [7, GFNJ as the generative distribution of high-scoring solutions.
Contribution In this paper, we propose bootstrapped training of score-conditioned generator (BootGEN) for the offline design of biological sequences. Our key idea to enhance the score- conditioned generator by suggesting a variation of the classical ensemble strategy of bootstrapping and aggregating. We train multiple generators using bootstrapped datasets from training and com- bine them with proxy models to create reliable and diverse sampling solution.
In the bootstrapped training, we aim align a score-conditioned generator with proxy function by bootstrapping the training dataset of the generator. To be specific, we repeat multiple stages of (1) training the conditional generator on the training dataset with a focus on high-scoring sequences and (2) augmenting the training dataset using sequences that are sampled from the generator and la- beled using the proxy function. Intuitively, our framework improves the -ossuenc-e mapping (generator) to be consistent with the sequence-to-scom mapping (proxy function), which is typically more accurate.
When training the score-conditioned generator, we assign high rank-based weights [46] to high- scoring sequences. Sequences that are highly ranked among the training dataset are more frequently sampled to train the generator. This leads to shifting the training distribution towards an accurate generation of high-scoring samples. Compared with the value-based weighting scheme previously proposed by Kumar and Levine [32], the rank-based weighting scheme is more robust to the change of training dataset from bootstrapping.
To further boost the performance of our algorithm, we propose two post-processing processes after the training: filtering and diversity aggregation (DA) The filtering process aims filter samples from generators using the proxy function to gather samples with cross-agreement between the proxy and generator. On the other hand, DA collects sub-samples from multiple generators and combines them into complete samples. DA enables diverse decision-making with reduced variance in gener- ating quality, collects samples from multiple bootstrapped generators.
We perform extensive experiments on six offline biological design tasks: green fluorescent protein design [54, GFP], DNA optimization for expression level on untranslated region [41, UTRJ, tran- scription factor binding [5. TFBind8]. and RNA optimization for binding to three types of transcrip- tion factors [33, RNA-Binding]. Our BOOTGEN demonstrates superior performance, surpassing the 100th percentile score and 50th percentile score of the design-hench baselines [45], a generative flow network (GFN)-based work [29] and bidirectional learning method (BDI) [12]. Furthermore, we additionally verify the superior performance of BOOTGEN in various design scenarios, particu- larly when given few opportunities propose solutions.
2
Related Works
2.1 Automatic Design of Biological Sequences
Researchers have investigated machine learning methods to automatically design biological se- quences, e.g., Bayesian optimization [50, 6, 36, 38, 43], evolutionary methods [3, 8, 18, 4, 40], model-based reinforcement learning [2], and generative methods [9, 32, 20, 29, 19, 11, 14). These methods aim optimize biological sequence (c.g., protein, RNA, and DNA) for maximizing the target objective of binding activity and folding, which has crucial application in drug discovery and health care [29].
2.2 Offline Model-based Optimization
Offline model-based optimization (MBO) aims to find the design æ that maximizes score function f()), using only pre-collected offline data. The most common approach to use gradient-based optimization on differentiable proxy models trained on an offline dataset [49, 44, 52, 21, 12]. How-

---

Step A: Rank-based Weighted Training
Step B: Bootstrapping
High Score
Bootstrapping
Deriity
Generator Ps(aly)
Reweighting
Training
Generator Po(Aly')
b
Denwiv
Top
Proxy (a (R)
Training Dataset
Reweighted
Augmented
Dataset
Figure 2.1:
Illustration of the bootstrapped training process for learning score-conditioned generator.
ever, they can lead to poor scores due to the non-smoothness of the proxy landscape. To address this issue, Trabucco al. [44] proposed conservative objective models (COMs), which adversarial training to create smooth proxy. Yu et al. [52] suggests the direct imposition a Gaussian prior on the proxy model create smooth landscape and model adaptation to perform robust estimation on the specific set of candidate design space. While these methods are effective for high-ding high-dimensional continuous design tasks, their performance on discrete spaces is often inferior classical methods, e.g., gradient ascent [45].
2.3 Bootstrapping
Bootstrapping means maximizing the utilization of existing resources. In narrow sense, it refers to statistical method where the original dataset repeatedly sampled create various datasets [25]. In a broader sense, it also encompasses concepts frequently used in machine learning to improve machine learning scenarios where the label expensive, e.g., self-training and semi-supervised learning [37, 1, 22]. These methods utilize an iterative training scheme to augment the dataset with self-labeled samples with high confidence.
The bootstrapping strategy at machine learning showed great success various domains, e.g., fully- labeled classification [51], self-supervised learning [17] and offline reinforcement learning [48]. We introduce a novel bootstrapping strategy utilizing score-conditioned generators and apply it to offline biological sequence design, addressing the challenge of working with a limited amount of poor-quality offline datasets.
2.4 Design by Conditional Generation
Conditional generation a promising method with several high-impact applications, e.g., class- conditional image generation [35], language-to-image generation [39], reinforcement learning [16]. With the success of conditional generation, several studies proposed to use it for design tasks, e.g., molecule and biological sequence design.
Hottung et al. [27] proposed an restance-conditione variational auto-encoder [31] for routing prob- lems, which can generate near-optimal routing paths conditioned on routing instances. Igashov et al. [28] suggested a conditional diffusion model to generate 3D molecules given their fragments. Specifically, the molecular fragments injected into the latent space of the diffusion model, and the diffusion model generates links between fragments to make the 3D molecular compound.
3 Bootstrapped Training of Score-Condition Generator (BOOTGEN)
Problem definition We are interested in optimizing biological sequence to maximize given score function f()). We consider an offline setting where, during optimization, we not have access to the score function f(x)). Instead, we optimize the biological sequences using static dataset = {(*,, Un) bn consisting offline queries y/- f(x,) to the score function. Finally, consider evaluating set of sequences {xm IM
output of offline design algorithms.
Overview of BOOTGEN We first provide a high-level description of our bootstrapped training of score-conditioned generator, coined BOOTGEN. Our key idea to align the score-conditioned generation with proxy model via bootstrapped training (i.e., we train the generator on sequences labeled using the proxy model) and aggregate the decisions over multiple generators and proxies for reliable and diverse sampling of solutions.
3

---

Algorithm Bootrapped Training of Score-conditioned generators
Input: Offline dataset D { Un )
2: Update Φ to minimize WE - y)²
3: for 1,..., Ngen do
Training multiple generaters
4: Initialize Dr + D.
for ......,,........................ do
Rank-based weighted training
6:
Update 0j to maximize (im))E w(y. Drr log po, (ælvv)
7:
Sample x; ~ps, -o, (xly)) for I,...:L. Set yz fa(x)) for 1,...,,,
Bootstrapping
8:
9:
Set Dang as Kop-K scoring samples in { )
10: Set Drr + Drr PPUUZA Daug.
end for
12:
end for
13:
Output: trained score-conditioned generators pa, (eely), pori (x)ly).
Before BOOTGEN training, we pre-train proxy score function fo Re I f(x) only leveraging offline dataset D. After that, our BOOTGEN first initializes training dataset Dir the offline dataset 'D and then repeats the following steps:
A. BootGen optimizes the score-conditioned generator pe (a y) using the training dataset Dir- During training, assigns rank-based weights to each sequence the generator to focus high-scoring samples.
B. BOOTGEN bootstraps the training dataset Drr using samples from the generator pa (ælyy) conditioned the desired score y It uses proxy fz(z) of the score function to label the new samples.
After BOOTGEN training for multiple condiiconi generators po, (ærly)..... Po., (mlly) we ag- gregate samples from the generators with filtering of proxy score function fo to generate diverse and reliable samples. We provide the pseudo-code of the overall procedure in Algorithm I for training and Algorithm 2 for generating solutions from the trained model.
3.1 Rank-based Weighted Training
Here, we introduce framework train the score-conditioned generator. Our algorithm aims to train the score-conditioned generator with more focus on generating high-scoring designs. Such a goal is helpful for bootstrapping and evaluation of our framework, where query the generator conditioned on a high score.
Given training dataset Dr,, our BOOTGEN minimizes the following loss function:
(I: Dul + rank(y, Da))
(()) Σ w(y, Dog w(y, Da) Eimaye (xr + rank(y, DI)) (x.y)ED.
where w(y, D.r) is the score-wise rank-based weight [46]. Here, rank(y, Dr,) denotes the relative ranking of a score y with respect to the set of scores the dataset Dy. We note that a small weight-shifting factor & assigns high weights high-scoring samples.
weight-shifting factor, and
For mini-batch training of the score-conditioned generator, we approximate the loss function C()) via sampling with probability w(y, D..) for each sample (x,y). v).
We note that Tripp al. [46] proposed the rank-based weighting scheme for training unconditional generators to solve online design problems. At a high level, the weighting scheme guides the gener- ator focus more on generating high-scoring samples. Compared to weights that are proportional to scores [32]. using the rank-based weights promotes the training to be more robust against out- liers, e.g., samples with abnormally high weights. To be specific, the weighting factor w(y. D) less affected by outliers due to its upper bound that is achieved when rank(y, D) 1.

---

Algorithm A Aggregation Strategy for Sample Generation
1: Input: Trained score-conditioned generators pe, (2) y), .... p0 (aly), and trained proxy function fa(x).
2: Initialize Deumpees f- 0.
for 1,.... Ngen
Sample Imm ~ Pa, (alyy) for E [M].
5: Set Vin + fo( for E [M].
6: Set D suubssamples as Top- K scoring samples in { { Vm J Jm=1-
Filtering Diversity Aggregation
7: Set D Psamples < D samplss sub-samples
end for
9: Output: saaples.
3.2
Bootstrapping
Next, we introduce our hootstrapping strategy to augment training dataset with high-scoring sam- ples that are collected from the score-conditioned generator and labeled using a proxy model. Our key idea is to enlarge the dataset that the score-conditioned generation is consistent with predic- tions of the proxy model, in particular for the high-scoring samples. This enables self-training by utilizing the extrapolation capabilities of the generator and allows the proxy model to transfer its knowledge to the score-conditioned generation process.
We first generate set of samples ....... - from the generator (x yy) conditioned on the desired score y Then we compute the corresponding labels ....... VG, using the proxy model, i.c., we set yt = (xz) for I,...,L. L. Finally, we augment the training dataset using the of top-K samples Paug with respect to the proxy model, i.e., we set Drr U as the new training dataset Drr-
3.3 Aggregation Strategy for Sample Generation
Here, we introduce additional post-hoc aggregation strategies that can be used to further boost the quality of samples from our generator. See Algorithm 2 for a detailed process.
Filtering We follow Kumar and Levine [32] exploit the knowledge of the proxy function for filtering high-scoring samples from the generator. To be specific, when evaluating our model, we sample of candidate solutions and select the top samples with respect to the proxy function.
Diverse aggregation To enhance the diversity of candidate samples while maintaining reliable generating performances with low variance, we gather cross-aggregated samples from multiple score-conditioned generators. These generators are independently trained using our proposed boot- strapped training approach. Since each bootstrapped training process introduces high randomness due varying training datasets, combining the generative spaces of multiple generators yields more diverse space compared to single generator.
Moreover, this process helps reduce the variance in generating quality. By creating ensemble candi- date samples from multiple generators, we ensure stability and mitigate the risk of potential failure cases caused by adversarial samples. These samples may receive high scores from the proxy func- tion but have low actual scores. This approach resembles the classical ensemble strategy known as "bagging," which aggregates noisy bootstrapped samples from decision trees reduce variances.
Experiments
We present experimental results on six representative biological sequence design tasks to verify effectiveness of the proposed method. We also conduct ablation studies to verify the effectiveness of each component in our method. For training, we use single GPU NVIDIA A100, where the training time of one generator is approximately 10 minutes.
Following Chen et al. [12], assume that know the maximum score of the task.

---

Table 4.1: Experimental results on 100th percentile scores. The mean and standard deviation are reported for 8 independent solution generations. (bbest) indicate the maximum score of the offline dataset. The best-scored value is marked bold.
Method
RNA-A
RNA-B
RNA-C
TFBind8
GFP
UTR
Avg.
(best)
0.120
0.122
0.125
0.439
0.789
0,593
0.365
REINFORCE [45]
0.462 1: 0.080
0.437 = 0.033
0.463 1 0.043
0,936 1: 0.041
0.865 :: 0.003
0.685 1: 0.012
0.643
CMA-ES [24]
0.841 0.058
0.822 0.046
0.803
0.039
0.904
0.040
0.055
0.003
0.737
0.694
BO-qEI [50]
0.724 + 0.055
0.729 = 0.038
0.707
+ 0.034
0.798 = 0.083
0.254 = 0.352
0,684 = 0.000
0.649
CbAS [9]
0.541 0.042
0.647 = 0.057
0.644
0.077
0.913 = 0.025
0.865 : 0.004
0.692 = 0.008
0.717
Auto. CbAS (20)
0.524 0.055
0.562 0.031
0.495 0.048
0.890 1 0.050
0.865 1 0.003
0.693 1: 0.009
0.672
MIN [32]
0.376 0.039
0.374 0.041
0.404-0.044
0.892 + 0.060
0.865 + 0.001
0.691
0.011
0.600
Grid [45]
0.821 0.048
0,720 0.047
0.035
0,965
0.030
0.862 + 0.003
0.013
0,792
COMs [44]
0.403 :: 0.062
0.393 0.076
0.494
0,945 = 0.033
0,861 = 0.009
0.699 0.011
0.633
AdaLead [42]
0.691 0.059
0,630 0.062
0.605 0.055
0.024
0.841 0.014
0.727
GFN-AL [29]
0.630 + 0.054
0,677
0.079
0.623
0.045
0,956
= 0.018
0.059 + 0.006
0.695 = 0.021
0.607
[12
0.700 +: 0.000
0.560 0.000
0.632 2: 0.000
0.973 = 0.000
0.864 + 0.000
0.667 0.000
0.733
BOOTGEN
0.902 1 0.039
0.931 0.055
0.831 0.044
0.979 0.001
0.865 0.000
0.865 0.000
0.895
Table 4.2: Experimental results on 50th percentile scores. The mean and standard deviation are reported for 8 independent solution generations. ((best) indicate the maximum score of the offline dataset. The best-scored value is marked bold.
Method
R
RNA-B
RNA-C
TFBind8
GFP
UTR
Avg.
(best)
0.120
0.122
0.125
0,439
0.789
0,593
0.365
REINFORCE [45]
0.159 : 0.011
0.162 = 0.007
0.177 - 0.011
0.450 = 0.017
0.845 0.003
0.575 = 0.018
0.395
CMA-ES [24)
0.558
0.012
0,531 0.010
0.535
0.012
0.526
0.017
0.047 1 0.000
0.497
0.449
BO [50]
0.389
0.009
0.397 + 0.015
0.391
+ 0.012
0.439 + 0.000
0,246 + 0.341
0,571 + 0.000
0.406
CbAS [9]
0.246 0.008
0,267 = 0.021
0.281 0.015
0.467 = 0.008
0.852 : 0.004
0.566 1 0.018
0,447
Auto. ChAS [20)
0,241 0.022
0,237 0.009
0.193 1: 0.007
0,413 1: 0.012
0.847 1: 0.003
0.563 :: 0.019
0.420
MIN [32]
0.146 0.009
0.143 0.007
0.174 0.007
0.417 + 0.012
0,830 + 0.011
0,586 + 0.000
0,383
Grad [45]
0.473 0.025
0,462 0.016
0.007 0.017
0.763 0.181
0.611
0.531
COMs [44]
0.172 0.026
0,184 = 0.039
0.228
0,512 - 0.051
0.737
+ 0.262
0.608 - 0.000
0.407
AdaLead [42]
0.407 0.018
0,353 0.029
0.019
0,485
0.186 0.216
0.392
GFN.AL. [29]
0,312 + 0.013
0,300- 0.012
0.324 0424+0000 0.009
0,538
+ 0.045
0.051 + 0.003
0,597 + 0.021
0.354
BDI
0,411 0.000
0,308
0.000
0.345
0.595
0.000
0.837 = 0.010
0.527 0.000
0,504
BOOTGEN
0.707 0.005
0.717 0.006
0.596 0.006
0,833
0.007
0.853 1 0.017
0.701 0.004
0.731
4.1 Experimental Setting
Tasks. We evaluate an offline design algorithm by (1) training it on an offline dataset and (2) using it to generate 128 samples for high scores. We measure the 50th percentile and 100th percentile scores of the generated samples. All the results are measured using eight independent random seeds.
We consider six biological sequence design tasks: green fluorescent protein (GFP), DNA optimiza- tion for expression level on untranslated region (UTR). DNA optimization tasks for transcription factor binding (TFBind8), and three RNA optimization tasks for transcription factor binding (RNA- Binding-A, RENA-Binding BB, and RNA-Binding--G -C. The scores of the biological sequences range in [o, 1]. We report the statistics of the offline datasets used for each task in Table A. We also provide a detailed description of the tasks in Appendix A.1.
Baselines We compare our BOOTGEN with the following baselines: gradient ascent with respect to a proxy score model [45, Grad.], REINFORCE [49], Bayesian optimization quasi-cxpected- improvement [50, BO-qEI], covariance matrix adaptation evolution strategy [24, CMA-ES), con- ditioning by adaptative sampling [9, CbAS), autofocused CbAS [20, Auto. CbAS], model inversion network [32, MINJ, where these are in the official design bench [45]. We compare with additional baselines of conservative objective models [44, COMs), generative flow network for active learning [29, GFN-ALJ and bidirectional learning [12, BDIJ
Implementation We parameterize the conditional distribution po(z,|11=--1,10 y) using a 2-layer long short-term memory [26, LSTM] network with 512 hidden dimensions. The condition y is injected into the LSTM using a linear projection layer. We parameterize the proxy model using a multi-layer perceptron (MLP) with 2048 hidden dimensions and a sigmoid activation function. Our parameterization consistent across all the tasks. We provide a detailed description of the hyperparameters in Appendix A. We also note the importance of the desired score y/ to condition during bootstrapping and evaluation. this regard, we set it as the maximum score that is achievable for the given problem, i.e., we set y 1. We assume that such value is known following [12).

---

1.4
144
Scrre
Backee
Daad
Doal
003
BDJ
BD0
BDI
ClAS
ClAS
100
100
Numier of mealuntione (8C)
Number of eraluniione (N)
Number of dvaluaaiiin (8C
(a) RNA Binding A
(b) RNA A..ndiniing-- B
(c) RNA Binding
010
1.813
1030
0.75
Saam
2:000
Bastioa
BostEve
Gral
Grul
Grad.
IIDE
085
6850
ChAS
Number evalasttanna (20)
Nomber ooaalnationn
TFBind8
(i)
Figure 4.1: Evaluation- penerionn-erformncee graph to compare with representative offline biological design baselines.
The number of evaluations E [1, 128] stands for the number of candidate designs to be evaluated by the Oracle score function. The average value and standard deviation error bar for 8 independent runs reported. Our method outperforms other baselines at every task for almost all K.
4.2 Performance Evaluation
In Table 4.1 and Table 4.2, we report the performance of our BOOTGEN along with other baselines. One can observe how our BOOTGEN consistently outperforms the considered baselines across all six tasks. In particular, one can observe how our BoOTGEN achieves large gains in 50th percentile metrics. This highlights how our algorithm is able to create reliable set of candidates.
For TFbind8, which has relatively small search space (4%), having high performances on the 100th percentile is relatively easy. Indeed, the classical method of CMA-ES and Grad. gave pretty good performances. However, for the 50th percentile score, metric for measuring the method's reliabil- ity, BDI outperformed previous baselines by a large margin. Our method outperformed even BDI and achieved overwhelming score.
For higher dimensional tasks of UTR, even the 50th percentile score of BOOTGEN outperforms the 100th percentile score of other baselines by large margin. We note that our bootstrapping strategies and aggregation strategy greatly contributed to improving performances on UTR. For additional tasks of RNA, achieved the best score for both the 50th percentile and the 100th percentile. This result verifies that our method is task-expandable.
4.3 Varying the Evaluation Budget
In roal-world scenarios, there may be situations where only a few samples can be evaluated due to the expensive score function. For example, in extreme scenario, for the clinical trial of new protein drug, there may be only one or two chances to be evaluated. As we measure the 50th percentile and 100th percentile score among 128 samples following the design-bench [45] at Tables 4.1 and 4.2, also provide a 100th percentile score report at the fewer samples from sample to the 128 samples to evaluate the model's 's robustness the low-budget evaluation scenarios.
To account for this, we also provide budget-performant our model to the baselines using different numbers of evaluations. This allows us to observe the trade-off between performance and the number samples generated. Note that we select baselines as the Top 5 methods in terms of average percentile 100 scores reported at Table 4.1.
graph that compares the performance of

---

Table 4.3:
Experimental results 100th percentile scores (100th Per.). 50th percentile scores (50th Per.), average score (Avg. Score), diversity, and novelty, among 128 samples of UTR task. The mean and standard deviation of 8 independent runs for producing 128 samples is reported. The best-scored value marked in bold. The lowest standard deviation is marked as the underline. The DA stands for the diverse aggregation strategy.
Methods
100th Per.
50th Per.
Avg. Score
Diversity
Novelty
MIN [32]
0.691 0.011
0.587
0.012
0.554
0.010
28.53 0.095
18.32 = 0.091
CMA-ES [24]
0.746 0.018
0.498 0.012
0.520 + 0.013
24.69 + 0.150
19.95 + 0.925
Grad. [45]
0.682 = 0.013
0.513 = 0.007
0,521 + 0.006
25.63 + 0.615
16.89 = 0.426
GFN-AL [29)
0.700 = 0.015
0.602 = 0.014
0.580
0.014
30.89 1.220
20.25 = 2.272
Boot GEN w/o DA.
0.729 = 0.074
0.672
0.082
0.652
0.081
17.83 = 5,378
20.49 = 1.904
BOOTGEN (owrs)
0.858 0.003
0.701
0.698
0.001
31.57 0.073
21.40 0.057
BestGrn
Eouttinn
GENN-AL
Grad.
Gand
0:63
ChaS
Asno ChaS
Arg. Scote
Auto ChAS
0.60
MIN
0.00
0.50
0.55
0.50
22
201
34
16
16
1.1
201
22
Diveriit
Noviity
Figure 4.2: Multi-ohjectivity comparison of diversity and novelty on the average score for the UTR task. Each datapoint for 8 independent runs is depicted.
As shown in Fig. 4.1, our method outperforms every baseline for almost every evaluation budget. For the UTR task, our performance on single evaluation budget gives a better score than the other baselines' scores when they have budget of 128 evaluations. For RNA tasks, our method with an approximate budget of 30 achieves superior performance compared to other methods with a budget of 128. These results show that our method is the most reliable as its performance is most robust when the evaluation budget is limited.
4.4 Average Score with Diversity
For biological sequence design, measuring the diversity and novelty of the generated sequence is also crucial [29]. Following the evaluation metric of [29] we compare the performance of models in terms of diversity and novelty.
Here is measurement of diversity for sampled design dataset D { ÆM² from generator which is average of Levenshtein distance [23], denoted by w;), between arbitrary two biolog- ical sequences F, 2ú from the generated design candidates D:
Diversity (D) = (
Σ
Σ (x,
ZED sED)
Next, we measure the minimum distance from the offline dataset Domin which measures the novelty of generated design candidates D as:
Novelty(D, Dortine)
Σ
min d 8).
D
DEDamm
DED
Our method surpasses all baselines, including GFN-AL [29], in the UTR task, as evidenced by the Pareto frontier depicted Table 4.3 and Fig. 4.2. Given the highly dimensional nature of the UTR task and its expansive search space, the discovery of novel and diverse candidates appears to be directly related to their average score. This implies that extensive exploration of the high- dimensional space is crucial for improving scores in the UTR task.

---

Table 4.4: Ablation study for BOOTGEN. The average score among 128 samples reported. We make 8 independent runs produce 128 samples where the mean the standard deviation are reported. For every method, an aggregation strategy is applied by default. The best-scored value is marked in bold. The lowest standard deviation is underlined. The RR stands for rank based reweighting, the B stands for bootstrapping. and the stands for filtering.
Components
RNA-A
RNA-B
RNA-C
TFbind8
UTR
GI-P
0,388 + 0.007
0,350 + 0.008
0.394 = 0.010
0.579 = 0.010
0.549 = 0.009
0.457 + 0.044
(RR)
0.483 1 0.006
0.468 1 0.008
0.441 1 0.010
0.662 1 0.009
0.586 1 0.008
0.281 1: 0.031
(RR, B)
0.408 + 0.009
0.379 0.009
0.417 + 0.006
0.666 + 0.009
0.689 + 0.003
0.470 + 0.034
IR. F)
0.576 + 0.005
0.586 0.004
0.536 = 0.007
0.833 0.004
0.621 - 0.003
0.783 + 0.011
F.B.
0.607 1 0.009
0.612 0.005
0.554 0.007
0.840 0.004
0.698 0.001
0.804 0.002
It is worth noting that GFN-AL, which is specifically designed to generate diverse, high-quality samples through an explorative policy, secures a second place for diversity. Although GFN-AL occasionally exhibits better diversity than our method and achieves second-place average score, it consistently delivers poor average scores in the GFP and RNA tasks Table 4.2. This drawback can be attributed to its high explorative policy, which necessitates focused exploration in narrow regions. In contrast, BEOTGEN consistently produces reliable scores across all tasks Table 4.2. For a comprehensive comparison with GFN-AL, please refer to the additional experiments presented in Appendix C.
Diverse aggregation strategy Our diverse aggregation (DA) strategy significantly enhances di- versity, novelty, and score variance, as demonstrated in Table 4.3. This is especially beneficial for the UTR task, which necessitates extensive exploration of a vast solution space, posing a substantial risk to the bootstrapped training process. In this context, certain bootstrapped generators may yield exceedingly high scores, while others may produce low scores due to random exploration scenarios. By employing DA, we combine multiple generators to generate candidate samples, thereby greatly stabilizing the quality of the bootstrapped generator.
4.5 Ablation study
The effectiveness of our components, namely rank-based reweighting (RR), bootstrapping (B), and filtering (F), in improving performance is evident in Table 4.4. Across all tasks, these components consistently contribute to performance enhancements. The bootstrapping process is particularly more beneficial for high-dimension tasks like UTR and GFP. This correlation is intuitive since high-dimensional tasks require a larger amount of data for effective exploration. The bootstrapped training dataset augmentation facilitates this search process by leveraging proxy knowledge. Addi- tionally, the filtering technique proves to be powerful in improving scores. As observed from the diverse aggregation and filtering, the ensemble strategy greatly enhances score-conditioned genera- tors.
Future Works
Enhancing proxy robustness While our bootstrapping method shows promise for offline bio- sequential design tasks, it has inherent technical limitations. The assumption that the generator produces superior data to the training dataset may backfire if the generator samples have poor quality designs and the proxy used is inaccurate. While the current aggregation strategy effectively manages this risk, we can address this limitation by utilizing robust learning methods of proxies such as conservative proxies modeling [44], robust model adaptation techniques [52]. parallel mentoring proxies [13), and importance-aware co-teaching of proxies [53] for further improvement.
Our approach primarily employs a straightforward archi- tectural framework, with a primary emphasis on validating its algorithmic structures in the context of offline biological sequence design. To enhance the practical utility of our method, it will be ad- vantageous to incorporate established and robust architectural paradigms, exemplified in works such as [] and [14], into the framework of our method. One promising avenue for achieving this inte- gration the incorporation of pre-trained protein language models (pLMs) [34, 15), akin to those expounded upon in [14].
Enhancing architecture of BOOTGEN

---

Conclusion
This study introduces novel approach stabilize and enhance score-conditioned generators for offline biological sequence design, incorporating the classical concepts of bootstrapping and ag- gregation. Our novel method, named BOOTGEN, consistently outperformed baselines across six offline biological sequence design tasks, encompassing RNA, DNA, and protein optimization. Our strategy of bootstrapping and aggregation yielded remarkable improvements in achieving high scores, generating diverse samples, and minimizing performance variance.
Acknowledgements
We thank all the valuable comments and suggestions from anonymous reviewers who helped us improve and refine our paper. This work was supported by the Institute of Informa- tion & communications Technology Planning & Evaluation (ITP)) grant funded by the Korean govennen((MST)y222-00003 Development of Collective Collaboration Intelligence Frame- work for Internet of Autonomous Things).
References
[1] Amini and P. Gallinari. Semi-supervised logistic regression. In ECAI, volume 2, page 11, 2002.
[2] C. Angermueller, D. Dohan, D. Belanger, Deshpande, K. Murphy, and Colwell. Model- based reinforcement learning for biological sequence design. In International conference on learning representations, 2019.
[3] F. H. Amold. Design by directed evolution. Accounts of chemical research, 31(3):125-131, 1998.
[4] F. H. Amold. Directed evolution: bringing new chemistry to life. Angewandte Chemie Inter- national Edition, 57(16):4143 4148, 2018.
[5] L. A. Barrera, A. Vedenko, J. V. Kurland, J. M. Rogers, S. S. Gisselbrecht, E. J. Rossin, Woodard, L. Mariani, K. H. Kock, Inukai, al. Survey variation in human transcription factors reveals prevalent dna binding changes. Science, 351 (458))))4)000144450 2016.
[6] D. Belanger, S. Vora, Z. Mariet, R. Deshpande, D. Dohan, C. Angermueller, Murphy, O. Chapelle, and L. Colwell. Biological sequences design using batched bayesian optimiza- tion. 2019.
[7] E. Bengio. M. Jain, M. Korablyov, D. Precup, and Y. Bengio. Flow network based genera- tive models non-iterative diverse candidate generation. Advances Neural Information Processing Systems, 144:27381-27322 2021.
[8] D. Bloom and H. Arnold. In the light of directed evolution: pathways of adaptive protein evolution. Proceedings of the National Academy of Sciences, (odsupplement_1):999555 2009.
[9] D. Brookes, H. Park, and J. Listgarten. Conditioning by adaptive sampling for robust design. International conference on machine learning, pages 773-782, PMLR, 2019.
[10] D. H. Brookes and J. Listgarten. Design by adaptive sampling. urXiv:1810.031 2018.
arXiv preprint
[11] A. Chan, A. Madani, B. Krause, and N. Naik. Deep extrapolation for attibbute-ehannced gen- eration. Advances Neural Information Processing Systems, 34:1 34:140844 14044-14409.. 2021.
[12] C. Chen, Y. Zhang, J. Fu, X. Liu, and M. Coates. Bidirectional learning for offline infinite- width model-based optimization. In Advances Neural Information Processing Systems, 2022.
10

---

[13] C. Chen, C. Beckham, Z. Liu, X. Liu, and C. Pal. Parallel- mentoring for offline model-based optimization. arXiv preprint arXiv:2309.1155: 2023.
[14] C. Chen, Y. Zhang, X. Liu, and M. Coates. Bidirectional learning for offline model-based biological sequence design. arXiv preprint arXiv:2301.0293: 2023.
[15] C. Chen, Zhou, F. Wang. X. Liu, and D. Dou. Structure-awarr protein self-supervised learning. Bioinformatics, 2023.
[16] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and
I. Mordatch, Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-11000 2021.
[17] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems, 33: 11243-22255, 2020.
[18] A. Dalby, Strategy and success for the directed evolution of enzymes. Current opinion in structural biology, 21(4):473-488) 2011.
[19] C. Ekbote, M. Jain, Das, and Y. Bengio. Consistent training via energy- based gflownets for modeling discrete joint distributions. arXiv preprint arXiv:221 1.00668, 2022.
[20] C. Fannjiang and J. Listgarten. Autofocused oracles for model-based design. Advances in Neural Information Processing Systems, 33: 0111244--2955 2020.
[21] J. Fu and S. Levine. Offline model-based optimization via normalized maximum likelihood estimation. arXiv preprint arXiv:2102.07700, 2021.
[22] Y. Grandvalet and Y. Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004.
[23] R. Haldar and D. Mukhopadhyay. Levenshtein distance technique in dictionary lookup meth- ods: An improved approach. arXiv preprint arXiv:1101.122 2011.
[24] N. Hansen. The CMA evolution strategy: comparing review. Towards evolutionary computation, pages 75-102, 2006.
[25] T. Hesterberg. Bootstrap. Wiley Interdisciplinary Reviews: Computational Statistics, 3(6): 497-526, 2011.
[26] Hochreiter and Schmidhuber. Long short-term memory. Neural computation, 9()): 1735- 1780, 1997.
[27] A. Hottung, B. Bhandari, and K. Tierney Learning a latent search space for routing problems using variational autoencoders. In International Conference on Learning Representations, 2020.
[28] I. Igashov, H. Stärk, C. Vignac, V. G. Satorras, P. Frossard, M. Welling, M. Bronstein, and B. Correia. Equivariant 3d-conditional diffusion models for molecular linker design. arXiv preprint arXiv:2210.05274, 2022.
[29] M. Jain, E. Bengio, A. Hernandez-Garcia, J. Rector-Brooks, B. F. Dossou, C. A. Ekbote, J. Fu, Zhang, M. Kilgour, D. Zhang, al. Biological sequence design with gflownets. In Interna- tional Conference on Machine Learning, pages 9786-9801. PMLR, 2022.
[30] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6998 2014.
[31] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv:1312.6114 2013.
arXiv preprint
[32] A. Kumar and S. Levine. Model inversion networks for model-based optimization. Advances in Neural Information Processing Systems, 33:5126-5137, 2020.

---

[33] R. Lorenz, S. H. Bernhart, C. Höner zu Siederdissen, H. Tafer, C. Flamm, P. F. Stadler, and I. L. Hofacker. Viennarna package 2.0. Algorithms for molecular biology, 6(1):1-14, 2011.
[34] A. Madani, B. McCann, N. Naik, N. S. Keskar, N. Anand, R. R. Eguchi, P.-. Huang. and R. Socher. Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.033 034997 2020.
[35] M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv:1411.1784, 2014.
arXiv preprint
[36] H. Moss, D. Leslie, D. Beck, Gonzalez, and P. Rayson. Boss: Bayesian optimization over string spaces, Advances in neural information processing systems, 33:15476-15486, 2020.
[37] K. Nigam and R. Ghani. Analyzing the effectiveness and applicability of co-training. In Proceedings of the ninth international conference on Information and knowledge management, pages 86 93, 2000.
[38] E. O. Pyzer- Prapp. Research and Development, 62(6))2-1, 2018.
Bayesian optimization for accelerated drug discovery. IBM Journal of
[39] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.061 0.2000......... 2022.
[40] Z. Ren, J. Li, F. Ding, Y. Zhou, J. Ma, and J. Peng. Proximal exploration for model-guided protein sequence design. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 18520-18536. PMLR, 17- 23 Jul 2022. URL https::p//. : http//ppooeednngeeee/eeee/ee/eee mlr. ppwpps//11//ree222eeee html.
[41] P. Sample, B. Wang, D. W. Reid, V. Presnyak, I. J. McFadyen, D. R. Morris, and G. Seelig. Human 5' utr design and variant effect prediction from massively parallel translation assay. Nature biotechnology. 37(7):803-88) 2019.
[42] S. Sinai, R. Wang, A. Whatley, S. Slocum, E. Locane, and E. D. Kelsic. Adalead: A simple and robust adaptive greedy search algorithm for sequence design. arXiv preprint arXiv:2010.02144, 2020.
[43] K. Terayama, M. Sumita, R. Tamura, and K. Tsuda, Black-box optimization for automated discovery. Accounts of Chemical Research, 54(6):1334-1346 2021.
[44] B. Trabucco, A. Kumar, X. Geng, and S. Levine. Conservative objective models for effective offline model-based optimization. In International Conference Machine Learning, pages 10358-10368. PMLR, 2021.
[45] B. Trabucco, X. Geng, A. Kumar, and S. Levine. Design-bench: Benchmarks for data-driven offline model-based optimization. arXiv preprint arXiv:2202.08450, 2022.
[46] A. Tripp, Daxberger, and J. M. Hernández-Lobatd Sample-efficient optimization in the la- tent space of deep generative models via weighted retraining. Advances in Neural Information Processing Systems, 33:1 1259-112111 11272, 2020.
[47] H. Wang, A. Sakhadeo, A. M. White, J. M. Bell, V. Liu, X. Zhao, P. Liu, T. Kozuno, A. Fyshe, and M. White. No more pesky hyperparameters: Offline hyperparameter tuning for RL. Trans- actions on Machine Learning Research, 2022. URL A10U13440V.
[48] K. Wang, H. Zhao, X. Luo, K. Ren, W. Zhang, and D. Li. Bootstrapped transformer for offline reinforcement learning. arXiv preprint arXiv:2206.08566) 2022.
[49] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3):229-25 226-2,,..66, 1992.
[50] T. Wilson, R. Moriconi, F. Hutter, and M. P. Deisenroth. The reparameterization trick for acquisition functions. arXiv preprint arXiv:1712.00442,, 2017
12

---

[51] Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. Self-training with noisy student improves im- agenet classification. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 10687-10698, 2020.
[52] S. Yu, S. Ahn, L. Song, and J. Shin. Roma: Robust model adaptation for offline model-based optimization. Advances in Neural Information Processing Systems, 111:4199-6311 2021.
[53] Y. Yuan, C. Chen, Z. Liu, W. Neiswanger, and X. Liu. Importance-aware co-teaching for offline model-basee optimization. arXiv preprint arXiv:2309.//600, 2023.
[54] M. Zimmer. Green fluorescent protein (GFP): applications, structure, and related photophysical behavior. Chemical reviews, 102(3):759-72,, 2002.
13

---

A Additional Experimental Settings
Table A.I: Details of the offline datasets. We let [at] and [D] denote the sizes of the search space and the offline dataset, respectively.
Seq. Length
Vocab size
|
[D]
GFP
20
237
20
5,000
UTR
50
504
140,000
TFBind8
4
32,898
RNA-Binding
14
4
114
5,000
A.1 Datasets
GFP [41] is a task to optimize protein sequence of length 237 consisting of one of 20 amino acids, i.e., the search space is 20237. Its objective is to find protein with high fluorescence. Following Trabucco al. [45], we prepare the offline dataset using 5000 samples with 50 to 60 percentile scores in the original data.
UTR [41] task to optimize a DNA sequence length 50 consisting of one of four nucleobases: adenine (A), guanine (G), cytosine (C), thymine (T). Its objective is max- imize the expression level of the corresponding S'UTR region. For the construction of the offline dataset D, following Trabucco et al. [45], provide samples with scores under the
50th percentile data of 140, 000 examples.
TFBind8 [5] is a task that optimizes DNA similar the UTR. The objective is find length 8 sequence to maximize the binding activity with human transcription factors. For the offline dataset we provide under 50th percentile data of 32,898 examples following Trabucco et al. [45].
RNA-Binding [33] a task that optimizes RNA, a sequence that contains four vocab words of nucleobases: adenine (A), uracil (U), cytosine (C), and guanine (G). The objective is to find a length 14 sequence to maximize the binding activity with the target transcription factor. We present three target transcriptions of RNA termed RNA- ANA--Bnding-AA (for L14 RNAI), RNN-Binndngg provide under 0.12 scored data for the offline dataset D among randomly generated 5,000 sequences using open-source code
(for L14 RNA2), and RNA-Binding-C (for L14 RNA3). We
A.2 Implementation of Baselines
This section provides detailed implementation of baselines of offline biological sequence design.
Baselines from Design Bench [45]. Most baselines are from the offline model-based optimiza- tion (MBO) benchmark called design-bench [45]. The design bench contains biological sequence tasks of the GFP, UTR, and TFbind8, where it contains baselines of REINFORCE, CMA-ES [24], BO-qEI [50], CbAS 19), Auto. Cbas [20]. MIN [32], gradient ascent (Grad.), and COMS [44]. We reproduce them by following the official source code For the RNA tasks, follow hyperparam- eters of TFBind8 as the number of vocab are same as 4, and the sequence length similar where the TFBind8 has length 8 and RNA tasks have length 14 as method follows the same.
BDI [12]. For BDI, we follow hyperparameter setting at the paper [12] and implementation at the opensource code 4, For RNA tasks, we follow the hyperparameter for TFBind8 tasks, our method follows the same.
GEN-AL [29]. For GFN-AL we follow hyperparameters setting at the paper [29] and implementa- tion on open-source code". Because they only reported the TFbind8 and the GFP tasks, we use the hyperparameter of the GFP for the UTR tasks hyperparameter of the TFBind8 for RNA tasks, as our method follows the same.
"nttpa://iithub.co
14

---

A.3 Hyperparameters
Table A.2: Hyperparameters. denotes the number of bootstrapping iterations after pretraining. I' denotes the number of pretaining iterations, represents number of samples used inference, stands number of filtered samples among M candidates, and Ngen refers to the number of aggregated generators in the experiments.
r'
M
K
Ngw gen
2,500
12,500
1,280
128
8
Training. We give consistency hyperparameters for all tasks except the learning rate. We the generator's s learning rate to 10-5 for short-length tasks (lengths 8 and 14) of TFBind and RNA tasks and 10-5 for longer-length tasks (lengths 50 and 237) of UTR and GFP. We trained the generator with 12,500 steps before bootstrapping. Bootstrapping is applied with 2, 500 in additional
steps. The batch size of training 256. We set the weighting parameter k 10 -2, Note that carly-stopped the generator iteration of GFP with the 3, 000 step based on monitoring the calibration model of Appendix B. For bootstrapping, the generator samples 2 candidates every 5 steps. For
Top-K sampling at the bootstrapping, we sample with L = 1, 000 and select the Top 2 samples to augment the training dataset.
Testing. For filtering, we generated M = 1, 280 candidate samples and collected the Top-K samples where K 128 based the proxy score. For diverse aggregation, collect K 16 samples from 8 generators, making a total of 128 samples.
Proxy model. For the proxy model, applied weight regularization of 10 set the learning rate to 10- and used dropout rate of 0.1. We used early stopping with a tolerance and train/validate ratio of 9:1 following Jain al. [29]. We used the Adam optimizer [30] for the training generator, proxy, and calibration model.
15

---

B Calibration Model
100th Percentlc
50th Percentle
Calibration model
0.0
0.5
0.7
Calileatiinn 5000
006
Scate
Score
0.7
00.
04
0.3
0.6
0.5
0.4
0.5
12500
15000
7300
10:00
12500
15000
Iteration
Heratiion
(a) TFbind8
(b) GFP
Figure B.I: Calibration model's tendency.
Tuning the hyperparameters of offline design algorithms is challenging due to the lack of access to true score function. Therefore, existing works have proposed various strategies to circumvent this issue, e.g., choosing hyperparameter that is transferrable between different tasks [45] tuning the hyperparameter based on training statistics [52].
In this work, we leverage the calibration function. Inspired by Wang et al. [47], we train the cal- ibration function on the offline dataset to approximate the true score function similar to the proxy function. Then we use the calibration function to select score-conditioned model that achieves higher performance with respect the calibration function. We also choose the number of training steps and early stopping points using the same criterion.
As shown in Fig. B, the calibration model accurately predicts early stopping points the GFP task unstable and has a narrow high score region which gives high chance to be overfitted into the low-scored region (Table 4.1 shows that score of GFP highly polarized). By using the calibration function, we can simply choose an early stopping point for GFP. Note we simply leverage the proxy model calibration model with exact sample training scheme and hyperparameters.
16

---

C Diversity and Novelty Comparison with GFN-AL [29]
Table C.I: Experimental results on 100th percentile scores (100th Per.). 50th percentile scores (50th Per.), average score (Avg. Score), diversity, and novelty, among 128 samples of low dimensional tasks comparing with GFN-AL. The mean and standard deviation of 8 independent runs for producing 128 samples is reported.
The best-scored value is marked in bold.
Methods 100th Per.
50th Per.
Avg. Score
Diversity
Novelty
TFBind8 RNA-C RNA-B RNA-A
GFN-AL
0.630 + 0.054
0.312 + 0.013
0.320 + 0.010
8,858 + 0.045
4.269 + 0.130
BOOTGEN
0.898 0.039
0.694 1 0.009
0.699 1 0.008
5.694
0.008
7.509 1 0.049
BOOTGENt
0.750 0.041
0.382 + 0.014
0.399 + 0.008
8.917
0.078
4.957 + 0.052
GFN-AL
0.677 + 0.080
0.300 + 0.012
0.311 = 0.011
8.846 + 0.050
4.342 = 0,128
BOOTGEN
0.886 + 0.028
0.689 + 0.007
0.693 = 0.007
5.192 = 0.073
7.981 = 0.036
BBoTGEEN
0.686 0.052
0.355 1 0.018
0.371 11 0.017
8.929
0.121
4.967 + 0.132
GFN-AL
0.623 0.045
0.324 + 0.010
0.333 + 0.010
8.831
0.046
4.151 + 0.088
BOOTGEN
0.837 1: 0.045
0.598 1 0.006
0.606 1 0.006
4,451 1 0.071
7.243 + 0.051
BOOTGEN
0.651 + 0.056
0.370 + 0.011
0.376 = 0.013
8,913 = 0.087
4.597 + 0.073
GEN-AL
0.951 0.026
0.537 + 0.055
0.575 + 0.037
5.001
0.178
0.778 + 0.143
BOOTGEN
0.977 + 0.004
0.848 + 0.010
0.839 + 0.009
3.118 + 0.045
1.802 = 0.025
BOOTGEN
0,970 1 0.018
0.613 1 0.017
0.627 1 0.014
5.048 1 0.039
0.965 1 0.033
Table C.2: Experimental results on 100th percentile scores (100th Per.). 50th percentile scores (50th Per.), average score (Avg. Score). diversity, and novelty, among 128 samples of 6 six biological sequential tasks comparing with GFN-AL. The mean and standard deviation of 8 independent runs for producing 128 samples reported. The best-scored value is marked hold. The 'Random' stands for uniform random generator.
Methods 100th Per.
50th Per.
Arg. Score
Diversity
Novelty
Random
0.053 + 0.000
0,051 +:2 0.000
0.051 + 0.000
219.840 + 0.207
216.960 + 0.330 208.610 = 46.271
GEN-AL
0.057 + 0.001
0.051 = 0.004
0.052 + 0.004
130.113 = 41.202
BOOTGEN
0.865 + 0.000
0.854 + 0.002
0.813 + 0.011 7,969 + 0.460
2.801 + 0.163
Building upon the S 4.4 findings of the UTR, we present further multi-objective experimental results for the remaining 5 tasks, comparing them closely with the GFN-AL [29] approach. The GFN-AL model aims to achieve extensive exploration by prioritizing sample diversity and novelty, leading to the generation of diverse, high-quality biological sequences. Nevertheless, the diversity measure occasionally introduces a trade-off between sample scores, particularly when certain tasks exhibit a narrow score landscape, resulting in only limited number of samples with high scores.
We conducted detailed analysis to shed light on the relationship between score metrics (average, 100th percentile, 50th percentile) and diversity metrics (diversity and novelty). The results, pre- sented in Table C.2, demonstrate that our proposed method, BOOTGEN, outperforms GFN-AL in terms of score performance. However, it is noteworthy that GFN-AL exhibits high diversity, par- ticularly in the case of GFP. On the contrary, GEN-AL generates extremely low scores for GFP, almost comparable to those produced by uniform random generator. This observation indicates that the GFP task possesses a narrow score landscape, making it relatively easy to generate diverse yet loo-icoring samples.
For the TFbind8 and RNA tasks, GFN-AL achieves high diversity but a low novelty. This suggests that GFN-AL struggles to discover samples beyond the scope of the offline training dataset, resulting in less novel but diverse samples with low scores. In contrast, BOOTGEN successfully identifies high-scoring and novel samples. Consequently, in this scenario, consider high diversity coupled with low novelty and score to be somewhat meaningless, as such results can also be achieved by a random generator.
To substantiate our claim regarding diversity, we present experimental results of an enhanced di- versity version of BOOTGEN. In order to achieve increased diversity, BOOTGEN makes certain sacrifices terms of score performance. One approach we employ is interpolation with uniform random sequence generator. Specifically, we combine our generator with the uniform random gen-
17

---

erator to generate random samples in portion of the sequence (we make 3/4 samples from the random generator and 1/4 from BOOTGEN). Additionally, we can filter out low-diversity sequences without requiring score evaluation, thereby generating more diverse set of samples by referring code of GFN-AL [29]. To this end, we introduce the diversity-improvel version of our method, denoted as BOOTGENT. It is important to note that BOoTGENt sacrifices score performance, as diversity and score are inherent trade-offs, and it focuses primarily on diversity to provide a more direct comparison with GFN-AL by manually adjusting diversity.
As shown in Table C.2, BOOTGEN! exhibits similar diversity levels in RNA-A, RNA-B, RNA-C, and TFBind8, while achieving higher score metrics and novelty. We attribute these results to GFN's underfitting issue, as it fails adequately fit within the high-scoring region of the score landscape, particularly for the nigh-dimensional tasks of UTR and GFP.
18

---

D Rank-based weighting vs. Value-based weighting
0.80
Soore
0.75
(T=0.7)
0.70
(T=01)
VW (T=1.0)
VW 033:
RO (Ouns
20000
4000
G00D 8000
1000O 12000 14000
Iteration
Figure D.I: Comparison of weighting (RW) and represents the case where no weighting is applied the training distribution. In the VW case, we explored different weighting temperatures, {0.1, 0.3, 0.7, L.0). The 50th percentile scores of TFBind8 are reported, and the results include a bootstrapping procedure applied from iteration 12,500 to 15,000.
weighting (VW) methods. The NW
We verify the contribution of the rank-based weighting (RW) scheme compared weighting (NW) and the existing value-bases weighting (VW) proposed by Kumar and Levine [32]. To implement VW, we set the sample-wise weight proportional exp(ly y"l/T), where y" the maximum score in the training dataset and T E {0.1, 0.3, 0.7, 1.0} is hyperparameter. As shown in Fig. D.I, the results indicate that RW outperforms both NW and VW. This validates our design choice for BOOTGEN.
with
19


Reviews:
[{"review_id": "cAlWaVOkMN", "reviewer": "Reviewer_LsCS", "summary": "The authors propose a novel algorithm:  bootstrapped training of score conditioned generators (BOOTGEN), for the offline design of biological sequences. The key idea is to enhance the score-conditioned generator by suggesting a novel variation of the classical ensemble strategy of bootstrapping and aggregating. The method requires training multiple generators using bootstrapped datasets from training and combining them with proxy models to create a reliable and diverse sampling solution.", "questions": "See above", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "It is interesting the paper proposes a bootstrapping strategy to augment a training dataset with high-scoring samples that are collected from the score-conditioned generator and labeled using a proxy model.", "weaknesses": "Experiments: \n- The choice of baselines: \n   - The majority of baselines seem not biological sequence model specific (biological sequence design, e.g. https://openreview.net/forum?id=HklxbgBKvr and https://arxiv.org/abs/2006.03227, many others. ), instead the baselines are mostly general optimization methods.  Applying a general method to a new application can lead to lower accuracy. The paper can be more convincing to compare with methods target on the same application. \n   - The rank based weighted score idea may be partially inspired by reference [41], which is rank-based weighting scheme for training unconditional generators   See (sec 3.1). Why not compare performance against that score design? \n\n\nIdea: \n\n-  The paper is focused on high scores samples.\nNot including distribution very on median and lower score region, which might harm learning of the entire distribution. The negative samples may be helpful to learn the entire distribution, e.g. those in energy based model\n-  The idea (using sequence-to-score to enhance score-to-sequence) is a relatively common idea. It will be convincing to compare with similar results. \n\nAblation study: \n- plotting the number of generators vs accuracy can be helpful to show the efficacy of the methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Comparison with Biological Sequence Model-specific Baselines\n\nThank you for your recommendation. We have already compared it with the recent state-of-the-art method GFN-AL (\"Biological Sequence Design with GFlowNets\") [1] in the main text, which is an optimization method for biological sequence design. Based on your suggestion, we also made a comparison with the DyNA-PPO (https://openreview.net/forum?id=HklxbgBKvr):\n\n|  | TFBind8 (100th Percentile) | TFBind8 (50th Percentile) |GFP (100th Percentile) | GFP (50th Percentile) |\n| -------- | -------- | -------- |-------- | -------- |\n| DyNa-PPO   | 0.942 $\\pm$ 0.025  |  0.562 $\\pm$ 0.025   |0.790 $\\pm$ 0.003  |  0.790 $\\pm$ 0.005   |\n| Ours   | **0.979 $\\pm$ 0.001**  |  **0.833 $\\pm$ 0.007**   | **0.865 $\\pm$ 0.000** | **0.853 $\\pm$ 0.004**     |\n\nNote many biological sequential design methods, including the papers you suggested (**Dyna-PPO**; https://openreview.net/forum?id=HklxbgBKvr, **P3BO**; https://arxiv.org/abs/2006.03227), rely on online (model-based) optimization, where continuous score function evaluations are required for model updates. In contrast, our benchmark focuses on offline design optimization, where access to the real-time Oracle score function is limited, and only the offline dataset is available for generating high-score samples. \n\n---\n\n### Response for \"Why not compare the performance of an unconditional generator with a score-conditioned generator?\"\n\nThank you for the insightful feedback. Here are the comparison results: \n\n|  | TFBind8 (Avg. Score) |\n| -------- | -------- | \n| RR + Unconditional Generator    | 0.505 $\\pm$ 0.010  |  \n| RR + Score-Conditional Generator   | **0.662 $\\pm$ 0.009**  |\n\nNote the RR stands for rank-based weighting.  Our approach adapts the original rank-based weighting technique, designed for unconditional generators in latent space optimization, to work with score-conditional generators. By leveraging the score-conditional generator's ability to infer high-score regions based on query conditions, we can increment the generator's performance orthogonally. This enhancement allows for a more accurate focus on high-score regions. We have conducted thorough experiments comparing rank-based weighting and high-score conditioning, demonstrating their respective advantages and contributions to the quality of generated samples. These results offer valuable insights into the effectiveness of each approach.\n\n---\n\nThank you for providing additional ideas for this method. \n\n1. **Using lower score region** Our score-conditioned generator, denoted as $p(x|y)$, is trained not only on high-score data but also on low-score data. It aims to map low-score samples to their corresponding data points. By adding weighting to high-score samples, even lower-score samples are still considered probabilistically. The study aims not only to learn an accurate distribution from diverse score-related data but also to bias the distribution towards high-sample scores for design optimization and selecting good samples. It's crucial to consider reviewers' opinions and lower-score samples throughout this process, and I wholeheartedly agree with this approach.\n\n\n2. **Comparing with common idea** First, we already empirically compare with prior common ideas (see Table 1 and Table 2). Our method builds upon the common idea of using sequence-to-score information to enhance score-to-sequence models, as studied by MIN [3] and GFN-AL [1]. In our comparisons with these methods (shown in Tables 1 and 2), our approach outperforms them significantly. The key difference from MIN is using a proxy model for training rather than inference, which is mitigated through diverse aggregation. Compared to GFN-AL, our method generates high scores and avoids underfitting issues in the low-score region. We will further discuss these differences to provide a comprehensive analysis of the strengths and weaknesses of each approach.\n\n---\n\n### Reponse for the additional ablation study\n\n\nThank you for the suggestion. We conducted an ablation study on the number of generators and their performance gain on the UTR task. The results of this study will be included in the main text, Chapter 4.4. For detailed findings, please refer to the attached PDF.\n\n---\n\n### References\n\n[1] Jain, Moksh, et al. \"Biological sequence design with gflownets.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Trabucco, Brandon, et al. \"Design-bench: Benchmarks for data-driven offline model-based optimization.\" International Conference on Machine Learning. PMLR, 2022.\n\n[3] Kumar, Aviral, and Sergey Levine. \"Model inversion networks for model-based optimization.\" Advances in Neural Information Processing Systems 33 (2020): 5126-5137."}, {"review_id": "kHizphZx94", "reviewer": "Reviewer_ogL2", "summary": "The paper proposes BootGen, a model-based sequence optimization algorithm, the author apply to the task of biological sequence design. BootGen has two stages where the first stage trains multiple sequence generators to give higher probability to sequences predicted to have higher scores from a proxy score model. The second stage uses bootstrapping by way of the trained generators to generate more data and train new generators. Samples are then aggregated in a way to encourage diversity and high fitness. BootGen is compared against previous sequence design methods on six biological sequence tasks on which BootGen achieves state-of-the-art results. Ablations show each component of the method -- bootstrapping, filtering, and rank-based reweighting -- to help performance.", "questions": "- I am confused by Algorithm 2 line 2. It says N generators are trained but N is the also the number of training examples. Is this saying the number of generators trained is the same as the number of examples?\n\n- Can the authors put the main hyperparameters in the main text? It is important to know what N, I, M are to be able to put in context how expensive the method is next to the results.\n\n- Appendix A.3 was hard to read. \"Bootstrapping is applied with 2, 500 in additional steps\" what does this mean? It is important to put variables to these numbers and explain where in the method they are used.\n\n- As stated above, was the same proxy model used across all proxy-based baselines? i.e. GFN?\n\n- How are diverse are the generators? I would suspect they all generate similar sequences especialyl given the small dataset. Was this not a problem?\n\n- Can the authors put some representative sequences in the appendix? It is surprising to me the method can get such high fitness in UTR with high novelty, i.e. almost half the sequence changing. Are you sure it's not adversarial examples that are being generated?", "limitations": "The authors mentioned the limitation of a poor proxy that could \"backfire\" and cause BootGen to perform worse. However, this is a general limitation for all proxy based methods. I believe there is a bigger limitation with regards to the run-time of the method with needing lots of samples and lots of training.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The principle behind BootGen is simple: use bootstrapping to distill signal from multiple models into one. The consensus between all the models will reduce variance and improve performance. Seeing that BootGen can outperform arguably more complicated methods is good to see.\n\n- The combination of steps -- ranking, bootstrapping, filtering, and diversity aggregation -- are logical and synergistic. It's not a surprise they help each other.", "weaknesses": "- **The novelty is overstated**. Bootstrapping is a standard statistics idea, score-wise rank weights was proposed in prior work. The idea of filtering and diversity aggregation are not novel either. Adalead for instance will use the proxy model to select its final sequences. Aggregation is simply taking the union over all the samples. I don't believe there is technical novelty other than bringing together existing ideas in a straightforward way. A bioinformatics journal seems more appropriate for this work.\n\n- There seems to be a large jump in performance by using the proxy model to filter the sequences. In fact, the performance of the method seems extremely dependent on the accuracy of the proxy model. It is not stated that the same proxy model was used across all methods in the experiments. **This leads to a unfair comparison**. I believe a fair comparison would require using the same proxy model to isolate the methodological cotributions put forth in BootGen.\n\n- The amount of bootstrapping seems very high. The appendix states 1280 samples are needed from each generator. The number of generators is not specified clearly. The exact number of model evaluations is unclear and this would have been important to analyze. Otherwise it is difficult to tell if the method is working well due to lots of compute over other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive review and feedback. We provide responses for addressing the concerns below. \n\n\n### Response for Novelty\n\nWe made a novel combination of well-known techniques. Our novel high-level algorithm involves (a) propelling the generator to discover novel data points beyond the offline dataset by distilling high-likelihood high-score regions with self-confidence and (b) aggregating samples from multiple self-confident generators to minimize potential risks from adversarial samples. This algorithmic strategy consistently outperforms every baseline across all six benchmarks, highlighting the diversity and novelty of the generated samples. \n\nIt would be greatly appreciated if you could note that numerous machine learning studies continue to explore innovative amalgamations of established techniques, as exemplified by the seminar paper \"Rainbow: Combining Improvements in Deep Reinforcement Learning\" [4]. In a similar vein, prior works like GFN-AL [1] (\"Biological Sequence Design with GFlowNets\") integrate the GFN [5] framework with the concept of active learning (AL) to address biological sequence design challenges. It's important to highlight that this choice could be attributed to several factors while published in ICML rather than a dedicated bioinformatics journal. Firstly, the intricacies of biological sequence design pose a significant challenge for the machine learning community. Secondly, leveraging existing ML techniques for these intricate applications offers a platform for validation and testing. Lastly, the multitude of insights and experimental outcomes from these endeavors can serve as inspiration for the wider ML community.\n\n---\n\n### Unfair Comparison for Proxy Usage\n\nOur proxy model follows the same structure as a recent prior work [1], a simple MLP regressor trained on the offline training dataset. \n\n---\n### Training time for the BootGen\n\nBootGen exhibits impressive training efficiency. Our comprehensive process capitalizes on parallel execution through eight generators, enabling the concurrent generation of 1280 samples in under a second. **Consequently, the impact of bootstrapping on training time remains minimal**. A comparative analysis against the latest offline model-based optimization work, BDI, highlights BootGen's superior speed, as demonstrated in the table below.\n\n\n\n| | Training time| \n| -------- | -------- | \n| BDI  [2]   |  1h    | \n| Ours (1 gen)    | 3min   | \n| Ours (8 gen serially)    | 24min   | \n\nThis experiment is done with a single Nvidie A100 GPU. BootGen's training speed is measured per generator. When utilizing eight generators in series, the total training time amounts to approximately 24 minutes. However, this training time can be significantly reduced by adopting parallel generator training. \n\nIt's worth noting that training time is not the primary focus of offline design optimization methods. In real-world scenarios, the objective evaluation, such as testing protein expressivity or binding activity, often becomes the bottleneck, taking days or even months.\n\n\n\n---\n\n### Answer for each question\n\n\n**A1.** Thank you for the clarification. We will avoid using the letter \"N\" to prevent confusion and explicitly state that we leverage 8 generators.\n\n**A2.** We appreciate the suggestion. We will include the hyperparameters as a table in the main text for better accessibility; see the attached PDF.\n\n**A3.** We acknowledge the oversight and apologize for any confusion. The bootstrapping process involves two samples with 500 iterations each, resulting in a total of 1000 bootstrapped samples. We will revise the explanation and include it in the main text.\n\n**A4.** Understood, and thank you for clarifying. We use the same proxy model as GFN-AL [1] to ensure a fair comparison. While other baselines may have slightly different learning methods for their proxy models, the core architecture remains consistent with a 2-layer MLP with a 2048 width.\n\n**A5.** You are correct. Each generator focuses on specific regions, which may lead to limited diversity compared to a highly diversified generator like GFN. However, the parallel bootstrapping process ensures that each generator concentrates on different regions, contributing to enhanced diversity. When aggregating samples from multiple generators, we can achieve better diversity than GFN-AL [1].\n\n**A6.** Thank you for the suggestion, and we look forward to the representative sequences in the appendix. Regarding adversarial examples in the context of offline design optimization, the definition involves two conditions: (1) the proxy model giving a high score, and (2) the oracle score function providing a low score. In the case of the offline model-based optimization (MBO) benchmark [3], where we assume a given oracle score function (pretrained ResNET for the 280,000 UTR dataset), our situation does not meet the criteria for adversarial examples. Both the proxy function and the oracle function provide high scores.\n\n---\n\n### References\n\n[1] Jain, Moksh, et al. \"Biological sequence design with gflownets.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Chen, Can, et al. \"Bidirectional learning for offline infinite-width model-based optimization.\" Advances in Neural Information Processing Systems 35 (2022): 29454-29467.\n\n[3] Trabucco, Brandon, et al. \"Design-bench: Benchmarks for data-driven offline model-based optimization.\" International Conference on Machine Learning. PMLR, 2022.\n\n[4] Hessel, Matteo, et al. \"Rainbow: Combining improvements in deep reinforcement learning.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.\n\n[5] Bengio, Emmanuel, et al. \"Flow network based generative models for non-iterative diverse candidate generation.\" Advances in Neural Information Processing Systems 34 (2021): 27381-27394."}, {"review_id": "e3ruLvID9l", "reviewer": "Reviewer_jSje", "summary": "This paper proposes to solve the problem of generating novel objects (in this case biological sequences) by learning weighted MLE models, and augmenting the training set of those MLE models with virtual data whose score is based on extrapolations from a proxy.\n\nThis method is tested on standard biological sequence generation problems and found to yield good scores and diversity.", "questions": "I don't have too many questions, but I have a minor concern about the fairness of comparisons; I wonder if all baselines see the same amount of data and receive roughly equivalent amounts of compute. In particular, the BootGen experiments seem to leverage an ensemble of generators (\"we gather cross-aggregated samples from multiple score-conditioned generators\"). Many of the baselines could also be trivially improved via ensembling; is this comparison done?\n\nSmall comment: I'd suggest using shapes (in addition of colors) in Fig 4.2\n", "limitations": "Yes, the authors address limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I think the main strength of this paper is it confirms that extrapolation through MBO-style proxy+generative model that's recently gained popularity still can be improved and explored in a variety of ways.\n\nThe paper is well written and it was easy for me to understand the method.", "weaknesses": "I think the main weakness of this work is that it feels like an aggregation of methods that work well together, rather than a strong single contribution that deeply improves our understanding of MBO.  \nThis is seen fairly well in e.g. Tables 4.3 and 4.4, there's an accumulation of things that improve performance; while it is commendable to improve the performance of ML methods, it's not clear exactly what we've learned from this paper. If there's one way I think the authors could improve this paper it's by either showing or arguing that there is a central contribution, a nugget of knowledge gained by doing these experiments.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for providing a constructive review.\n\n**\n\n**What can we learn from this paper?**\n\nFocusing on fundamental and rigorous approaches is more important in offline design optimization tasks than relying on fancy techniques.\n\nOffline design optimization is inherently challenging as it prohibits access to the Oracle score function during training. As a result, achieving statistically stable and powerful extrapolation over the offline dataset becomes crucial. The main contribution of this study is the reinterpretation/transformation of traditional statistical concepts, such as ensemble and bootstrapping, into deep learning forms for utilization in offline design optimization. The ensemble technique can be employed as an exploration strategy to explore various modes effectively. Bootstrapping demonstrated its effectiveness as an exploitation strategy to reach high-score regions with limited samples. Particularly, through various experiments, it has been proven that the combination of these two simple yet robust concepts can serve as a powerful offline design optimization technique.\n\n1. **Rank-based weighting**: Make a generative model focusing on the high-score regions on the offline dataset\n2. **Score conditioned generator**: Make a conditional generative model to learn to map score to a sequence which learns the relationship between score and sequence and can extrapolate to high score region by leveraging the relationship between low-scored data of the offline dataset\n3. **Bootstrapping using 1 and 2**: Augment training dataset using 1,2 and proxy score function to make the generative model more confident by distillation knowledge of proxy and generator itself.\n4. **Diverse aggregation**: stabilize possible risk from 3 and leverage multiple diversified generators (trained on possible different bootstrapping scenarios) and diversified sampling.\n\nOur algorithm structure seamlessly integrates and can be viewed as a deep learning-based renovation of the classical ensemble approach involving bootstrapping and aggregation.\n\nIn conclusion, our paper demonstrates the importance of carefully approaching and revisiting simple classical schemes within the context of deep learning tasks. This approach proves to be more powerful than relying on fancy techniques, as evidenced by our experiments' clear and compelling results.\n\n\n---\n\n**Answers to the questions**\n\nOur experimental setting ensures fairness in two key aspects. Firstly, all baselines use the same offline dataset, and none of them have access to the Oracle score function, ensuring a level playing field for comparison. Secondly, the computation time is similar for every task, with almost all tasks completed within an hour (our method is notably fast).\n\nCompared with the most recent offline model-based optimization work, BDI [1], BootGen demonstrates faster speed, as shown in the table below.\n\n\n\n| | Training time| \n| -------- | -------- | \n| BDI     |  1h    | \n| Ours (1 gen)    | 3min   | \n| Ours (8 gen)    | 24min   | \n\nThe speed of BootGen is measured per one generator, and we utilize 8 generators in parallel. If we were to measure the speed as serial computation (worst case), the total time taken would be approximately $3 \\times 8 = 24$ minutes.\n\nIt's important to note that computation time is not the main focus in offline black-box optimization tasks since the black-box score function becomes the bottleneck for computation. The primary concern lies in efficiently handling the objective evaluation process, which often requires significant time and resources.\n\n---\n### Response to the review of \"Many of the baselines could also be trivially improved via ensembling.\" \n\nOur method introduces novelty through the ensemble process, which involves creating ensembles from multiple bootstrapping generators. While ensemble techniques are widely used in this field, our approach demonstrates that the direct application of ensembles does not trivially improve performance, as observed in other baselines  [2]. \n\n|  | UTR (100th percentile)|\n| -------- | -------- | \n| Grad.  [2]   | 0.695 $\\pm$ 0.013   | \n| Grad. (Mean ensemble)   [2]  | 0.696 $\\pm$  0.009     | \n| Grad. (Min ensemble)  [2]   | 0.693 $\\pm$ 0.010    | \n| BootGen (w/o DA ensemble)     | 0.729 $\\pm$ 0.074   | \n| BootGen (w DA ensemble)     | 0.858 $\\pm$ 0.003     | \n\nPlease note that \"Grad.\" refers to the gradient ascent method mentioned in [2]. The \"Mean ensemble\" represents an ensemble of proxy models that select the score value based on the mean value among proxies, while the \"Min ensemble\" selects the score value from the minimum value among proxies. The experimental results of the \"Grad.\" method can be found in Table 5 of [2].\n\n\n---\n\nThank you for pointing out the typo; we have now corrected it in the main paper.\n\n---\n\n### References\n\n[1] Chen, Can, et al. \"Bidirectional learning for offline infinite-width model-based optimization.\" Advances in Neural Information Processing Systems 35 (2022): 29454-29467.\n\n[2] Trabucco, Brandon, et al. \"Design-bench: Benchmarks for data-driven offline model-based optimization.\" International Conference on Machine Learning. PMLR, 2022.\n"}, {"review_id": "ddJHQPpgtA", "reviewer": "Reviewer_M7pC", "summary": "This paper introduces a novel algorithm, the Bootstrapped Training of Score-Conditioned Generator (BOOTGEN), to optimize the design of biological sequences. BOOTGEN overcomes the challenge of high-cost evaluations and vast search space by training a score-based generator using rank-based weights and a bootstrapping process. This generator is then augmented with self-generated data labeled by a proxy score function. The process results in diverse and accurate biological sequence designs. The efficacy of BOOTGEN is demonstrated via extensive experiments across six offline biological design tasks.", "questions": "See Weaknesses.", "limitations": "Limitations discussed.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is clear and well-presented. The authors provide a comprehensive introduction and explanation of the BOOTGEN algorithm, making it easily understandable even for those who may not be specialists in this specific field. The experimental results are also clearly laid out and explained, adding to the overall lucidity of the paper.\n\nFurthermore, the extensive experiments and superior results over existing methods reinforce the potential value of this work to both academic researchers and industry practitioners.", "weaknesses": "1. The authors have overlooked a crucial piece of literature, \"Deep Extrapolation for Attribute-Enhanced Generation\" (https://arxiv.org/abs/2107.02968), presented at NeurIPS 2021. This paper also explores protein sequence design and employs a generator/scorer framework. The authors need to acknowledge this work and conduct comparative analysis with it.\n\n2. The first word of the title \"Automatic design of biological sequences\" should be capitalized. This correction should also be made in Section 4.2, where the title reads \"Varying the evaluation budget.\"\n\n3. Claiming the ranking-based weighting as a significant contribution over the value-based weighting is not entirely novel and may not warrant its inclusion as a key contribution.\n\n4. The paper's differentiation from \"Conditioning by adaptive sampling for robust design,\" which also uses a generative model and a proxy for labeling, is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for providing a valuable review. \n\n**W1: About the paper \"Deep Extrapolation for Attribute-Enhance Generation\"**\n\n\nThank you for pointing out the relevant literature. GENhance [1] and our method share the common goal of extrapolating from offline datasets using generative models. BootGen is a high-level algorithm that leverages existing (conditional) generative models. In contrast, GENhance is a novel generative model with significant neural architecture and learning loss. Therefore those works are orthogonal. The combination of GENhance and BootGen seems complementary, presenting a promising direction for future work. We will dedicate a discussion section further to analyze this integration's potential benefits and implications.\n\n---\n\n**W2: About typo**\n\nThank you for pointing out the typo; we will revise it. \n\n---\n\n**W3: Contribution claims of rank-based weighting**\n\nThank you for your understanding. While we acknowledge the differences from applying rank-based weighting to a conditional generator, we agree that this aspect alone may not be the essential contribution. Our principal gifts lie in the novel combination of techniques and the high-level algorithmic structure, which facilitates stable extrapolation for offline biological sequence optimization. We highly value your feedback and will emphasize these core contributions in our paper.\n\n---\n\n**W4: Unclear differentiation from \"Conditioning by adaptive sampling for robust design,**\n\nOur algorithm addresses the challenge of offline black-box optimization through a combination of reweighting processes and extrapolation of queries for score-conditioned models. The objective is to generate novel samples beyond the scope of the offline dataset without relying on an oracle function. To achieve this, we employ an aggregation method to ensure the stability of the process.\n\nThe CbAS [2] method is a representative black-box optimization technique that excels in generating candidate samples to maximize the oracle function in an online setting. However, when applied to offline scenarios, it falls short in performance. Unlike CbAS, which relies on continuous updates from online Oracle queries, our approach leverages reweighting and extrapolation processes to excel in offline settings. Tables 1 and 2 in the main text demonstrate our superior performance across all tasks compared to CbAS.\n\n\n---\n\n### References\n\n[1] Chan, Alvin, et al. \"Deep extrapolation for attribute-enhanced generation.\" Advances in Neural Information Processing Systems 34 (2021): 14084-14096.\n\n[2] Brookes, David, Hahnbeom Park, and Jennifer Listgarten. \"Conditioning by adaptive sampling for robust design.\" International conference on machine learning. PMLR, 2019."}]

Author rebuttal:
Thank you to all the reviewers for providing valuable and constructive feedback on our work. We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript. In response to each reviewer's comments, we have provided a detailed explanation and made necessary revisions to improve the quality of our work. Additionally, we have included a PDF file containing figures that present the results of the additional ablation study, as suggested.

We made two major responses for the two main points as overall comments:

1. **Novelty of this work.**
2. **Experimental fairness in terms of training speed and proxy model.**

---

### Novelty

Our novelty lies in the high-level algorithmic structure that combines simple yet intuitive low-level techniques for effective offline design optimization. Our high-level algorithmic structure is the reinterpretation/transformation of traditional statistical concepts, such as ensemble and bootstrapping, into deep learning forms for utilization in offline design optimization. The ensemble technique can be employed as an exploration strategy to effectively explore various modes while bootstrapping demonstrated its effectiveness as an exploitation strategy to reach high-score regions with limited samples. Particularly, through various experiments, it has been proven that the combination of these two simple yet robust concepts can serve as a powerful offline design optimization technique.

---

### Fairness of experiments

We acknowledge and appreciate the questions raised by some reviewers concerning two aspects of our approach: (a) the training time and (b) the usage of a proxy model, which they believe could potentially introduce unfairness.

However, we made fair experiments because:

1. Our approach exhibits a remarkably short training time, with each generator training process completed in approximately 3 minutes.

2. We used a simple proxy model, precisely the same as prior work [1].

Note that In offline biological sequence optimization tasks, the oracle score function is a significant bottleneck due to its computationally expensive nature; in the case of drug development, this is equivalent to the expensive testing of the drug itself, that can take days or even months. Efficiently addressing this bottleneck is therefore crucial for such offline biological design.

[1] Jain, Moksh, et al. "Biological sequence design with gflownets." International Conference on Machine Learning. PMLR, 2022.


Meta-review: The paper discusses a bootstrapping technique for training of a generator for offline design of biological sequences. Reviewers raised concerns, among others about experiments, ablation studies, overstating of novelty and unfair experimentation. The rebuttal was able to address these concerns. Reviewers arrived at an unanimous accept recommendation. AC doesn't see reasons to overturn an unanimous expert consensus and recommends poster because the paper targets a smaller subset of the NeurIPS community.
Decision: Accept (poster)