Title: Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequences
Venue: NeurIPS 2023
Accepted: yes
Keywords: Biological sequence design

offline model based optimization

conditional generation

bootstrapping

ensemble
Abstract: We study the problem of optimizing biological sequences, e.g., proteins, DNA, and RNA, to maximize a black-box score function that is only evaluated in an offline dataset. We propose a novel solution, bootstrapped training of score-conditioned generator (BootGen) algorithm. Our algorithm repeats a two-stage process. In the first stage, our algorithm trains the biological sequence generator with rank-based weights to enhance the accuracy of sequence generation based on high scores. The subsequent stage involves bootstrapping, which augments the training dataset with self-generated data labeled by a proxy score function. Our key idea is to align the score-based generation with a proxy score function, which distills the knowledge of the proxy score function to the generator. After training, we aggregate samples from multiple bootstrapped generators and proxies to produce a diverse design. Extensive experiments show that our method outperforms competitive baselines on biological sequential design tasks. We provide reproducible source code: https://github.com/kaist-silab/bootgen.

Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequences

Abstract

We study the problem of optimizing biological sequences, e.g., proteins, DNA, and RNA, to maximize a black-box score function that is only evaluated in an offline dataset. We propose a novel solution, bootstrapped training of scoreconditioned generator (BOOTGEN) algorithm. Our algorithm repeats a two-stage process. In the first stage, our algorithm trains the biological sequence generator with rank-based weights to enhance the accuracy of sequence generation based on high scores. The subsequent stage involves bootstrapping, which augments the training dataset with self-generated data labeled by a proxy score function. Our key idea is to align the score-based generation with a proxy score function, which distills the knowledge of the proxy score function to the generator. After training, we aggregate samples from multiple bootstrapped generators and proxies to produce a diverse design. Extensive experiments show that our method outperforms competitive baselines on biological sequential design tasks. We provide reproducible source code: https://github.com/kaist-silab/bootgen.

1 Introduction

The automatic design of biological sequences, e.g., DNA, RNA, and proteins, with a specific property, e.g., high binding affinity, is a vital task within the field of biotechnology [5, 54, 41, 33]. To solve this problem, researchers have developed algorithms to optimize a biological sequence to maximize a score function [40, 9, 10, 2, 29]. Here, the main challenge is the expensive evaluation of the score function that requires experiments in a laboratory setting or clinical trials.

To resolve this issue, recent works have investigated offline model-based optimization [32, 21, 44, 52, 12, 45, MBO]. Given an offline dataset of biological sequences paired with scores, offline MBO algorithms train a proxy for the score function, e.g., a deep neural network (DNN), and maximize the proxy function without querying the true score function. Therefore, such offline MBO algorithms bypass the expense of iteratively querying the true score function whenever a new solution is proposed. However, even optimizing such a proxy function is challenging due to the vast search space over the biological sequences.

On the one hand, several works [21, 44, 52, 12] considered applying gradient-based maximization of the proxy function. However, when the proxy function is parameterized using a DNN, these methods often generate solutions where the true score is low despite the high proxy score. This is due to the fragility of DNNs against adversarial optimization of inputs [52, 44, 21]. Furthermore, the gradient-based methods additionally require reformulating biological sequence optimization as a continuous optimization, e.g., continuous relaxation [21, 44, 12] or mapping discrete designs to a continuous latent space [52].

On the other hand, one may consider training deep generative models to learn a distribution over high-scoring designs [32, 29]. They learn to generate solutions from scratch, which amortizes optimization over the design space.

To be specific, Kumar and Levine [32] suggests learning an inverse map from a score to a solution with a focus on generating high-scoring solutions. Next, Jain et al. [29] proposed training a generative flow network [7, GFN] as the generative distribution of high-scoring solutions.

Contribution In this paper, we propose a bootstrapped training of score-conditioned generator (BOOTGEN) for the offline design of biological sequences. Our key idea is to enhance the scoreconditioned generator by suggesting a variation of the classical ensemble strategy of bootstrapping and aggregating. We train multiple generators using bootstrapped datasets from training and combine them with proxy models to create a reliable and diverse sampling solution.

In the bootstrapped training, we aim to align a score-conditioned generator with a proxy function by bootstrapping the training dataset of the generator. To be specific, we repeat multiple stages of (1) training the conditional generator on the training dataset with a focus on high-scoring sequences and (2) augmenting the training dataset using sequences that are sampled from the generator and labeled using the proxy function. Intuitively, our framework improves the score-to-sequence mapping (generator) to be consistent with the sequence-to-score mapping (proxy function), which is typically more accurate.

When training the score-conditioned generator, we assign high rank-based weights [46] to highscoring sequences. Sequences that are highly ranked among the training dataset are more frequently sampled to train the generator. This leads to shifting the training distribution towards an accurate generation of high-scoring samples. Compared with the value-based weighting scheme previously proposed by Kumar and Levine [32], the rank-based weighting scheme is more robust to the change of training dataset from bootstrapping.

To further boost the performance of our algorithm, we propose two post-processing processes after the training: filtering and diversity aggregation (DA). The filtering process aims to filter samples from generators using the proxy function to gather samples with cross-agreement between the proxy and generator. On the other hand, DA collects sub-samples from multiple generators and combines them into complete samples. DA enables diverse decision-making with reduced variance in generating quality, as it collects samples from multiple bootstrapped generators.

We perform extensive experiments on six offline biological design tasks: green fluorescent protein design [54, GFP], DNA optimization for expression level on an untranslated region [41, UTR], transcription factor binding [5, TFBind8], and RNA optimization for binding to three types of transcription factors [33, RNA-Binding]. Our BOOTGEN demonstrates superior performance, surpassing the 100th percentile score and 50th percentile score of the design-bench baselines [45], a generative flow network (GFN)-based work [29] and bidirectional learning method (BDI) [12]. Furthermore, we additionally verify the superior performance of BOOTGEN in various design scenarios, particularly when given a few opportunities to propose solutions.

2 Related Works

2.1 Automatic Design of Biological Sequences

Researchers have investigated machine learning methods to automatically design biological sequences, e.g., Bayesian optimization [50, 6, 36, 38, 43], evolutionary methods [3, 8, 18, 4, 40], model-based reinforcement learning [2], and generative methods [9, 32, 20, 29, 19, 11, 14]. These methods aim to optimize biological sequence (e.g., protein, RNA, and DNA) for maximizing the target objective of binding activity and folding, which has crucial application in drug discovery and health care [29].

2.2 Offline Model-based Optimization

Offline model-based optimization (MBO) aims to find the design x that maximizes a score function f ( x ) , using only pre-collected offline data. The most common approach is to use gradient-based optimization on differentiable proxy models trained on an offline dataset [49, 44, 52, 21, 12]. How-

𝑝𝜃(𝒙|𝑦 †

)

Generator

𝐿

𝐾

labelling

Top K

Augmented Dataset

y

Density

Step B: Bootstrapping

𝑓 𝜙 (𝒙)

Proxy

Bootstrapping

y

Density

Reweighting

Training

𝑝𝜃(𝒙|𝑦)

Generator

y

Density

Training

Dataset

Reweighted

Dataset

Step A: Rank-based Weighted Training

High Score

𝑦 †

Industrial & Systems Engineering 1 ever, they can lead to poor scores due to the non-smoothness of the proxy landscape. To address this issue, Trabucco et al. [44] proposed conservative objective models (COMs), which use adversarial training to create a smooth proxy. Yu et al. [52] suggests the direct imposition of a Gaussian prior on the proxy model to create a smooth landscape and model adaptation to perform robust estimation on the specific set of candidate design space. While these methods are effective for high-dimensional continuous design tasks, their performance on discrete spaces is often inferior to classical methods, e.g., gradient ascent [45].

2.3 Bootstrapping

Bootstrapping means maximizing the utilization of existing resources. In a narrow sense, it refers to a statistical method where the original dataset is repeatedly sampled to create various datasets [25]. In a broader sense, it also encompasses concepts frequently used in machine learning to improve machine learning scenarios where the label is expensive, e.g., self-training and semi-supervised learning [37, 1, 22]. These methods utilize an iterative training scheme to augment the dataset with self-labeled samples with high confidence.

The bootstrapping strategy at machine learning showed great success in various domains, e.g., fullylabeled classification [51], self-supervised learning [17] and offline reinforcement learning [48]. We introduce a novel bootstrapping strategy utilizing score-conditioned generators and apply it to offline biological sequence design, addressing the challenge of working with a limited amount of poor-quality offline datasets.

2.4 Design by Conditional Generation

Conditional generation is a promising method with several high-impact applications, e.g., classconditional image generation [35], language-to-image generation [39], reinforcement learning [16]. With the success of conditional generation, several studies proposed to use it for design tasks, e.g., molecule and biological sequence design.

Hottung et al. [27] proposed an instance-conditioned variational auto-encoder [31] for routing problems, which can generate near-optimal routing paths conditioned on routing instances. Igashov et al. [28] suggested a conditional diffusion model to generate 3D molecules given their fragments. Specifically, the molecular fragments are injected into the latent space of the diffusion model, and the diffusion model generates links between fragments to make the 3D molecular compound.

3 Bootstrapped Training of Score-Conditioned Generator (BOOTGEN)

Problem definition We are interested in optimizing a biological sequence x to maximize a given score function f ( x ) . We consider an offline setting where, during optimization, we do not have access to the score function f ( x ) . Instead, we optimize the biological sequences using a static dataset D = { ( x n , y n ) } N n =1 consisting of offline queries y n = f ( x n ) to the score function. Finally, we consider evaluating a set of sequences { x m } M m =1 as an output of offline design algorithms.

Overview of BOOTGEN We first provide a high-level description of our bootstrapped training of score-conditioned generator, coined BOOTGEN. Our key idea is to align the score-conditioned generation with a proxy model via bootstrapped training (i.e., we train the generator on sequences labeled using the proxy model) and aggregate the decisions over multiple generators and proxies for reliable and diverse sampling of solutions.

Before BOOTGEN training, we pre-train proxy score function f ϕ ( x ) ≈ f ( x ) only leveraging offline dataset D . After that, our BOOTGEN first initializes a training dataset D tr as the offline dataset D and then repeats the following steps:

BootGen optimizes the score-conditioned generator p θ ( x | y ) using the training dataset D tr . During training, it assigns rank-based weights to each sequence for the generator to focus on high-scoring samples.

BOOTGEN bootstraps the training dataset D tr using samples from the generator p θ ( x | y † ) conditioned on the desired score y † . It uses a proxy f ϕ ( x ) of the score function to label the new samples.

After BOOTGEN training for multiple score-conditioned generators p θ 1 ( x | y ) , ..., p θ n ( x | y ) , we aggregate samples from the generators with filtering of proxy score function f ϕ to generate diverse and reliable samples. We provide the pseudo-code of the overall procedure in Algorithm 1 for training and Algorithm 2 for generating solutions from the trained model.

3.1 Rank-based Weighted Training

Here, we introduce our framework to train the score-conditioned generator. Our algorithm aims to train the score-conditioned generator with more focus on generating high-scoring designs. Such a goal is helpful for bootstrapping and evaluation of our framework, where we query the generator conditioned on a high score.

Given a training dataset D tr, our BOOTGEN minimizes the following loss function:

where w ( y, D tr ) is the score-wise rank-based weight [46]. Here, k is a weight-shifting factor, and rank( y, D tr ) denotes the relative ranking of a score y with respect to the set of scores in the dataset D tr. We note that a small weight-shifting factor k assigns high weights to high-scoring samples.

For mini-batch training of the score-conditioned generator, we approximate the loss function L ( θ ) via sampling with probability w ( y, D tr ) for each sample ( x , y ) .

We note that Tripp et al. [46] proposed the rank-based weighting scheme for training unconditional generators to solve online design problems. At a high level, the weighting scheme guides the generator to focus more on generating high-scoring samples. Compared to weights that are proportional to scores [32], using the rank-based weights promotes the training to be more robust against outliers, e.g., samples with abnormally high weights. To be specific, the weighting factor w ( y, D ) is less affected by outliers due to its upper bound that is achieved when rank( y, D ) = 1 .

Algorithm 2 Aggregation Strategy for Sample Generation

1: Input: Trained score-conditioned generators p θ 1 ( x | y ) , ..., p θ N gen ( x | y ) , and trained proxy function f ϕ ( x ) . 2: Initialize D samples ←∅ . 3: for i = 1 , . . . , N gen do 4: Sample x ∗ m ∼ p θ i ( x | y † ) for m ∈ [ M ] . 5: Set y ∗ m ← f ϕ ( x ∗ m ) for m ∈ [ M ] . 6: Set D sub-samples as TopK scoring samples in { x ∗ m , y ∗ m } M m =1 . ▷ Filtering 7: Set D samples ←D samples ∪ D sub-samples ▷ Diversity Aggregation 8: end for 9: Output: D samples .

3.2 Bootstrapping

Next, we introduce our bootstrapping strategy to augment a training dataset with high-scoring samples that are collected from the score-conditioned generator and labeled using a proxy model. Our key idea is to enlarge the dataset so that the score-conditioned generation is consistent with predictions of the proxy model, in particular for the high-scoring samples. This enables self-training by utilizing the extrapolation capabilities of the generator and allows the proxy model to transfer its knowledge to the score-conditioned generation process.

Wefirst generate a set of samples x ∗ 1 , . . . , x ∗ L from the generator p θ ( x | y † ) conditioned on the desired score y † 1 Then we compute the corresponding labels y ∗ 1 , . . . , y ∗ L using the proxy model, i.e., we set y ℓ = f ϕ ( x ℓ ) for ℓ = 1 , . . . , L . Finally, we augment the training dataset using the set of topK samples D aug with respect to the proxy model, i.e., we set D tr ∪D aug as the new training dataset D tr .

3.3 Aggregation Strategy for Sample Generation

Here, we introduce additional post-hoc aggregation strategies that can be used to further boost the quality of samples from our generator. See Algorithm 2 for a detailed process.

Filtering We follow Kumar and Levine [32] to exploit the knowledge of the proxy function for filtering high-scoring samples from the generator. To be specific, when evaluating our model, we sample a set of candidate solutions and select the top samples with respect to the proxy function.

Diverse aggregation To enhance the diversity of candidate samples while maintaining reliable generating performances with low variance, we gather cross-aggregated samples from multiple score-conditioned generators. These generators are independently trained using our proposed bootstrapped training approach. Since each bootstrapped training process introduces high randomness due to varying training datasets, combining the generative spaces of multiple generators yields a more diverse space compared to a single generator.

Moreover, this process helps reduce the variance in generating quality. By creating ensemble candidate samples from multiple generators, we ensure stability and mitigate the risk of potential failure cases caused by adversarial samples. These samples may receive high scores from the proxy function but have low actual scores. This approach resembles the classical ensemble strategy known as 'bagging,' which aggregates noisy bootstrapped samples from decision trees to reduce variances.

4 Experiments

We present experimental results on six representative biological sequence design tasks to verify the effectiveness of the proposed method. We also conduct ablation studies to verify the effectiveness of each component in our method. For training, we use a single GPU of NVIDIA A100, where the training time of one generator is approximately 10 minutes.

4.1 Experimental Setting

Tasks. We evaluate an offline design algorithm by (1) training it on an offline dataset and (2) using it to generate 128 samples for high scores. We measure the 50 th percentile and 100 th percentile scores of the generated samples. All the results are measured using eight independent random seeds.

We consider six biological sequence design tasks: green fluorescent protein (GFP), DNA optimization for expression level on untranslated region (UTR), DNA optimization tasks for transcription factor binding (TFBind8), and three RNA optimization tasks for transcription factor binding (RNABinding-A, RNA-Binding-B, and RNA-Binding-C). The scores of the biological sequences range in [0 , 1] . We report the statistics of the offline datasets used for each task in Table A.1. We also provide a detailed description of the tasks in Appendix A.1.

Baselines We compare our BOOTGEN with the following baselines: gradient ascent with respect to a proxy score model [45, Grad.], REINFORCE [49], Bayesian optimization quasi-expectedimprovement [50, BO-qEI], covariance matrix adaptation evolution strategy [24, CMA-ES], conditioning by adaptative sampling [9, CbAS], autofocused CbAS [20, Auto. CbAS], model inversion network [32, MIN], where these are in the official design bench [45]. We compare with additional baselines of conservative objective models [44, COMs], generative flow network for active learning [29, GFN-AL] and bidirectional learning [12, BDI].

Implementation We parameterize the conditional distribution p θ ( x t | x 1: t -1 , y ) using a 2 -layer long short-term memory [26, LSTM] network with 512 hidden dimensions. The condition y is injected into the LSTM using a linear projection layer. We parameterize the proxy model using a multi-layer perceptron (MLP) with 2048 hidden dimensions and a sigmoid activation function. Our parameterization is consistent across all the tasks. We provide a detailed description of the hyperparameters in Appendix A. We also note the importance of the desired score y † to condition during bootstrapping and evaluation. In this regard, we set it as the maximum score that is achievable for the given problem, i.e., we set y † = 1 . We assume that such a value is known following [12].

0

20

40

60

80

100

120

Number of evaluations (

K

)

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

Score

BootGen

Grad.

BDI

CbAS

CMA-ES

(a) RNA-Binding-A

0

20

40

60

80

100

120

Number of evaluations (

K

)

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

Score

BootGen

Grad.

BDI

CbAS

CMA-ES

(b) RNA-Binding-B

0

20

40

60

80

100

120

Number of evaluations (

K

)

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

Score

BootGen

Grad.

BDI

CbAS

CMA-ES

(c) RNA-Binding-C

0

20

40

60

80

100

120

Number of evaluations (

K

)

0

.

800

0

.

825

0

.

850

0

.

875

0

.

900

0

.

925

0

.

950

0

.

975

1

.

000

Score

BootGen

Grad.

BDI

CbAS

CMA-ES

(d) TFBind8

0

20

40

60

80

100

120

Number of evaluations (

K

)

0

.

856

0

.

858

0

.

860

0

.

862

0

.

864

Score

BootGen

Grad.

BDI

CbAS

CMA-ES

(e) GFP

0

20

40

60

80

100

120

Number of evaluations (

K

)

0

.

50

0

.

55

0

.

60

0

.

65

0

.

70

0

.

75

0

.

80

0

.

85

0

.

90

Score

BootGen

Grad.

BDI

CbAS

CMA-ES

(f) UTR

4.2 Performance Evaluation

In Table 4.1 and Table 4.2, we report the performance of our BOOTGEN along with other baselines. One can observe how our BOOTGEN consistently outperforms the considered baselines across all six tasks. In particular, one can observe how our BOOTGEN achieves large gains in 50 th percentile metrics. This highlights how our algorithm is able to create a reliable set of candidates.

For TFbind8, which has a relatively small search space ( 4 8 ), having high performances on the 100th percentile is relatively easy. Indeed, the classical method of CMA-ES and Grad. gave pretty good performances. However, for the 50th percentile score, a metric for measuring the method's reliability, BDI outperformed previous baselines by a large margin. Our method outperformed even BDI and achieved an overwhelming score.

For higher dimensional tasks of UTR, even the 50th percentile score of BOOTGEN outperforms the 100th percentile score of other baselines by a large margin. We note that our bootstrapping strategies and aggregation strategy greatly contributed to improving performances on UTR. For additional tasks of RNA, we achieved the best score for both the 50th percentile and the 100th percentile. This result verifies that our method is task-expandable.

4.3 Varying the Evaluation Budget

In real-world scenarios, there may be situations where only a few samples can be evaluated due to the expensive score function. For example, in an extreme scenario, for the clinical trial of a new protein drug, there may be only one or two chances to be evaluated. As we measure the 50th percentile and 100th percentile score among 128 samples following the design-bench [45] at Tables 4.1 and 4.2, we also provide a 100th percentile score report at the fewer samples from 1 sample to the 128 samples to evaluate the model's robustness on the low-budget evaluation scenarios.

To account for this, we also provide a budget-performance graph that compares the performance of our model to the baselines using different numbers of evaluations. This allows us to observe the trade-off between performance and the number of samples generated. Note that we select baselines as the Top 5 methods in terms of average percentile 100 scores reported at Table 4.1.

22

24

26

28

30

32

34

0

.

50

0

.

55

0

.

60

0

.

65

0

.

70

Avg. Score

BootGen

GFN-AL

Grad.

CMA-ES

CbAS

Auto. CbAS

MIN

REINFORCE

BDI

16

17

18

19

20

21

22

0

.

50

0

.

55

0

.

60

0

.

65

0

.

70

Avg. Score

BootGen

GFN-AL

Grad.

CMA-ES

CbAS

Auto. CbAS

MIN

REINFORCE

BDI

Diversity

Novelty

As shown in Fig. 4.1, our method outperforms every baseline for almost every evaluation budget. For the UTR task, our performance on a single evaluation budget gives a better score than the other baselines' scores when they have a budget of 128 evaluations. For RNA tasks, our method with an approximate budget of 30 achieves superior performance compared to other methods with a budget of 128. These results show that our method is the most reliable as its performance is most robust when the evaluation budget is limited.

4.4 Average Score with Diversity

For biological sequence design, measuring the diversity and novelty of the generated sequence is also crucial [29]. Following the evaluation metric of [29] we compare the performance of models in terms of diversity and novelty.

Here is measurement of diversity for sampled design dataset D = { x 1 , ..., x M } from generator which is average of Levenshtein distance [23], denoted by d ( x i , x j ) , between arbitrary two biological sequences x i , x j from the generated design candidates D :

Next, we measure the minimum distance from the offline dataset D offline which measures the novelty of generated design candidates D as:

Our method surpasses all baselines, including GFN-AL [29], in the UTR task, as evidenced by the Pareto frontier depicted in Table 4.3 and Fig. 4.2. Given the highly dimensional nature of the UTR task and its expansive search space, the discovery of novel and diverse candidates appears to be directly related to their average score. This implies that extensive exploration of the highdimensional space is crucial for improving scores in the UTR task.

It is worth noting that GFN-AL, which is specifically designed to generate diverse, high-quality samples through an explorative policy, secures a second place for diversity. Although GFN-AL occasionally exhibits better diversity than our method and achieves a second-place average score, it consistently delivers poor average scores in the GFP and RNA tasks Table 4.2. This drawback can be attributed to its high explorative policy, which necessitates focused exploration in narrow regions. In contrast, BOOTGEN consistently produces reliable scores across all tasks Table 4.2. For a comprehensive comparison with GFN-AL, please refer to the additional experiments presented in Appendix C.

Diverse aggregation strategy Our diverse aggregation (DA) strategy significantly enhances diversity, novelty, and score variance, as demonstrated in Table 4.3. This is especially beneficial for the UTR task, which necessitates extensive exploration of a vast solution space, posing a substantial risk to the bootstrapped training process. In this context, certain bootstrapped generators may yield exceedingly high scores, while others may produce low scores due to random exploration scenarios. By employing DA, we combine multiple generators to generate candidate samples, thereby greatly stabilizing the quality of the bootstrapped generator.

4.5 Ablation study

The effectiveness of our components, namely rank-based reweighting (RR), bootstrapping (B), and filtering (F), in improving performance is evident in Table 4.4. Across all tasks, these components consistently contribute to performance enhancements. The bootstrapping process is particularly more beneficial for high-dimensional tasks like UTR and GFP. This correlation is intuitive since high-dimensional tasks require a larger amount of data for effective exploration. The bootstrapped training dataset augmentation facilitates this search process by leveraging proxy knowledge. Additionally, the filtering technique proves to be powerful in improving scores. As we observed from the diverse aggregation and filtering, the ensemble strategy greatly enhances score-conditioned generators.

5 Future Works

Enhancing proxy robustness While our bootstrapping method shows promise for offline biosequential design tasks, it has inherent technical limitations. The assumption that the generator produces superior data to the training dataset may backfire if the generator samples have poor quality designs and the proxy used is inaccurate. While the current aggregation strategy effectively manages this risk, we can address this limitation by utilizing robust learning methods of proxies such as conservative proxies modeling [44], robust model adaptation techniques [52], parallel mentoring proxies [13], and importance-aware co-teaching of proxies [53] for further improvement.

Enhancing architecture of BOOTGEN Our approach primarily employs a straightforward architectural framework, with a primary emphasis on validating its algorithmic structures in the context of offline biological sequence design. To enhance the practical utility of our method, it will be advantageous to incorporate established and robust architectural paradigms, exemplified in works such as [11] and [14], into the framework of our method. One promising avenue for achieving this integration is the incorporation of pre-trained protein language models (pLMs) [34, 15], akin to those expounded upon in [14].

6 Conclusion

This study introduces a novel approach to stabilize and enhance score-conditioned generators for offline biological sequence design, incorporating the classical concepts of bootstrapping and aggregation. Our novel method, named BOOTGEN, consistently outperformed all baselines across six offline biological sequence design tasks, encompassing RNA, DNA, and protein optimization. Our strategy of bootstrapping and aggregation yielded remarkable improvements in achieving high scores, generating diverse samples, and minimizing performance variance.

References

M.-R. Amini and P. Gallinari. Semi-supervised logistic regression. In ECAI , volume 2, page 11, 2002.

C. Angermueller, D. Dohan, D. Belanger, R. Deshpande, K. Murphy, and L. Colwell. Modelbased reinforcement learning for biological sequence design. In International conference on learning representations , 2019.

F. H. Arnold. Design by directed evolution. Accounts of chemical research , 31(3):125-131, 1998.

F. H. Arnold. Directed evolution: bringing new chemistry to life. Angewandte Chemie International Edition , 57(16):4143-4148, 2018.

L. A. Barrera, A. Vedenko, J. V. Kurland, J. M. Rogers, S. S. Gisselbrecht, E. J. Rossin, J. Woodard, L. Mariani, K. H. Kock, S. Inukai, et al. Survey of variation in human transcription factors reveals prevalent dna binding changes. Science , 351(6280):1450-1454, 2016.

D. Belanger, S. Vora, Z. Mariet, R. Deshpande, D. Dohan, C. Angermueller, K. Murphy, O. Chapelle, and L. Colwell. Biological sequences design using batched bayesian optimization. 2019.

E. Bengio, M. Jain, M. Korablyov, D. Precup, and Y. Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems , 34:27381-27394, 2021.

J. D. Bloom and F. H. Arnold. In the light of directed evolution: pathways of adaptive protein evolution. Proceedings of the National Academy of Sciences , 106(supplement_1):9995-10000, 2009.

D. Brookes, H. Park, and J. Listgarten. Conditioning by adaptive sampling for robust design. In International conference on machine learning , pages 773-782. PMLR, 2019.

D. H. Brookes and J. Listgarten. Design by adaptive sampling. arXiv preprint arXiv:1810.03714 , 2018.

A. Chan, A. Madani, B. Krause, and N. Naik. Deep extrapolation for attribute-enhanced generation. Advances in Neural Information Processing Systems , 34:14084-14096, 2021.

C. Chen, Y. Zhang, J. Fu, X. Liu, and M. Coates. Bidirectional learning for offline infinitewidth model-based optimization. In Advances in Neural Information Processing Systems , 2022.

C. Chen, C. Beckham, Z. Liu, X. Liu, and C. Pal. Parallel-mentoring for offline model-based optimization. arXiv preprint arXiv:2309.11592 , 2023.

C. Chen, Y. Zhang, X. Liu, and M. Coates. Bidirectional learning for offline model-based biological sequence design. arXiv preprint arXiv:2301.02931 , 2023.

C. Chen, J. Zhou, F. Wang, X. Liu, and D. Dou. Structure-aware protein self-supervised learning. Bioinformatics , 2023.

L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems , 34:15084-15097, 2021.

T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems , 33: 22243-22255, 2020.

P. A. Dalby. Strategy and success for the directed evolution of enzymes. Current opinion in structural biology , 21(4):473-480, 2011.

C. Ekbote, M. Jain, P. Das, and Y. Bengio. Consistent training via energy-based gflownets for modeling discrete joint distributions. arXiv preprint arXiv:2211.00568 , 2022.

C. Fannjiang and J. Listgarten. Autofocused oracles for model-based design. Advances in Neural Information Processing Systems , 33:12945-12956, 2020.

J. Fu and S. Levine. Offline model-based optimization via normalized maximum likelihood estimation. arXiv preprint arXiv:2102.07970 , 2021.

Y. Grandvalet and Y. Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems , 17, 2004.

R. Haldar and D. Mukhopadhyay. Levenshtein distance technique in dictionary lookup methods: An improved approach. arXiv preprint arXiv:1101.1232 , 2011.

N. Hansen. The CMA evolution strategy: a comparing review. Towards a new evolutionary computation , pages 75-102, 2006.

T. Hesterberg. Bootstrap. Wiley Interdisciplinary Reviews: Computational Statistics , 3(6): 497-526, 2011.

S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation , 9(8):17351780, 1997.

A. Hottung, B. Bhandari, and K. Tierney. Learning a latent search space for routing problems using variational autoencoders. In International Conference on Learning Representations , 2020.

I. Igashov, H. Stärk, C. Vignac, V. G. Satorras, P. Frossard, M. Welling, M. Bronstein, and B. Correia. Equivariant 3d-conditional diffusion models for molecular linker design. arXiv preprint arXiv:2210.05274 , 2022.

M. Jain, E. Bengio, A. Hernandez-Garcia, J. Rector-Brooks, B. F. Dossou, C. A. Ekbote, J. Fu, T. Zhang, M. Kilgour, D. Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning , pages 9786-9801. PMLR, 2022.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

A. Kumar and S. Levine. Model inversion networks for model-based optimization. Advances in Neural Information Processing Systems , 33:5126-5137, 2020.

R. Lorenz, S. H. Bernhart, C. Höner zu Siederdissen, H. Tafer, C. Flamm, P. F. Stadler, and I. L. Hofacker. Viennarna package 2.0. Algorithms for molecular biology , 6(1):1-14, 2011.

A. Madani, B. McCann, N. Naik, N. S. Keskar, N. Anand, R. R. Eguchi, P.-S. Huang, and R. Socher. Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.03497 , 2020.

M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 , 2014.

H. Moss, D. Leslie, D. Beck, J. Gonzalez, and P. Rayson. Boss: Bayesian optimization over string spaces. Advances in neural information processing systems , 33:15476-15486, 2020.

K. Nigam and R. Ghani. Analyzing the effectiveness and applicability of co-training. In Proceedings of the ninth international conference on Information and knowledge management , pages 86-93, 2000.

E. O. Pyzer-Knapp. Bayesian optimization for accelerated drug discovery. IBM Journal of Research and Development , 62(6):2-1, 2018.

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Z. Ren, J. Li, F. Ding, Y. Zhou, J. Ma, and J. Peng. Proximal exploration for model-guided protein sequence design. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 18520-18536. PMLR, 1723 Jul 2022. URL https://proceedings.mlr.press/v162/ren22a.html .

P. J. Sample, B. Wang, D. W. Reid, V. Presnyak, I. J. McFadyen, D. R. Morris, and G. Seelig. Human 5' utr design and variant effect prediction from a massively parallel translation assay. Nature biotechnology , 37(7):803-809, 2019.

S. Sinai, R. Wang, A. Whatley, S. Slocum, E. Locane, and E. D. Kelsic. Adalead: A simple and robust adaptive greedy search algorithm for sequence design. arXiv preprint arXiv:2010.02141 , 2020.

K. Terayama, M. Sumita, R. Tamura, and K. Tsuda. Black-box optimization for automated discovery. Accounts of Chemical Research , 54(6):1334-1346, 2021.

B. Trabucco, A. Kumar, X. Geng, and S. Levine. Conservative objective models for effective offline model-based optimization. In International Conference on Machine Learning , pages 10358-10368. PMLR, 2021.

B. Trabucco, X. Geng, A. Kumar, and S. Levine. Design-bench: Benchmarks for data-driven offline model-based optimization. arXiv preprint arXiv:2202.08450 , 2022.

A. Tripp, E. Daxberger, and J. M. Hernández-Lobato. Sample-efficient optimization in the latent space of deep generative models via weighted retraining. Advances in Neural Information Processing Systems , 33:11259-11272, 2020.

H. Wang, A. Sakhadeo, A. M. White, J. M. Bell, V. Liu, X. Zhao, P. Liu, T. Kozuno, A. Fyshe, and M. White. No more pesky hyperparameters: Offline hyperparameter tuning for RL. Transactions on Machine Learning Research , 2022. URL https://openreview.net/forum?id= AiOUi3440V .

K. Wang, H. Zhao, X. Luo, K. Ren, W. Zhang, and D. Li. Bootstrapped transformer for offline reinforcement learning. arXiv preprint arXiv:2206.08569 , 2022.

R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3):229-256, 1992.

J. T. Wilson, R. Moriconi, F. Hutter, and M. P. Deisenroth. The reparameterization trick for acquisition functions. arXiv preprint arXiv:1712.00424 , 2017.

Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10687-10698, 2020.

S. Yu, S. Ahn, L. Song, and J. Shin. Roma: Robust model adaptation for offline model-based optimization. Advances in Neural Information Processing Systems , 34:4619-4631, 2021.

Y. Yuan, C. Chen, Z. Liu, W. Neiswanger, and X. Liu. Importance-aware co-teaching for offline model-based optimization. arXiv preprint arXiv:2309.11600 , 2023.

M. Zimmer. Green fluorescent protein (GFP): applications, structure, and related photophysical behavior. Chemical reviews , 102(3):759-782, 2002.

A Additional Experimental Settings

A.1 Datasets

GFP [41] is a task to optimize a protein sequence of length 237 consisting of one of 20 amino acids, i.e., the search space is 20 237 . Its objective is to find a protein with high fluorescence. Following Trabucco et al. [45], we prepare the offline dataset using 5000 samples with 50 to 60 percentile scores in the original data.

UTR [41] is a task to optimize a DNA sequence of length 50 consisting of one of four nucleobases: adenine (A), guanine (G), cytosine (C), thymine (T). Its objective is to maximize the expression level of the corresponding 5'UTR region. For the construction of the offline dataset D , following Trabucco et al. [45], we provide samples with scores under the 50th percentile data of 140 , 000 examples.

TFBind8 [5] is a task that optimizes DNA similar to the UTR. The objective is to find a length 8 sequence to maximize the binding activity with human transcription factors. For the offline dataset D , we provide under 50th percentile data of 32,898 examples following Trabucco et al. [45].

RNA-Binding [33] is a task that optimizes RNA, a sequence that contains four vocab words of nucleobases: adenine (A), uracil (U), cytosine (C), and guanine (G). The objective is to find a length 14 sequence to maximize the binding activity with the target transcription factor. We present three target transcriptions of RNA termed RNA-Binding-A (for L14 RNA1), RNA-Binding-B (for L14 RNA2), and RNA-Binding-C (for L14 RNA3). We provide under 0.12 scored data for the offline dataset D among randomly generated 5,000 sequences using open-source code 2 .

A.2 Implementation of Baselines

This section provides a detailed implementation of baselines of offline biological sequence design.

Baselines from Design Bench [45]. Most baselines are from the offline model-based optimization (MBO) benchmark called design-bench [45]. The design bench contains biological sequence tasks of the GFP, UTR, and TFbind8, where it contains baselines of REINFORCE, CMA-ES [24], BO-qEI [50], CbAS [9], Auto. Cbas [20], MIN [32], gradient ascent (Grad.), and COMS [44]. We reproduce them by following the official source code 3 . For the RNA tasks, we follow hyperparameters of TFBind8 as the number of vocab are same as 4, and the sequence length is similar where the TFBind8 has length 8 and RNA tasks have length 14 as our method follows the same.

BDI [12]. For BDI, we follow hyperparameter setting at the paper [12] and implementation at the opensource code 4 . For RNA tasks, we follow the hyperparameter for TFBind8 tasks, as our method follows the same.

GFN-AL [29]. For GFN-AL we follow hyperparameters setting at the paper [29] and implementation on open-source code 5 . Because they only reported the TFbind8 and the GFP tasks, we use the hyperparameter of the GFP for the UTR tasks hyperparameter of the TFBind8 for RNA tasks, as our method follows the same.

A.3 Hyperparameters

Training. We give consistency hyperparameters for all tasks except the learning rate. We set the generator's learning rate to 10 -5 for short-length tasks (lengths 8 and 14) of TFBind and RNA tasks and 5 × 10 -5 for longer-length tasks (lengths 50 and 237) of UTR and GFP. We trained the generator with 12,500 steps before bootstrapping. Bootstrapping is applied with 2 , 500 in additional steps. The batch size of training is 256. We set the weighting parameter k = 10 -2 . Note that we early-stopped the generator iteration of GFP with the 3 , 000 step based on monitoring the calibration model of Appendix B. For bootstrapping, the generator samples 2 candidates every 5 steps. For Top-K sampling at the bootstrapping, we sample with L = 1 , 000 and select the Top 2 samples to augment the training dataset.

Testing. For filtering, we generated M = 1 , 280 candidate samples and collected the Top-K samples where K = 128 based on the proxy score. For diverse aggregation, we collect K = 16 samples from 8 generators, making a total of 128 samples.

Proxy model. For the proxy model, we applied a weight regularization of 10 -4 , set the learning rate to 10 -4 , and used a dropout rate of 0.1. We used early stopping with a tolerance of 5 and a train/validate ratio of 9:1 following Jain et al. [29]. We used the Adam optimizer [30] for the training generator, proxy, and calibration model.

B Calibration Model

100th Percentile

50th Percentile

Calibration model

0

2500

5000

7500

10000

12500

15000

Iteration

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

Score

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

Calibration score

(a) TFbind8

0

2500

5000

7500

10000

12500

15000

Iteration

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Score

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

Calibration score

(b) GFP

Tuning the hyperparameters of offline design algorithms is challenging due to the lack of access to the true score function. Therefore, existing works have proposed various strategies to circumvent this issue, e.g., choosing a hyperparameter that is transferrable between different tasks [45] or tuning the hyperparameter based on training statistics [52].

In this work, we leverage the calibration function. Inspired by Wang et al. [47], we train the calibration function on the offline dataset to approximate the true score function similar to the proxy function. Then we use the calibration function to select a score-conditioned model that achieves higher performance with respect to the calibration function. We also choose the number of training steps and early stopping points using the same criterion.

As shown in Fig. B.1, the calibration model accurately predicts early stopping points as the GFP task is unstable and has a narrow high score region which gives a high chance to be overfitted into the low-scored region (Table 4.1 shows that score of GFP is highly polarized). By using the calibration function, we can simply choose an early stopping point for GFP. Note we simply leverage the proxy model as a calibration model with an exact sample training scheme and hyperparameters.

C Diversity and Novelty Comparison with GFN-AL [29]

Building upon the § 4.4 findings of the UTR, we present further multi-objective experimental results for the remaining 5 tasks, comparing them closely with the GFN-AL [29] approach. The GFN-AL model aims to achieve extensive exploration by prioritizing sample diversity and novelty, leading to the generation of diverse, high-quality biological sequences. Nevertheless, the diversity measure occasionally introduces a trade-off between sample scores, particularly when certain tasks exhibit a narrow score landscape, resulting in only a limited number of samples with high scores.

We conducted a detailed analysis to shed light on the relationship between score metrics (average, 100th percentile, 50th percentile) and diversity metrics (diversity and novelty). The results, presented in Table C.2, demonstrate that our proposed method, BOOTGEN, outperforms GFN-AL in terms of score performance. However, it is noteworthy that GFN-AL exhibits high diversity, particularly in the case of GFP. On the contrary, GFN-AL generates extremely low scores for GFP, almost comparable to those produced by a uniform random generator. This observation indicates that the GFP task possesses a narrow score landscape, making it relatively easy to generate diverse yet low-scoring samples.

For the TFbind8 and RNA tasks, GFN-AL achieves high diversity but a low novelty. This suggests that GFN-AL struggles to discover samples beyond the scope of the offline training dataset, resulting in less novel but diverse samples with low scores. In contrast, BOOTGEN successfully identifies high-scoring and novel samples. Consequently, in this scenario, we consider high diversity coupled with low novelty and score to be somewhat meaningless, as such results can also be achieved by a random generator.

To substantiate our claim regarding diversity, we present experimental results of an enhanced diversity version of BOOTGEN. In order to achieve increased diversity, BOOTGEN makes certain sacrifices in terms of score performance. One approach we employ is interpolation with a uniform random sequence generator. Specifically, we combine our generator with the uniform random gen-

erator to generate random samples in a portion of the sequence (we make 3 / 4 samples from the random generator and 1 / 4 from BOOTGEN). Additionally, we can filter out low-diversity sequences without requiring score evaluation, thereby generating a more diverse set of samples by referring code of GFN-AL [29]. To this end, we introduce the diversity-improved version of our method, denoted as BOOTGEN † . It is important to note that BOOTGEN † sacrifices score performance, as diversity and score are inherent trade-offs, and it focuses primarily on diversity to provide a more direct comparison with GFN-AL by manually adjusting diversity.

As shown in Table C.2, BOOTGEN † exhibits similar diversity levels in RNA-A, RNA-B, RNA-C, and TFBind8, while achieving higher score metrics and novelty. We attribute these results to GFN's underfitting issue, as it fails to adequately fit within the high-scoring region of the score landscape, particularly for the high-dimensional tasks of UTR and GFP.

D Rank-based weighting vs. Value-based weighting

0

2000

4000

6000

8000

10000 12000 14000

Iteration

0

.

65

0

.

70

0

.

75

0

.

80

0

.

85

Score

NW

VW (T=0.1)

VW (T=0.3)

VW (T=0.7)

VW (T=1.0)

RW (ours)

We verify the contribution of the rank-based weighting (RW) scheme compared to ours with no weighting (NW) and the existing value-based weighting (VW) proposed by Kumar and Levine [32]. To implement VW, we set the sample-wise weight proportional to exp( | y -y ∗ | /T ) , where y ∗ is the maximum score in the training dataset and T ∈ { 0 . 1 , 0 . 3 , 0 . 7 , 1 . 0 } is a hyperparameter. As shown in Fig. D.1, the results indicate that RW outperforms both NW and VW. This validates our design choice for BOOTGEN.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel algorithm, the Bootstrapped Training of Score-Conditioned Generator (BOOTGEN), to optimize the design of biological sequences. BOOTGEN overcomes the challenge of high-cost evaluations and vast search space by training a score-based generator using rank-based weights and a bootstrapping process. This generator is then augmented with self-generated data labeled by a proxy score function. The process results in diverse and accurate biological sequence designs. The efficacy of BOOTGEN is demonstrated via extensive experiments across six offline biological design tasks.\n\nSTRENGTHS:\nThe paper is clear and well-presented. The authors provide a comprehensive introduction and explanation of the BOOTGEN algorithm, making it easily understandable even for those who may not be specialists in this specific field. The experimental results are also clearly laid out and explained, adding to the overall lucidity of the paper.\n\nFurthermore, the extensive experiments and superior results over existing methods reinforce the potential value of this work to both academic researchers and industry practitioners.\n\nWEAKNESSES:\n1. The authors have overlooked a crucial piece of literature, \"Deep Extrapolation for Attribute-Enhanced Generation\" (https://arxiv.org/abs/2107.02968), presented at NeurIPS 2021. This paper also explores protein sequence design and employs a generator/scorer framework. The authors need to acknowledge this work and conduct comparative analysis with it.\n\n2. The first word of the title \"Automatic design of biological sequences\" should be capitalized. This correction should also be made in Section 4.2, where the title reads \"Varying the evaluation budget.\"\n\n3. Claiming the ranking-based weighting as a significant contribution over the value-based weighting is not entirely novel and may not warrant its inclusion as a key contribution.\n\n4. The paper's differentiation from \"Conditioning by adaptive sampling for robust design,\" which also uses a generative model and a proxy for labeling, is unclear.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nLimitations discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to solve the problem of generating novel objects (in this case biological sequences) by learning weighted MLE models, and augmenting the training set of those MLE models with virtual data whose score is based on extrapolations from a proxy.\n\nThis method is tested on standard biological sequence generation problems and found to yield good scores and diversity.\n\nSTRENGTHS:\nI think the main strength of this paper is it confirms that extrapolation through MBO-style proxy+generative model that's recently gained popularity still can be improved and explored in a variety of ways.\n\nThe paper is well written and it was easy for me to understand the method.\n\nWEAKNESSES:\nI think the main weakness of this work is that it feels like an aggregation of methods that work well together, rather than a strong single contribution that deeply improves our understanding of MBO.  \nThis is seen fairly well in e.g. Tables 4.3 and 4.4, there's an accumulation of things that improve performance; while it is commendable to improve the performance of ML methods, it's not clear exactly what we've learned from this paper. If there's one way I think the authors could improve this paper it's by either showing or arguing that there is a central contribution, a nugget of knowledge gained by doing these experiments.\n\nQUESTIONS:\nI don't have too many questions, but I have a minor concern about the fairness of comparisons; I wonder if all baselines see the same amount of data and receive roughly equivalent amounts of compute. In particular, the BootGen experiments seem to leverage an ensemble of generators (\"we gather cross-aggregated samples from multiple score-conditioned generators\"). Many of the baselines could also be trivially improved via ensembling; is this comparison done?\n\nSmall comment: I'd suggest using shapes (in addition of colors) in Fig 4.2\n\nLIMITATIONS:\nYes, the authors address limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes BootGen, a model-based sequence optimization algorithm, the author apply to the task of biological sequence design. BootGen has two stages where the first stage trains multiple sequence generators to give higher probability to sequences predicted to have higher scores from a proxy score model. The second stage uses bootstrapping by way of the trained generators to generate more data and train new generators. Samples are then aggregated in a way to encourage diversity and high fitness. BootGen is compared against previous sequence design methods on six biological sequence tasks on which BootGen achieves state-of-the-art results. Ablations show each component of the method -- bootstrapping, filtering, and rank-based reweighting -- to help performance.\n\nSTRENGTHS:\n- The principle behind BootGen is simple: use bootstrapping to distill signal from multiple models into one. The consensus between all the models will reduce variance and improve performance. Seeing that BootGen can outperform arguably more complicated methods is good to see.\n\n- The combination of steps -- ranking, bootstrapping, filtering, and diversity aggregation -- are logical and synergistic. It's not a surprise they help each other.\n\nWEAKNESSES:\n- **The novelty is overstated**. Bootstrapping is a standard statistics idea, score-wise rank weights was proposed in prior work. The idea of filtering and diversity aggregation are not novel either. Adalead for instance will use the proxy model to select its final sequences. Aggregation is simply taking the union over all the samples. I don't believe there is technical novelty other than bringing together existing ideas in a straightforward way. A bioinformatics journal seems more appropriate for this work.\n\n- There seems to be a large jump in performance by using the proxy model to filter the sequences. In fact, the performance of the method seems extremely dependent on the accuracy of the proxy model. It is not stated that the same proxy model was used across all methods in the experiments. **This leads to a unfair comparison**. I believe a fair comparison would require using the same proxy model to isolate the methodological cotributions put forth in BootGen.\n\n- The amount of bootstrapping seems very high. The appendix states 1280 samples are needed from each generator. The number of generators is not specified clearly. The exact number of model evaluations is unclear and this would have been important to analyze. Otherwise it is difficult to tell if the method is working well due to lots of compute over other methods.\n\nQUESTIONS:\n- I am confused by Algorithm 2 line 2. It says N generators are trained but N is the also the number of training examples. Is this saying the number of generators trained is the same as the number of examples?\n\n- Can the authors put the main hyperparameters in the main text? It is important to know what N, I, M are to be able to put in context how expensive the method is next to the results.\n\n- Appendix A.3 was hard to read. \"Bootstrapping is applied with 2, 500 in additional steps\" what does this mean? It is important to put variables to these numbers and explain where in the method they are used.\n\n- As stated above, was the same proxy model used across all proxy-based baselines? i.e. GFN?\n\n- How are diverse are the generators? I would suspect they all generate similar sequences especialyl given the small dataset. Was this not a problem?\n\n- Can the authors put some representative sequences in the appendix? It is surprising to me the method can get such high fitness in UTR with high novelty, i.e. almost half the sequence changing. Are you sure it's not adversarial examples that are being generated?\n\nLIMITATIONS:\nThe authors mentioned the limitation of a poor proxy that could \"backfire\" and cause BootGen to perform worse. However, this is a general limitation for all proxy based methods. I believe there is a bigger limitation with regards to the run-time of the method with needing lots of samples and lots of training.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel algorithm:  bootstrapped training of score conditioned generators (BOOTGEN), for the offline design of biological sequences. The key idea is to enhance the score-conditioned generator by suggesting a novel variation of the classical ensemble strategy of bootstrapping and aggregating. The method requires training multiple generators using bootstrapped datasets from training and combining them with proxy models to create a reliable and diverse sampling solution.\n\nSTRENGTHS:\nIt is interesting the paper proposes a bootstrapping strategy to augment a training dataset with high-scoring samples that are collected from the score-conditioned generator and labeled using a proxy model.\n\nWEAKNESSES:\nExperiments: \n- The choice of baselines: \n   - The majority of baselines seem not biological sequence model specific (biological sequence design, e.g. https://openreview.net/forum?id=HklxbgBKvr and https://arxiv.org/abs/2006.03227, many others. ), instead the baselines are mostly general optimization methods.  Applying a general method to a new application can lead to lower accuracy. The paper can be more convincing to compare with methods target on the same application. \n   - The rank based weighted score idea may be partially inspired by reference [41], which is rank-based weighting scheme for training unconditional generators   See (sec 3.1). Why not compare performance against that score design? \n\n\nIdea: \n\n-  The paper is focused on high scores samples.\nNot including distribution very on median and lower score region, which might harm learning of the entire distribution. The negative samples may be helpful to learn the entire distribution, e.g. those in energy based model\n-  The idea (using sequence-to-score to enhance score-to-sequence) is a relatively common idea. It will be convincing to compare with similar results. \n\nAblation study: \n- plotting the number of generators vs accuracy can be helpful to show the efficacy of the methods.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}