SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implementation We parameterize the conditional distribution  $p_{\theta}(x_t | \boldsymbol{x}_{1:t-1}, y)$  using a 2-layer long short-term memory [26, LSTM] network with 512 hidden dimensions. The condition  $y$  is injected into the LSTM using a linear projection layer. We parameterize the proxy model using a multi-layer perceptron (MLP) with 2048 hidden dimensions and a sigmoid activation function. Our parameterization is consistent across all the tasks. We provide a detailed description of the hyperparameters in Appendix A. We also note the importance of the desired score  $y^{\dagger}$  to condition during bootstrapping and evaluation. In this regard, we set it as the maximum score that is achievable for the given problem, i.e., we set  $y^{\dagger} = 1$ . We assume that such a value is known following [12].
(a) RNA-Binding-A
(d) TFBind8
Figure 4.1: Evaluation-performance graph to compare with representative offline biological design baselines. The number of evaluations  $K \in [1,128]$  stands for the number of candidate designs to be evaluated by the Oracle score function. The average value and standard deviation error bar for 8 independent runs are reported. Our method outperforms other baselines at every task for almost all  $K$ .
(b) RNA-Binding-B
(e) GFP
(c) RNA-Binding-C
(f) UTR
4.2 Performance Evaluation
In Table 4.1 and Table 4.2, we report the performance of our BOOTGEN along with other baselines. One can observe how our BOOTGEN consistently outperforms the considered baselines across all six tasks. In particular, one can observe how our BOOTGEN achieves large gains in 50th percentile metrics. This highlights how our algorithm is able to create a reliable set of candidates.
For TFbind8, which has a relatively small search space  $(4^{8})$ , having high performances on the 100th percentile is relatively easy. Indeed, the classical method of CMA-ES and Grad. gave pretty good performances. However, for the 50th percentile score, a metric for measuring the method's reliability, BDI outperformed previous baselines by a large margin. Our method outperformed even BDI and achieved an overwhelming score.
For higher dimensional tasks of UTR, even the 50th percentile score of BOOTGEN outperforms the 100th percentile score of other baselines by a large margin. We note that our bootstrapping strategies and aggregation strategy greatly contributed to improving performances on UTR. For additional tasks of RNA, we achieved the best score for both the 50th percentile and the 100th percentile. This result verifies that our method is task-expandable.
4.3 Varying the Evaluation Budget
In real-world scenarios, there may be situations where only a few samples can be evaluated due to the expensive score function. For example, in an extreme scenario, for the clinical trial of a new protein drug, there may be only one or two chances to be evaluated. As we measure the 50th percentile and 100th percentile score among 128 samples following the design-bench [45] at Tables 4.1 and 4.2, we also provide a 100th percentile score report at the fewer samples from 1 sample to the 128 samples to evaluate the model's robustness on the low-budget evaluation scenarios.
To account for this, we also provide a budget-performance graph that compares the performance of our model to the baselines using different numbers of evaluations. This allows us to observe the trade-off between performance and the number of samples generated. Note that we select baselines as the Top 5 methods in terms of average percentile 100 scores reported at Table 4.1.
Table 4.3: Experimental results on 100th percentile scores (100th Per.), 50th percentile scores (50th Per.), average score (Avg. Score), diversity, and novelty, among 128 samples of UTR task. The mean and standard deviation of 8 independent runs for producing 128 samples is reported. The best-scored value is marked in bold. The lowest standard deviation is marked as the underline. The DA stands for the diverse aggregation strategy.
Methods	100th Per.	50th Per.	Avg. Score	Diversity	Novelty
MIN [32]	0.691 ± 0.011	0.587 ± 0.012	0.554 ± 0.010	28.53 ± 0.095	18.32 ± 0.091
CMA-ES [24]	0.746 ± 0.018	0.498 ± 0.012	0.520 ± 0.013	24.69 ± 0.150	19.95 ± 0.925
Grad. [45]	0.682 ± 0.013	0.513 ± 0.007	0.521 ± 0.006	25.63 ± 0.615	16.89 ± 0.426
GFN-AL [29]	0.700 ± 0.015	0.602 ± 0.014	0.580 ± 0.014	30.89 ± 1.220	20.25 ± 2.272
BOOTGEN w/o DA.	0.729 ± 0.074	0.672 ± 0.082	0.652 ± 0.081	17.83 ± 5.378	20.49 ± 1.904
BOOTGEN w/ DA. (ours)	0.858 ± 0.003	0.701 ± 0.004	0.698 ± 0.001	31.57 ± 0.073	21.40 ± 0.057
Figure 4.2: Multi-objectivity comparison of diversity and novelty on the average score for the UTR task. Each datapoint for 8 independent runs is depicted.
As shown in Fig. 4.1, our method outperforms every baseline for almost every evaluation budget. For the UTR task, our performance on a single evaluation budget gives a better score than the other baselines' scores when they have a budget of 128 evaluations. For RNA tasks, our method with an approximate budget of 30 achieves superior performance compared to other methods with a budget of 128. These results show that our method is the most reliable as its performance is most robust when the evaluation budget is limited.
4.4 Average Score with Diversity
For biological sequence design, measuring the diversity and novelty of the generated sequence is also crucial [29]. Following the evaluation metric of [29] we compare the performance of models in terms of diversity and novelty.
Here is measurement of diversity for sampled design dataset  $\mathcal{D} = \{\pmb{x}_1,\dots,\pmb{x}_M\}$  from generator which is average of Levenshtein distance [23], denoted by  $d(\pmb{x}_i,\pmb{x}_j)$ , between arbitrary two biological sequences  $\pmb{x}_i,\pmb{x}_j$  from the generated design candidates  $\mathcal{D}$ :
$$
\operatorname {D i v e r s i t y} (\mathcal {D}) := \frac {1}{| \mathcal {D} | (| \mathcal {D} | - 1)} \sum_ {\boldsymbol {x} \in \mathcal {D}} \sum_ {\boldsymbol {s} \in \mathcal {D} \backslash \{\boldsymbol {x} \}} d \left(\boldsymbol {x}, \boldsymbol {s}\right).
$$
Next, we measure the minimum distance from the offline dataset  $\mathcal{D}_{\mathrm{offline}}$  which measures the novelty of generated design candidates  $\mathcal{D}$  as:
$$
\text {N o v e l t y} (\mathcal {D}, \mathcal {D} _ {\text {o f f l i n e}}) = \frac {1}{| \mathcal {D} |} \sum_ {\boldsymbol {x} \in \mathcal {D}} \min  _ {\boldsymbol {s} \in \mathcal {D} _ {\text {o f f l i n e}}} d (\boldsymbol {x}, \boldsymbol {s}).
$$
Our method surpasses all baselines, including GFN-AL [29], in the UTR task, as evidenced by the Pareto frontier depicted in Table 4.3 and Fig. 4.2. Given the highly dimensional nature of the UTR task and its expansive search space, the discovery of novel and diverse candidates appears to be directly related to their average score. This implies that extensive exploration of the high-dimensional space is crucial for improving scores in the UTR task.
Table 4.4: Ablation study for BOOTGEN. The average score among 128 samples is reported. We make 8 independent runs to produce 128 samples where the mean and the standard deviation are reported. For every method, an aggregation strategy is applied by default. The best-scored value is marked in bold. The lowest standard deviation is underlined. The RR stands for rank-based reweighting, the B stands for bootstrapping, and the F stands for filtering.
Components	RNA-A	RNA-B	RNA-C	TFbind8	UTR	GFP
∅	0.388 ± 0.007	0.350 ± 0.008	0.394 ± 0.010	0.579 ± 0.010	0.549 ± 0.009	0.457 ± 0.044
{RR}	0.483 ± 0.006	0.468 ± 0.008	0.441 ± 0.010	0.662 ± 0.009	0.586 ± 0.008	0.281 ± 0.031
{RR, B}	0.408 ± 0.009	0.379 ± 0.009	0.417 ± 0.006	0.666 ± 0.009	0.689 ± 0.003	0.470 ± 0.034
{RR, F}	0.576 ± 0.005	0.586 ± 0.004	0.536 ± 0.007	0.833 ± 0.004	0.621 ± 0.003	0.783 ± 0.011
{RR, F, B}	0.607 ± 0.009	0.612 ± 0.005	0.554 ± 0.007	0.840 ± 0.004	0.698 ± 0.001	0.804 ± 0.002
It is worth noting that GFN-AL, which is specifically designed to generate diverse, high-quality samples through an explorative policy, secures a second place for diversity. Although GFN-AL occasionally exhibits better diversity than our method and achieves a second-place average score, it consistently delivers poor average scores in the GFP and RNA tasks Table 4.2. This drawback can be attributed to its high explorative policy, which necessitates focused exploration in narrow regions. In contrast, BOOTGEN consistently produces reliable scores across all tasks Table 4.2. For a comprehensive comparison with GFN-AL, please refer to the additional experiments presented in Appendix C.
Diverse aggregation strategy Our diverse aggregation (DA) strategy significantly enhances diversity, novelty, and score variance, as demonstrated in Table 4.3. This is especially beneficial for the UTR task, which necessitates extensive exploration of a vast solution space, posing a substantial risk to the bootstrapped training process. In this context, certain bootstrapped generators may yield exceedingly high scores, while others may produce low scores due to random exploration scenarios. By employing DA, we combine multiple generators to generate candidate samples, thereby greatly stabilizing the quality of the bootstrapped generator.
4.5 Ablation study
The effectiveness of our components, namely rank-based reweighting (RR), bootstrapping (B), and filtering (F), in improving performance is evident in Table 4.4. Across all tasks, these components consistently contribute to performance enhancements. The bootstrapping process is particularly more beneficial for high-dimensional tasks like UTR and GFP. This correlation is intuitive since high-dimensional tasks require a larger amount of data for effective exploration. The bootstrapped training dataset augmentation facilitates this search process by leveraging proxy knowledge. Additionally, the filtering technique proves to be powerful in improving scores. As we observed from the diverse aggregation and filtering, the ensemble strategy greatly enhances score-conditioned generators.
5 Future Works
Enhancing proxy robustness While our bootstrapping method shows promise for offline bio-sequential design tasks, it has inherent technical limitations. The assumption that the generator produces superior data to the training dataset may backfire if the generator samples have poor quality designs and the proxy used is inaccurate. While the current aggregation strategy effectively manages this risk, we can address this limitation by utilizing robust learning methods of proxies such as conservative proxies modeling [44], robust model adaptation techniques [52], parallel mentoring proxies [13], and importance-aware co-teaching of proxies [53] for further improvement.
Enhancing architecture of BOOTGEN Our approach primarily employs a straightforward architectural framework, with a primary emphasis on validating its algorithmic structures in the context of offline biological sequence design. To enhance the practical utility of our method, it will be advantageous to incorporate established and robust architectural paradigms, exemplified in works such as [11] and [14], into the framework of our method. One promising avenue for achieving this integration is the incorporation of pre-trained protein language models (pLMs) [34, 15], akin to those expounded upon in [14].
This study introduces a novel approach to stabilize and enhance score-conditioned generators for offline biological sequence design, incorporating the classical concepts of bootstrapping and aggregation. Our novel method, named BOOTGEN, consistently outperformed all baselines across six offline biological sequence design tasks, encompassing RNA, DNA, and protein optimization. Our strategy of bootstrapping and aggregation yielded remarkable improvements in achieving high scores, generating diverse samples, and minimizing performance variance.
We thank all the valuable comments and suggestions from anonymous reviewers who helped us improve and refine our paper. This work was supported by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korean government(MSIT)(2022-0-01032, Development of Collective Collaboration Intelligence Framework for Internet of Autonomous Things).
[1] M.-R. Amini and P. Gallinari. Semi-supervised logistic regression. In ECAI, volume 2, page 11, 2002.
[2] C. Angermueller, D. Dohan, D. Belanger, R. Deshpande, K. Murphy, and L. Colwell. Model-based reinforcement learning for biological sequence design. In International conference on learning representations, 2019.
[3] F. H. Arnold. Design by directed evolution. Accounts of chemical research, 31(3):125-131, 1998.
[4] F. H. Arnold. Directed evolution: bringing new chemistry to life. Angewandte Chemie International Edition, 57(16):4143-4148, 2018.
[5] L. A. Barrera, A. Vedenko, J. V. Kurland, J. M. Rogers, S. S. Gisselbrecht, E. J. Rossin, J. Woodard, L. Mariani, K. H. Kock, S. Inukai, et al. Survey of variation in human transcription factors reveals prevalent DNA binding changes. Science, 351(6280):1450-1454, 2016.
[6] D. Belanger, S. Vora, Z. Mariet, R. Deshpande, D. Dohan, C. Angermueller, K. Murphy, O. Chapelle, and L. Colwell. Biological sequences design using batched bayesian optimization. 2019.
[7] E. Bengio, M. Jain, M. Korablyov, D. Precup, and Y. Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381-27394, 2021.
[8] J. D. Bloom and F. H. Arnold. In the light of directed evolution: pathways of adaptive protein evolution. Proceedings of the National Academy of Sciences, 106(supplement_1):9995-10000, 2009.
[9] D. Brookes, H. Park, and J. Listgarten. Conditioning by adaptive sampling for robust design. In International conference on machine learning, pages 773-782. PMLR, 2019.
[10] D. H. Brookes and J. Listgarten. Design by adaptive sampling. arXiv preprint arXiv:1810.03714, 2018.
[11] A. Chan, A. Madani, B. Krause, and N. Naik. Deep extrapolation for attribute-enhanced generation. Advances in Neural Information Processing Systems, 34:14084-14096, 2021.
[12] C. Chen, Y. Zhang, J. Fu, X. Liu, and M. Coates. Bidirectional learning for offline infinitewidth model-based optimization. In Advances in Neural Information Processing Systems, 2022.
[13] C. Chen, C. Beckham, Z. Liu, X. Liu, and C. Pal. Parallel-mentoring for offline model-based optimization. arXiv preprint arXiv:2309.11592, 2023.
[14] C. Chen, Y. Zhang, X. Liu, and M. Coates. Bidirectional learning for offline model-based biological sequence design. arXiv preprint arXiv:2301.02931, 2023.
[15] C. Chen, J. Zhou, F. Wang, X. Liu, and D. Dou. Structure-aware protein self-supervised learning. Bioinformatics, 2023.
[16] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
[17] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems, 33: 22243-22255, 2020.
[18] P. A. Dalby. Strategy and success for the directed evolution of enzymes. Current opinion in structural biology, 21(4):473-480, 2011.
[19] C. Ekbote, M. Jain, P. Das, and Y. Bengio. Consistent training via energy-based gflows for modeling discrete joint distributions. arXiv preprint arXiv:2211.00568, 2022.
[20] C. Fannjiang and J. Listgarten. Autofocused oracles for model-based design. Advances in Neural Information Processing Systems, 33:12945-12956, 2020.
[21] J. Fu and S. Levine. Offline model-based optimization via normalized maximum likelihood estimation. arXiv preprint arXiv:2102.07970, 2021.
[22] Y. Grandvalet and Y. Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004.
[23] R. Haldar and D. Mukhopadhyay. Levenshtein distance technique in dictionary lookup methods: An improved approach. arXiv preprint arXiv:1101.1232, 2011.
[24] N. Hansen. The CMA evolution strategy: a comparing review. Towards a new evolutionary computation, pages 75-102, 2006.
[25] T. Hesterberg. Bootstrap. Wiley Interdisciplinary Reviews: Computational Statistics, 3(6): 497-526, 2011.
[26] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.
[27] A. Hottung, B. Bhandari, and K. Tierney. Learning a latent search space for routing problems using variational autoencoders. In International Conference on Learning Representations, 2020.
[28] I. Igashov, H. Stärk, C. Vignac, V. G. Satorras, P. Frossard, M. Welling, M. Bronstein, and B. Correia. Equivariant 3d-conditional diffusion models for molecular linker design. arXiv preprint arXiv:2210.05274, 2022.
[29] M. Jain, E. Bengio, A. Hernandez-Garcia, J. Rector-Brooks, B. F. Dossou, C. A. Ekbote, J. Fu, T. Zhang, M. Kilgour, D. Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning, pages 9786-9801. PMLR, 2022.
[30] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[31] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[32] A. Kumar and S. Levine. Model inversion networks for model-based optimization. Advances in Neural Information Processing Systems, 33:5126-5137, 2020.
[51] Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. Self-training with noisy student improves imageNet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020.
[52] S. Yu, S. Ahn, L. Song, and J. Shin. Roma: Robust model adaptation for offline model-based optimization. Advances in Neural Information Processing Systems, 34:4619-4631, 2021.
[53] Y. Yuan, C. Chen, Z. Liu, W. Neiswanger, and X. Liu. Importance-aware co-teaching for offline model-based optimization. arXiv preprint arXiv:2309.11600, 2023.
[54] M. Zimmer. Green fluorescent protein (GFP): applications, structure, and related photophysical behavior. Chemical reviews, 102(3):759-782, 2002.
A Additional Experimental Settings
Table A.1: Details of the offline datasets. We let  $|\mathcal{X}|$  and  $|\mathcal{D}|$  denote the sizes of the search space and the offline dataset, respectively.
Seq. Length	Vocab size	|X|	|D|
GFP	20	237	20237	5,000
UTR	50	4	504	140,000
TFBind8	8	8	48	32,898
RNA-Binding	14	4	414	5,000
- GFP [41] is a task to optimize a protein sequence of length 237 consisting of one of 20 amino acids, i.e., the search space is  $20^{237}$ . Its objective is to find a protein with high fluorescence. Following Trabucco et al. [45], we prepare the offline dataset using 5000 samples with 50 to 60 percentile scores in the original data.
- UTR [41] is a task to optimize a DNA sequence of length 50 consisting of one of four nucleobases: adenine (A), guanine (G), cytosine (C), thymine (T). Its objective is to maximize the expression level of the corresponding  $5^{\prime}$  UTR region. For the construction of the offline dataset  $\mathcal{D}$ , following Trabucco et al. [45], we provide samples with scores under the 50th percentile data of 140,000 examples.
- TFBind8 [5] is a task that optimizes DNA similar to the UTR. The objective is to find a length 8 sequence to maximize the binding activity with human transcription factors. For the offline dataset  $\mathcal{D}$ , we provide under 50th percentile data of 32,898 examples following Trabucco et al. [45].
- RNA-Binding [33] is a task that optimizes RNA, a sequence that contains four vocab words of nucleobases: adenine (A), uracil (U), cytosine (C), and guanine (G). The objective is to find a length 14 sequence to maximize the binding activity with the target transcription factor. We present three target transcriptions of RNA termed RNA-Binding-A (for L14 RNA1), RNA-Binding-B (for L14 RNA2), and RNA-Binding-C (for L14 RNA3). We provide under 0.12 scored data for the offline dataset  $\mathcal{D}$  among randomly generated 5,000 sequences using open-source code<sup>2</sup>.
A.2 Implementation of Baselines
This section provides a detailed implementation of baselines of offline biological sequence design.
Baselines from Design Bench [45]. Most baselines are from the offline model-based optimization (MBO) benchmark called design-bench [45]. The design bench contains biological sequence tasks of the GFP, UTR, and TFbind8, where it contains baselines of REINFORCE, CMA-ES [24], BO-qEI [50], CbAS [9], Auto. Cbas [20], MIN [32], gradient ascent (Grad.), and COMS [44]. We reproduce them by following the official source code. For the RNA tasks, we follow hyperparameters of TFBind8 as the number of vocab are same as 4, and the sequence length is similar where the TFBind8 has length 8 and RNA tasks have length 14 as our method follows the same.
BDI [12]. For BDI, we follow hyperparameter setting at the paper [12] and implementation at the opensource code. For RNA tasks, we follow the hyperparameter for TFBind8 tasks, as our method follows the same.
GFN-AL [29]. For GFN-AL we follow hyperparameters setting at the paper [29] and implementation on open-source code<sup>5</sup>. Because they only reported the TFbind8 and the GFP tasks, we use the hyperparameter of the GFP for the UTR tasks hyperparameter of the TFBind8 for RNA tasks, as our method follows the same.
$^{2}$ https://github.com/samsinai/FLEXS
<sup>3</sup>https://github.com/brandontrabucco/design-baselines
<sup>4</sup>https://github.com/GGchen1997/BDI
<sup>5</sup>https://github.com/MJ10/BioSeq-GFN-AL
A.3 Hyperparameters
Table A.2: Hyperparameters.  $I$  denotes the number of bootstrapping iterations after pretraining,  ${I}^{\prime }$  denotes the number of pretraining iterations,  $M$  represents the number of samples used in inference,  $K$  stands for the number of filtered samples among  $M$  candidates,and  ${N}_{\text{gen }}$  refers to the number of aggregated generators in the experiments.
I	I'	M	K	Ngen
2,500	12,500	1,280	128	8
Training. We give consistency hyperparameters for all tasks except the learning rate. We set the generator's learning rate to  $10^{-5}$  for short-length tasks (lengths 8 and 14) of TFBind and RNA tasks and  $5 \times 10^{-5}$  for longer-length tasks (lengths 50 and 237) of UTR and GFP. We trained the generator with 12,500 steps before bootstrapping. Bootstrapping is applied with 2,500 in additional steps. The batch size of training is 256. We set the weighting parameter  $k = 10^{-2}$ . Note that we early-stopped the generator iteration of GFP with the 3,000 step based on monitoring the calibration model of Appendix B. For bootstrapping, the generator samples 2 candidates every 5 steps. For Top-K sampling at the bootstrapping, we sample with  $L = 1,000$  and select the Top 2 samples to augment the training dataset.
Testing. For filtering, we generated  $M = 1$ , 280 candidate samples and collected the Top-K samples where  $K = 128$  based on the proxy score. For diverse aggregation, we collect  $K = 16$  samples from 8 generators, making a total of 128 samples.
Proxy model. For the proxy model, we applied a weight regularization of  $10^{-4}$ , set the learning rate to  $10^{-4}$ , and used a dropout rate of 0.1. We used early stopping with a tolerance of 5 and a train/validate ratio of 9:1 following Jain et al. [29]. We used the Adam optimizer [30] for the training generator, proxy, and calibration model.
B Calibration Model
(a) TFbind8
Figure B.1: Calibration model's tendency.
(b) GFP
Tuning the hyperparameters of offline design algorithms is challenging due to the lack of access to the true score function. Therefore, existing works have proposed various strategies to circumvent this issue, e.g., choosing a hyperparameter that is transferrable between different tasks [45] or tuning the hyperparameter based on training statistics [52].