SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this paper, we benchmark our method against both gradient-based and non-gradient-based approaches. The gradient-based methods include: (1) Grad: optimizes the design against the learned proxy via simple gradient ascent; (2) DE (Deep Ensemble)[7]: optimizes the design against the mean ensemble of three proxies via gradient ascent; (3) GB (Gradient Boosting) [22]: sequentially trains new proxies to obtain a robust proxy, followed by gradient ascent using the proxy; (4) COMs [7]: lower bounds the proxy's prediction on out-of-distribution designs and subsequently carries out gradient ascent; (5) ROMA [23]: incorporates the smoothness prior into the proxy and optimizes the design against the proxy; (6) NEMO [24]: leverages normalized maximum likelihood to constrain the distance between the proxy and the ground-truth, and acquires new designs by gradient ascent; (7) BDI [25]: proposes to distill the information from the static dataset into the high-scoring design; (8) IOM [26]: enforces the invariance between the representations of the static dataset and generated designs to achieve a natural trade-off. Since our tri-mentoring adopts three proxies, methods using one proxy including COMs, ROMA and IOM are equipped with three proxies for a fair comparison. We also explore combining our method with ROMA and COMs and please refer to Appendix A.4 for an in-depth discussion and corresponding empirical results.
The non-gradient-based methods include: (1) BO-qEI [27]: fits a Gaussian Process, proposes candidate designs utilizing the quasi-Expected Improvement acquisition function, and assigns labels to the candidates with the proxy; (2) CMA-ES [28]: labels the sampled designs and gradually adapts the covariance matrix distribution towards the high-scoring part among the sampled designs; (3) REINFORCE [29]: parameterizes a design distribution and optimizes the distribution towards the optimal design by policy gradient; (4) CbAS [30]: trains a VAE model and progressively adapts the model to focus on the high-scoring designs; (5) Auto.CbAS [31]: retrans the proxy adopted in CbAS by leveraging importance sampling; (6) MIN [32]: learns an inverse map from a score to a design and then samples from the map conditioned an optimal score value.
We follow the settings in [7, 25] if not specified. We adopt a three-layer MLP network with the ReLU function as the activation. We train the MLP model on the static dataset with a  $1 \times 10^{-3}$  learning rate and an Adam optimizer. The fine-tuning learning rate  $\gamma$  is set as  $1 \times 10^{-3}$  and the soft-labeling learning rate  $\lambda$  is set as  $1 \times 10^{-1}$ . The standard deviation  $\delta$  is set as  $1 \times 10^{-1}$  and the number of the samples  $K$  is set as 10. All experiments are performed on a single V100 GPU. To ensure the robustness of our results, we perform 16 trials for each setting and report the mean and standard error. We've detailed the training time and computational overhead of our approach in Appendix A.5 to provide a comprehensive view of its practicality.
In Table 1 and Table 2, we present the results of our experiments for continuous and discrete tasks, respectively. A delineating line is drawn to separate the gradient-based methods from the non-gradient-based methods. Results for non-gradient-ascent methods are taken from [1]. The highest score of the static dataset for each task is denoted by  $\mathcal{D}(\text{best})$ . For each task, we highlight the algorithms that fall within one standard deviation of the highest performance by bolding their results.
Continuous tasks. (1) Table 1 demonstrates tri-mentoring attains the best results across the board, highlighting its effectiveness. Its consistent gains over Grad confirm its ability to tackle the out-of-distribution issue. We further test our tri-mentoring for out-of-distribution issues as detailed in Appendix A.6. (2) Compared to DE and GB, which also use multiple proxies, tri-mentoring achieves
Table 1: Results (maximum normalized score) on continuous tasks.
Method	Superconductor	Ant Morphology	D’Kitty Morphology	Hopper Controller
D(best)	0.399	0.565	0.884	1.000
BO-qEI	0.402 ± 0.034	0.819 ± 0.000	0.896 ± 0.000	0.550 ± 0.018
CMA-ES	0.465 ± 0.024	1.214 ± 0.732	0.724 ± 0.001	0.604 ± 0.215
REINFORCE	0.481 ± 0.013	0.266 ± 0.032	0.562 ± 0.196	-0.020 ± 0.067
CbAS	0.503 ± 0.069	0.876 ± 0.031	0.892 ± 0.008	0.141 ± 0.012
Auto.CbAS	0.421 ± 0.045	0.882 ± 0.045	0.906 ± 0.006	0.137 ± 0.005
MIN	0.499 ± 0.017	0.445 ± 0.080	0.892 ± 0.011	0.424 ± 0.166
Grad	0.495 ± 0.011	0.934 ± 0.011	0.944 ± 0.017	1.797 ± 0.116
DE	0.514 ± 0.015	0.937 ± 0.016	0.956 ± 0.014	1.805 ± 0.105
GB	0.496 ± 0.012	0.926 ± 0.029	0.948 ± 0.012	1.793 ± 0.429
COMs	0.491 ± 0.028	0.856 ± 0.040	0.938 ± 0.015	0.642 ± 0.167
ROMA	0.508 ± 0.014	0.914 ± 0.029	0.930 ± 0.012	1.728 ± 0.266
NEMO	0.502 ± 0.002	0.958 ± 0.011	0.954 ± 0.007	0.481 ± 0.003
IOM	0.522 ± 0.018	0.926 ± 0.030	0.943 ± 0.012	1.015 ± 0.380
BDI	0.513 ± 0.000	0.906 ± 0.000	0.919 ± 0.000	1.993 ± 0.000
Tri-mentoring	0.514 ± 0.018	0.948 ± 0.014	0.966 ± 0.010	1.983 ± 0.110
Table 2: Results (maximum normalized score) on discrete tasks & ranking on all tasks.
Method	TF Bind 8	TF Bind 10	NAS	Rank Mean	Rank Median
D(best)	0.439	0.467	0.436
BO-qEI	0.798 ± 0.083	0.652 ± 0.038	1.079 ± 0.059	10.1/15	11/15
CMA-ES	0.953 ± 0.022	0.670 ± 0.023	0.985 ± 0.079	6.4/15	4/15
REINFORCE	0.948 ± 0.028	0.663 ± 0.034	-1.895 ± 0.000	11.4/15	15/15
CbAS	0.927 ± 0.051	0.651 ± 0.060	0.683 ± 0.079	9.1/15	9/15
Auto.CbAS	0.910 ± 0.044	0.630 ± 0.045	0.506 ± 0.074	11.6/15	11/15
MIN	0.905 ± 0.052	0.616 ± 0.021	0.717 ± 0.046	11.0/15	12/15
Grad	0.886 ± 0.035	0.647 ± 0.021	0.624 ± 0.102	7.9/15	9/15
DE	0.900 ± 0.056	0.659 ± 0.033	0.655 ± 0.059	5.3/15	4/15
GB	0.922 ± 0.050	0.630 ± 0.041	0.716 ± 0.088	7.6/15	6/15
COMs	0.496 ± 0.065	0.622 ± 0.003	0.783 ± 0.029	10.0/15	11/15
ROMA	0.917 ± 0.039	0.672 ± 0.035	0.927 ± 0.071	5.7/15	6/15
NEMO	0.943 ± 0.005	0.711 ± 0.021	0.737 ± 0.010	5.0/15	4/15
IOM	0.861 ± 0.079	0.647 ± 0.027	0.559 ± 0.081	7.9/15	7/15
BDI	0.870 ± 0.000	0.605 ± 0.000	0.722 ± 0.000	8.1/15	9/15
Tri-mentoring	0.970 ± 0.001	0.722 ± 0.017	0.759 ± 0.102	2.1/15	2/15
better results in all four tasks, indicating its improved robustness by sharing ranking supervision signals among proxies. (3) DE typically outperforms simple gradient ascent due to ensemble prediction robustness, consistent with findings in [1]. (4) Other gradient-based methods, such as COMs, fail to achieve the performance as tri-mentoring, further highlighting its superior effectiveness. (5) In low-dimensional TF Bind 8 tasks, gradient-based methods (average rank 8.8) underperform compared to non-gradient-based methods (average rank 6.8); however, in high-dimensional TF Bind 10 tasks, gradient-based methods (average rank 7.7) surpass non-gradient-based methods (average rank 8.2). This suggests non-gradient-based methods, like REINFORCE and generative modeling, are more suited for low-dimensional design due to their global search ability, while gradient-based methods provide more direct guidance in high-dimensional designs.
Discrete tasks. (1) Table 2 shows that tri-mentoring achieves the best results in two out of the three tasks, with a marginal difference in the third. This indicates that tri-mentoring is a potent method for discrete tasks as well. (2) However, in complex tasks such as NAS, where each design is represented as a 64-length sequence of 5-category one-hot vectors, the performance of tri-mentoring is slightly compromised. This could be attributed to the encoding in design-bench not fully accounting for the sequential and hierarchical nature of network architectures, leading to less effective gradient updates. Our proposed method, also demonstrates its effectiveness on high-dimensional biological sequence design, achieving maximum normalized scores of 0.865 and 0.699 on GFP and UTR respectively.
In summary, tri-mentoring achieves the highest ranking as shown in Table 2 and Figure 3, and delivers the best performance in six out of the seven tasks.
Table 3: Ablation studies on tri-mentoring.
Task	tri-mentoring	w/o voting-based ps	w/o ada soft-labeling	w/o neighbor	post selection
SuperC	0.514 ± 0.018	0.505 ± 0.014	0.504 ± 0.014	0.516 ± 0.017	0.512 ± 0.011
Ant	0.948 ± 0.014	0.938 ± 0.021	0.945 ± 0.018	0.945 ± 0.012	0.945 ± 0.009
D’Kitty	0.966 ± 0.010	0.956 ± 0.010	0.947 ± 0.008	0.958 ± 0.008	0.970 ± 0.013
Hopper	1.983 ± 0.110	1.902 ± 0.138	1.916 ± 0.108	1.839 ± 0.112	1.901 ± 0.148
TF Bind 8	0.970 ± 0.001	0.971 ± 0.003	0.944 ± 0.026	0.950 ± 0.018	0.949 ± 0.006
TF Bind 10	0.722 ± 0.017	0.694 ± 0.030	0.710 ± 0.025	0.643 ± 0.009	0.635 ± 0.027
NAS	0.759 ± 0.102	0.509 ± 0.074	0.538 ± 0.082	0.666 ± 0.089	0.519 ± 0.076
In this subsection, the proposed method serves as the baseline, and we systematically remove each module including voting-based pairwise supervision and adaptive soft-labeling to verify its contribution. The results are presented in Table 3. Besides the performance results here, we also provide an evaluation of the accuracy of generated pairwise labels to further verify the effectiveness of the two modules in Appendix A.7.
Voting-based pairwise supervision. Instead of using the proposed module, we compute the mean prediction of the ensemble and use this prediction to create pairwise consensus labels. We denote this as w/o voting-based ps. Our results in Table 3 show a decline in performance when adopting this alternative. A plausible explanation for this performance decline is that the alternative module fails to effectively exploit weak ranking supervision signals from individual proxies, resulting in reduced information exchange and collaboration among the proxies compared to the proposed module.
Adaptive soft-labeling. We remove this module and resort to using one-hot consensus labels. The performance across all tasks generally deteriorates compared to the full tri-mentoring. A possible explanation for this is that this module ensures that fine-tuned proxies are optimized to perform well on the static dataset by introducing soft-labels, reducing the risk of overfitting to consensus labels derived from individual proxy predictions. This demonstrates the significance of the adaptive soft-labeling module in mitigating the effects of label noise and enhancing the ensemble performance.
Furthermore, we assess the impact of neighborhood sampling in our method and a post selection strategy related to our method, with results outlined in Table 3.
Neighborhood sampling. Typically, we sample  $K$  neighborhood points at the optimization point  $\boldsymbol{x}_t$  for pairwise labels. When neighborhood sampling is excluded (w/o neighbor), labels are directly generated near the static dataset. The performance generally deteriorates for w/o neighbor in Table 3, due to the lack of local ranking information around the optimization point.
Post selection. We investigate a variant, post selection, where the mean of two proxies is used to select the third proxy's candidates. From Table 3, we find that this variant generally yields worse results compared to the full tri-mentoring. This finding suggests that using the ranking supervision signals directly for design selection is less effective than allowing proxies to exchange ranking supervision signals within the ensemble to produce a more robust ensemble.
In this section, we study the sensitivity of our method to hyperparameters, specifically the number of neighborhood samples  $K$  and the number of optimization steps  $T$ , on two tasks, the continuous Ant and the discrete TFB8. We also discuss the standard deviation hyperparameter  $\delta$  in Appendix A.8.
Number of neighborhood samples  $(K)$ . We evaluate the performance of our method for different values of  $K$ , i.e., the number of neighborhood samples around the optimization point. We test  $K$  values of 5, 10, 15, 20, and 25, with  $K = 10$  being the default value used in this paper. The results are normalized by dividing them by the result obtained for  $K = 10$ . As shown in Figure 4, the performance of the tri-mentoring method is quite robust to changes in  $K$  for both tasks.
Number of optimization steps  $(T)$ . We also investigate the impact of the number of optimization steps on the performance of our method. As indicated in Figure 5, the method is robust to changes in the number of optimization steps for both Ant and TFB8 tasks.
In summary, our sensitivity analysis demonstrates that the tri-mentoring method is robust to variations in key hyperparameters, ensuring stable performance across a range of settings.
Figure 3: Whiskers signify rank min/max; medians and means are shown as vertical lines and black triangles, respectively.
Figure 4: The ratio of the performance of our tri-mentoring method with  $K$  to the performance with  $K = 10$ .
Figure 5: The performance score as a function of the optimization step  $T$  for both TFB8 and Ant tasks.
Offline model-based optimization. Recently two broad groups of methods have emerged for offline model-based optimization. One group is based on generative modeling, where methods aim to gradually adapt a generative model towards the high-scoring design [31-33]. Another group is based on using gradient ascent to optimize existing designs via gradient information. These methods generally try to incorporate prior information into the proxy before using it for gradient ascent. Examples of this approach include COMs [7], ROMA [23], NEMO [24], BDI [25, 34] and IOM [26]. Our proposed method called parallel-mentoring (tri-mentoring) belongs to this category. We maintain an ensemble of proxies and aim to incorporate weak ranking signals from any pair of proxies into the third. This symmetric learning process, which cycles through all proxies, enhances the robustness and resilience of our ensemble. Our proposed ensemble training process, with its focus on parallel-mentoring, has the potential to improve the proxy and reward training [35, 33], thereby contributing to advancements in biological sequence design. Notably, the contemporaneous ICT method [36] exchanges direct proxy scores, which may be less robust than our pairwise comparison approach.
Tri-training. Our work is related to tri-training [37] which trains three classifiers and refines them using unlabeled samples. In each round, an unlabeled sample is labeled for one classifier if the other two agree on the label, under specific conditions. While tri-mentoring is inspired by tri-training, there are fundamental differences between them: (1) Tri-training aims to enhance classification tasks by mitigating label noise, while tri-mentoring focuses on producing a more robust regression ensemble; (2) Tri-training leverages unlabeled data, while tri-mentoring operates on samples near the current optimization point; (3) Tri-mentoring incorporates an adaptive soft-labeling module to mitigate label noise, which is not present in tri-training. In addition to tri-training and tri-mentoring, there are other research works [38] [39] [40] involving multiple proxies cooperating to improve learning. Unlike tri-training and tri-mentoring with three proxies, these methods focus on two proxies.
Bi-level optimization for hyperparameter optimization. Bi-level optimization has become increasingly popular for hyperparameter optimization [41-46] due to the hierarchical problem structure [47]. In the inner level, the relationship between model parameters and hyperparameters is established by minimizing the training loss. Meanwhile, the outer level optimizes hyperparameters through the connection built at the inner level by minimizing the validation loss. A specific category of hyperparameters is soft-label [48, 49], which is updated under the guidance of noise-free data to reduce noise. In this paper, we propose adaptive soft-labeling to reduce noise in the consensus labels.
Ensemble learning. Ensemble learning techniques train multiple base learners and aggregate their predictions to achieve better performance and generalization than individual base learners alone [8, 9, 50, 10, 11, 51, 52]. These methods can be broadly classified into boosting [53, 54], bagging [55], and stacking [56]. In contrast to our proposed tri-mentoring, where multiple proxies collaborate to enhance the learning process, ensemble learning mainly involves interaction during the aggregation phase, without influencing each other's training.
Pairwise learning to rank. Learning to rank [16, 57] has been extensively employed by commercial search engines for ranking search results [58, 59]. Unlike pointwise methods that score inputs, pairwise methods focus on relative order, aligning more with ranking concepts [57]. Recent research [60] applies pairwise binary cross-entropy loss for training reward models in reinforcement learning, a technique used in ChatGPT [18]. Our work expresses a proxy's ranking ability through pairwise comparison labels, generating consensus labels via majority voting to enable mutual mentoring.
In this work, we introduce parallel-mentoring to enhance ensemble robustness against out-of-distribution issues through mutual mentoring among proxies. Focusing on a three-proxy case, we instantiate this as tri-mentoring, with two modules: voting-based pairwise supervision for generating consensus labels, and adaptive soft-labeling which mitigates label noise through bi-level optimization. Experimental results validate our approach's effectiveness. We discuss potential negative impacts in Appendix A.9 and limitations in Appendix A.10.
We thank CIFAR for support under the AI Chairs program. This research was empowered in part by the computational support provided by Compute Canada (www.computecanada.ca).
[1] Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Benchmarks for data-driven offline model-based optimization. arXiv preprint arXiv:2202.08450, 2022.
[2] Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of a superconductor. Computational Materials Science, 2018.
[3] Karen S Sarkisyan et al. Local fitness landscape of the green fluorescent protein. Nature, 2016.
[4] Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In Proc. Int. Conf. Learning Rep. (ICLR), 2019.
[5] Luis A Barrera et al. Survey of variation in human transcription factors reveals prevalent dna binding changes. Science, 2016.
[6] Paul J Sample, Ban Wang, David W Reid, Vlad Presnyak, Iain J McFadyen, David R Morris, and Georg Seelig. Human 5 UTR design and variant effect prediction from a massively parallel translation assay. Nature Biotechnology, 2019.
[7] Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective models for effective offline model-based optimization. In Proc. Int. Conf. Learning Rep. (ICLR), 2021.
[8] Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions on pattern analysis and machine intelligence, 1990.
[9] Thomas G Dietterich. Ensemble methods in machine learning. In Multiple Classifier Systems: First International Workshop, 2000.
[10] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2017.
[11] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2019.
[12] Kevin K Yang, Zachary Wu, and Frances H Arnold. Machine-learning-guided directed evolution for protein engineering. Nature methods, 2019.
[13] Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In Proc. Int. Conf. Learning Rep. (ICLR), 2020.
[52] Olivier Laurent, Adrien Lafage, Enzo Tartaglione, Geoffrey Daniel, Jean marc Martinez, Andrei Bursuc, and Gianni Franchi. Packed ensembles for efficient uncertainty estimation. In Proc. Int. Conf. Learning Rep. (ICLR), 2023.
[53] Yoav Freund, Robert Schapire, and Naoki Abe. A short introduction to boosting. Journal-Japanese Society For Artificial Intelligence, 1999.
[54] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm SIGkdd international conference on knowledge discovery and data mining, 2016.
[55] Leo Breiman. Bagging predictors. Machine learning, 1996.
[56] David H Wolpert. Stacked generalization. Neural networks, 1992.
[57] Tie-Yan Liu et al. Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval, 2009.
[58] Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proc. Int. Conf. Learning Rep. (ICLR), 2005.
[59] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. Bpr: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618, 2012.
[60] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2020.
A.1 Scenarios for Parallel Mentoring with Multiple Proxies
A.1.1 Method
In the primary paper, we mainly focus on a scenario with three proxies. Here, we extend our method to incorporate  $M$  proxies. We revisit the two essential modules.
Voting-based pairwise supervision. We train  $M$  ( $M \geq 3$ ) parallel proxies,  $f_{\pmb{\theta}}^{1}(\cdot), f_{\pmb{\theta}}^{2}(\cdot), \dots, f_{\pmb{\theta}}^{M}(\cdot)$ , each initialized differently, on the static dataset. Their mean is utilized as the final prediction:
$$
f _ {\boldsymbol {\theta}} (\cdot) = \frac {1}{M} \left(f _ {\boldsymbol {\theta}} ^ {1} (\cdot) + f _ {\boldsymbol {\theta}} ^ {2} (\cdot) + \dots f _ {\boldsymbol {\theta}} ^ {M} (\cdot)\right). \tag {10}
$$
We generate the pairwise comparison labels  $\hat{y}^1, \hat{y}^2, \dots$ , and  $\hat{y}^M$  for each proxy in the same way. We extend the subsequent majority voting part and derive the pairwise consensus labels  $\hat{y}^V$  via an element-wise majority voting:
$$
\hat {\boldsymbol {y}} _ {i j} ^ {V} = \text {m a j o r i t y} _ {-} \text {v o t i n g} \left(\hat {\boldsymbol {y}} _ {i j} ^ {1}, \hat {\boldsymbol {y}} _ {i j} ^ {2}, \dots , \hat {\boldsymbol {y}} _ {i j} ^ {M}\right). \tag {11}
$$
Here,  $i$  and  $j$  are the indexes of the neighborhood samples.
Adaptive soft-labeling. This module remains the same as it is designed for an individual proxy. We carry out fine-tuning and soft-labeling via bi-level optimization to adaptively mentor the proxy.
Setting on  $M$ . In Eq.(11),  $M$  can be any positive number greater than 2 as a decision may not be reached with just two proxies. In this study, we consider  $M$  as an odd number to ensure a decisive outcome in the voting process. Cases with an even number of proxies can be handled by adopting strategies like maintaining the original labels and skipping the fine-tuning step when the proxies are evenly split in their labels. However, we do not delve into these cases for brevity. We examine scenarios with  $M$  equal to 3, 5, 7, 9, and 11.
A.1.2 Experiments
We conduct experiments on the Ant task and the TFB8 task. The performance ratio comparing the performance of parallel mentoring to that of tri-mentoring, is computed as a function of  $M$  (the number of proxies). The results are displayed in Figure 6.