SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We compare our approach with two categories of baselines: (1) those that use generative models for sampling purposes, and (2) those that apply gradient updates derived from existing designs. The generative model-based methods learn and sample from the distribution of high-scoring designs, including: (i) MIN [18], which maps scores to designs and searches this map for optimal designs; (ii) CbAS [19], which uses a VAE model to adapt the design distribution towards high-scoring areas; (iii) Auto.CbAS [20], which employs importance sampling to retrain a regression model based on CbAS.
The second category encompasses: (i) Grad: carries out a basic gradient ascent on existing designs to generate new ones; (ii) Grad. Min: optimizes the lowest prediction from an ensemble of learned objective functions; (iii) Grad. Mean: optimizes the ensemble's mean prediction; (iv) ROMA [8]: applies smoothness regularization on the DNN; (v) COMs [9]: uses regularization to assign lower scores to designs obtained through gradient ascent; (vi) NEMO [10]: constrains the gap between the proxy and the ground-truth function via normalized maximum likelihood before performing gradient ascent; (vii) BDI [11] uses forward and backward mappings to distill knowledge from the offline dataset to the design; (viii) IOM [21]: enforces representation invariance between the training dataset and the optimized designs.
We also compare with traditional methods in [1]: (i) CMA-ES [22]: gradually adjusts the distribution towards the optimal design by modifying the covariance matrix. (ii) BO-qEI [23]: executes Bayesian Optimization to maximize the proxy, suggests designs through the quasi-Expected-Improvement acquisition function, and labels the designs using the proxy function. (iii) REINFORCE [24]: optimizes the distribution over the input space using the learned proxy.
We adopt the training settings from [1] for all comparison methods unless otherwise specified. We use a 3-layer MLP (MultiLayer Perceptron) with ReLU activation for all gradient updating methods, and set the hidden size to 2048. Additional hyperparameter details are elaborated in Appendix A.3.
One of the top 128 designs from the offline dataset is iteratively selected as the starting point, as outlined in Line 2 of Algorithm 1. We reference results from [1] for non-gradient-ascent methods such as BO-qEI, CMA-ES, REINFORCE, CbAS, and Auto.CbAS. For gradient-based methods, we run each setting over 8 trials and report the mean and standard error. All experiments are run on a single NVIDIA GeForce RTX 3090 GPU.
4.4 Results and Analysis
Performance in Continuous Tasks. Table 1 presents the results across different continuous domains. In all four continuous tasks, our ICT method achieves the top performance. Notably, it surpasses the basic gradient ascent, Grad, demonstrating its ability to mitigate the out-of-distribution issue. The superior performance of Grad.mean over Grad can be attributed to the ensemble model's robustness in making predictions [25]. Furthermore, ICT generally outperforms ensemble methods and other gradient-based techniques such as COMs and ROMA, demonstrating the effectiveness of our strategy. Generative model-based methods, such as CbAS and MINs, however, struggle with the high-dimensional task Hopper Controller. Interestingly, ICT necessitates only three standard proxies and avoids the need for training a generative model, which can often be a challenging task. These results indicate that ICT is a simple yet potent baseline for offline MBO.
Table 1: Experimental results on continuous tasks for comparison.
Method	Superconductor	Ant Morphology	D’Kitty Morphology	Hopper Controller
D(best)	0.399	0.565	0.884	1.0
BO-qEI	0.402 ± 0.034	0.819 ± 0.000	0.896 ± 0.000	0.550 ± 0.018
CMA-ES	0.465 ± 0.024	1.214 ± 0.732	0.724 ± 0.001	0.604 ± 0.215
REINFORCE	0.481 ± 0.013	0.266 ± 0.032	0.562 ± 0.196	-0.020 ± 0.067
CbAS	0.503 ± 0.069	0.876 ± 0.031	0.892 ± 0.008	0.141 ± 0.012
Auto.CbAS	0.421 ± 0.045	0.882 ± 0.045	0.906 ± 0.006	0.137 ± 0.005
MIN	0.499 ± 0.017	0.445 ± 0.080	0.892 ± 0.011	0.424 ± 0.166
Grad	0.483 ± 0.025	0.920 ± 0.044	0.954 ± 0.010	1.791 ± 0.182
Mean	0.497 ± 0.011	0.943 ± 0.012	0.961 ± 0.012	1.815 ± 0.111
Min	0.505 ± 0.017	0.910 ± 0.038	0.936 ± 0.006	0.543 ± 0.010
COMs	0.472 ± 0.024	0.828 ± 0.034	0.913 ± 0.023	0.658 ± 0.217
ROMA	0.510 ± 0.015	0.917 ± 0.030	0.927 ± 0.013	1.740 ± 0.188
NEMO	0.502 ± 0.002	0.952 ± 0.002	0.950 ± 0.001	0.483 ± 0.005
BDI	0.513 ± 0.000	0.906 ± 0.000	0.919 ± 0.000	1.993 ± 0.000
IOM	0.520 ± 0.018	0.918 ± 0.031	0.945 ± 0.012	1.176 ± 0.452
ICT(ours)	0.503 ± 0.017	0.961 ± 0.007	0.968 ± 0.020	2.104 ± 0.357
Table 2: Experimental results on discrete tasks, and ranking on all tasks for comparison.
Method	TF Bind 8	TF Bind 10	NAS	Rank Mean	Rank Median
D(best)	0.439	0.467	0.436
BO-qEI	0.798 ± 0.083	0.652 ± 0.038	1.079 ± 0.059	9.9/15	11/15
CMA-ES	0.953 ± 0.022	0.670 ± 0.023	0.985 ± 0.079	6.1/15	3/15
REINFORCE	0.948 ± 0.028	0.663 ± 0.034	-1.895 ± 0.000	11.3/15	15/15
CbAS	0.927 ± 0.051	0.651 ± 0.060	0.683 ± 0.079	9.1/15	9/15
Auto.CbAS	0.910 ± 0.044	0.630 ± 0.045	0.506 ± 0.074	11.6/15	12/15
MIN	0.905 ± 0.052	0.616 ± 0.021	0.717 ± 0.046	11.0/15	12/15
Grad	0.906 ± 0.024	0.635 ± 0.022	0.598 ± 0.034	7.7/15	9/15
Mean	0.899 ± 0.025	0.652 ± 0.020	0.666 ± 0.062	6.6/15	6/15
Min	0.939 ± 0.013	0.638 ± 0.029	0.705 ± 0.011	7.3/15	8/15
COMs	0.452 ± 0.040	0.624 ± 0.008	0.810 ± 0.029	10.3/15	12/15
ROMA	0.924 ± 0.040	0.666 ± 0.035	0.941 ± 0.020	5.1/15	5/15
NEMO	0.941 ± 0.000	0.705 ± 0.000	0.734 ± 0.015	5.0/15	4/15
BDI	0.870 ± 0.000	0.605 ± 0.000	0.722 ± 0.000	7.9/15	8/15
IOM	0.878 ± 0.069	0.648 ± 0.023	0.274 ± 0.021	7.6/15	6/15
ICT(ours)	0.958 ± 0.008	0.691 ± 0.023	0.667 ± 0.091	3.1/15	2/15
Performance in Discrete Tasks. Table 2 showcases the outcomes across various discrete domains. ICT attains top performances in two out of the three tasks, TF Bind 8 and TF Bind 10. These results suggest that ICT is a powerful method in the discrete domain. However, in NAS, the performance of ICT is not as strong, which can be attributed to two factors. Firstly, the neural network design in NAS,
represented by a 64-length sequence of 5-categorical one-hot vectors, has a higher dimensionality than TF Bind 8 and TF Bind 10, making the optimization process more complex. Furthermore, the simplistic encoding-decoding strategy in design-bench may not accurately capture the intricacies of the neural network's accuracy, which can only be determined after training on CIFAR10.
Summary. ICT attains the highest rankings with a mean of 3.1/15 and median of 2/15 as shown in Table 2 and Figure 3, and also secures top performances in 6 out of the 7 tasks. We have further run a Welch's t-test between our method and the second-best method, obtaining p-values of 0.437 on SuperC, 0.004 on Ant, 0.009 on D'Kitty, 0.014 on Hopper, 0.000 on TF8, 0.045 on TF10, 0.490 on NAS. This demonstrates statistically significant improvement in 5 out of 7 tasks, reaffirming the effectiveness of our method.
To better understand the impact of pseudo-label-driven co-teaching (co-teaching) and meta-learning-based sample reweighting (reweighting) on the performance of our proposed ICT method, we conduct ablation studies by removing either co-teaching or reweighting from the full ICT approach. Table 3 presents the results. Beyond
Table 3: Ablation studies on two core steps of ICT.
Task	D	ICT	w/o co-teaching	w/o reweighting
TF8	8	0.958 ± 0.008	0.905 ± 0.042	0.910 ± 0.024
TF10	10	0.691 ± 0.023	0.653 ± 0.018	0.654 ± 0.023
NAS	64	0.667 ± 0.091	0.779 ± 0.071	0.666 ± 0.090
SuperC	86	0.503 ± 0.017	0.500 ± 0.017	0.501 ± 0.017
Ant	60	0.961 ± 0.007	0.927 ± 0.033	0.914 ± 0.015
D’Kitty	56	0.968 ± 0.020	0.962 ± 0.021	0.959 ± 0.013
Hopper	5126	2.104 ± 0.357	1.453 ± 0.734	1.509 ± 0.166
just assessing these performance indicators, we also verify the accuracy of the samples chosen by co-teaching, as well as the efficacy of the sample weights we have calculated. We do this by referring to the ground truth, with further details provided in Appendix A.4. Our reweighting module is also compared with the recently proposed RGD method [26] as detailed in the Appendix A.5.
For two of the discrete tasks (TF8 and TF10), the ICT method consistently exceeds the performance of both its ablated versions. This highlights the efficacy of the two steps when handling discrete tasks. Conversely, the exclusion of the co-teaching in NAS leads to an increase in performance. This could be attributed to the fact that the encoding-decoding strategy of NAS in design-bench is unable to accurately capture the inherent complexity of neural networks. As such, the co-teaching step, reliant on this strategy, might not be as effective. For the continuous tasks (SuperC, Ant, D'Kitty, and Hopper), we observe that the complete ICT method consistently achieves superior performance. This underlines the effectiveness of the two steps when dealing with continuous tasks. The performance gains are particularly substantial in the Hopper task when the complete ICT method is compared with the ablated versions, illustrating the power of the two steps in managing high-dimensional continuous tasks. Overall, our ablation studies demonstrate that the inclusion of both co-teaching and reweighting in the ICT method generally enhances performance across diverse tasks and input dimensions, underscoring their integral role in our approach.
4.6 Hyperparameter Sensitivity
We first assess the robustness of our ICT method by varying the number of samples  $(K)$  selected during the co-teaching process on the continuous D'Kitty Morphology task. For this analysis,  $K$  is varied among  $K = 8,16,32,64$ . In Figure 4 (a), we illustrate the  $100^{th}$  percentile normalized ground-truth score as a function of time step  $T$ , for each of these  $K$  values. The results demonstrate that the performance of ICT is resilient to variations in  $K$ , maintaining performances within a certain range. Additionally, ICT is capable of generating high-scoring designs early on in the process, specifically achieving such designs around the time step  $t = 50$ , and sustains this performance thereafter, demonstrating its robustness against the number of optimization steps  $T$ .
We further evaluate the robustness of our ICT method against the learning rate  $(\beta)$  for the meta-learning framework. As depicted in Figure 4 (b), ICT's performance remains relatively consistent across a variety of  $\beta$  values, further demonstrating ICT's robustness with respect to the hyperparameter  $\beta$ . We explore the fine-tuning learning rate  $\alpha$  and conduct further experiments and analysis on TF Bind 8. Details can be found in Appendix A.6.
Figure 3: The whiskers in the plot Figure 4: Ground-truth score of the design evolves as a function mark the rank minima and max- of step  $T$ , under different values of  $K$  (subplot (a)) and  $\beta$  (subplot ima, whereas the vertical lines (b)), in the D'Kitty task. These results indicate that ICT attains and black triangles respectively optimal designs swiftly and maintains them stably, as well as represent the median and mean. exhibits robustness to various choices of  $K$  and  $\beta$ .
Offline Model-based Optimization. Contemporary offline model-based optimization methods can be generally classified into two primary groups: (i) generating novel designs through generative models, and (ii) conducting gradient ascent on existing designs. The former methods learn and sample from the distribution of high-scoring designs including MIN [18], CbAS [19], Auto.CbAS [20] and BootGen [27]. Recently, gradient-based methods have gained popularity due to their ability to leverage deep neural networks (DNNs) for improved design generation. These methods apply regularization techniques to either the proxy itself [8-10] or the design under consideration [11, 12], enhancing the proxy's robustness and generalization capabilities. An interesting subfield of offline MBO includes biological sequence design, which has potential applications such as designing drugs for treating diseases [27, 28]. In particular, the work [27] also adopts a proxy as a pseudo-labeler and aligns the generator with the proxy, a technique that resonates with our method. ICT falls under this category, but adopts a unique approach to improve proxy performance: it incorporates valuable knowledge from a pseudo-labeled dataset into other proxies for fine-tuning, thereby enhancing the ensemble performance. Notably, while the concurrent work of parallel mentoring [29] also employs pseudo-labeling, it focuses on pairwise comparison labels, potentially sacrificing some information due to its discrete nature.
Sample Reweighting. Sample reweighting is commonly utilized to address the issue of label noise [30, 31], where each sample is assigned a larger weight if it is more likely to be accurate, using a carefully designed function. Recent studies [32-34] suggest using a meta-set to guide the learning of sample weights, which can enhance model training. Such an approach is grounded in a meta-learning framework which can be used to learn hyperparameters [35-37, 34, 38-43]. Inspired by distributionally robust optimization, recent work [26] proposes a re-weighted gradient descent algorithm that provides an efficient and effective means of reweighting. In this paper, the pseudolabeled dataset generated by co-teaching may still contain some inaccuracies, while the offline dataset is generally accurate. We propose a sample reweighting framework to reduce the inaccuracies in the pseudo-labeled dataset by leveraging the supervision signals from the offline dataset.
Co-teaching. Co-teaching [13] is an effective technique for mitigating label noise by leveraging insights from peer networks. It involves the concurrent training of two proxies where one proxy identifies small-loss samples within a noisy mini-batch for fine-tuning the other. Co-teaching bears similarities to decoupling [44] and co-training [45], as they all involve the interaction between two models to enhance the training process. In this study, we adapt co-teaching to work with a pseudo-labeled dataset generated by a trained proxy, instead of relying on a noisy original dataset. Specifically, we employ one proxy to select accurate samples from this pseudo-labeled dataset for fine-tuning the other, and vice versa.
In this study, we introduce the ICT (Importance-aware Co-Teaching) method for mitigating the out-of-distribution issue prevalent in offline model-based optimization. ICT is a two-step approach. The first step is pseudo-label-driven co-teaching, which iteratively selects a proxy to generate pseudolabeled data. Valuable data are identified by co-teaching to fine-tune other proxies. This process,
repeated three times with different pseudo-labelers, facilitates knowledge transfer. In the second step, meta-learning-based sample reweighting assigns and updates importance weights to samples selected by the co-teaching process, further improving the proxy fine-tuning. Our experimental findings demonstrate the success of ICT. We discuss its limitations in Appendix A.7
Future Work. Though we initially design ICT with three proxies, the method's inherent scalability and flexibility make it applicable to scenarios involving  $N$  proxies. In such a scenario, we can iteratively select one proxy out of  $N$  as the pseudo-labeler to generate data. Then, each of the remaining  $N - 1$  proxies could select small-loss samples from its perspective and provide these samples to the other  $N - 2$  proxies for fine-tuning. This process enhances knowledge transfer and facilitates cooperative learning among the proxies. Looking to the future, we plan to conduct further research into the dynamics of such an expanded ensemble of proxies.
Negative Impact. It is crucial to recognize that ICT's potential benefits come with possible negative consequences. Advanced optimization techniques can be applied for both constructive and destructive purposes, depending on their use. For example, while drug development and material design can have a positive impact on society, these techniques could also be misused to create harmful substances or products. As researchers, we must remain attentive and strive to ensure that our work is employed for the betterment of society while addressing any potential risks and ethical concerns.
This research was empowered in part by the computational support provided by Compute Canada (www.computecanada.ca).
[1] Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Benchmarks for data-driven offline model-based optimization. arXiv preprint arXiv:2202.08450, 2022.
[2] Thomas Liao, Grant Wang, Brian Yang, Rene Lee, Kristofer Pister, Sergey Levine, and Roberto Calandra. Data-efficient learning of morphology and controller for a microrobot. arXiv preprint arXiv:1905.01334, 2019.
[3] Karen S Sarkisyan et al. Local fitness landscape of the green fluorescent protein. Nature, 2016.
[4] Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In Proc. Int. Conf. Learning Rep. (ICLR), 2019.
[5] Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of a superconductor. Computational Materials Science, 2018.
[6] Luis A Barrera et al. Survey of variation in human transcription factors reveals prevalent dna binding changes. Science, 2016.
[7] Paul J Sample, Ban Wang, David W Reid, Vlad Presnyak, Iain J McFadyen, David R Morris, and Georg Seelig. Human 5 UTR design and variant effect prediction from a massively parallel translation assay. Nature Biotechnology, 2019.
[8] Sihyun Yu, Sungsoo Ahn, Le Song, and Jinwoo Shin. Roma: Robust model adaptation for offline model-based optimization. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2021.
[9] Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective models for effective offline model-based optimization. In Proc. Int. Conf. Learning Rep. (ICLR), 2021.
[10] Justin Fu and Sergey Levine. Offline model-based optimization via normalized maximum likelihood estimation. Proc. Int. Conf. Learning Rep. (ICLR), 2021.
[11] Can Chen, Yingxue Zhang, Jie Fu, Xue Liu, and Mark Coates. Bidirectional learning for offline infinite-width model-based optimization. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2022.
[30] Lu Jiang, Deyu Meng, Teruko Mitamura, and Alexander G Hauptmann. Easy samples first: Self-paced reranking for zero-example multimedia search. In Proceedings of the 22nd ACM international conference on Multimedia, 2014.
[31] Yixin Wang, Alp Kucukelbir, and David M Blei. Robust probabilistic modeling with bayesian data reweighting. In Proc. Int. Conf. Machine Lea. (ICML), 2017.
[32] Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In Proc. Int. Conf. Machine Lea. (ICML), 2018.
[33] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight-net: Learning an explicit mapping for sample weighting. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2019.
[34] Can Chen, Shuhao Zheng, Xi Chen, Erqun Dong, Xue Steve Liu, Hao Liu, and Dejing Dou. Generalized data weighting via class-level gradient manipulation. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2021.
[35] Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In Proc. Int. Conf. Machine Lea. (ICML), 2018.
[36] Can Chen, Xi Chen, Chen Ma, Zixuan Liu, and Xue Liu. Gradient-based bi-level optimization for deep learning: A survey. arXiv preprint arXiv:2207.11719, 2022.
[37] Fuyuan Lyu, Xing Tang, Huifeng Guo, Ruiming Tang, Xiuqiang He, Rui Zhang, and Xue Liu. Memorize, factorize, or be naive: Learning optimal feature interaction methods for ctr prediction. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), 2022.
[38] Can Chen, Chen Ma, Xi Chen, Sirui Song, Hao Liu, and Xue Liu. Unbiased implicit feedback via bi-level optimization. arXiv preprint arXiv:2206.00147, 2022.
[39] Can Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-aware protein self-supervised learning. Bioinformatics, 2023.
[40] Tommaso Giovannelli, Griffin Kent, and Luis Nunes Vicente. Bilevel optimization with a multi-objective lower-level problem: Risk-neutral and risk-averse formulations. arXiv preprint arXiv:2302.05540, 2023.
[41] Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Dataset distillation using parameter pruning. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, 2023.
[42] Zhixiang Chi, Yang Wang, Yuanhao Yu, and Jin Tang. Test-time fast adaptation for dynamic scene deblurring via meta-auxiliary learning. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recogn. (CVPR), 2021.
[43] Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, and Jin Tang. Metascal: A meta-learning approach for few-shot class incremental learning. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recogn. (CVPR), 2022.
[44] Eran Malach and Shai Shalev-Shwartz. Decoupling" when to update" from" how to update". Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2017.
[45] Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh annual conference on Computational learning theory, 1998.
[46] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. Int. Conf. Learning Rep. (ICLR), 2015.
A.1 Considerations for Sampling Around the Offline Dataset
In this subsection, we explore an alternative sampling strategy for the pseudo-labeling process. Instead of generating new samples around the current optimization point, this strategy generates samples directly around the offline dataset  $\mathcal{D}$ . To ascertain the effectiveness of our chosen strategy against this alternative, we perform experiments on two tasks: D'Kitty (continuous) and TF8 (discrete).
Table 4 showcases the results. For both tasks, our strategy consistently yields higher scores, affirming its superior performance over the alternative. The advantage of our chosen strategy can be attributed to its dynamic nature. By sampling around the current optimization point, we gather more insightful information for the local fine-tuning of the proxy. This strategy allows the co-teaching process to adapt and evolve according to the optimization trajectory, leading to improved performances.
Table 4: Comparison of Sampling Strategies.
Method	Sampling along Gradient Path (Ours)	Sampling from D
TF8	0.958 ± 0.008	0.871 ± 0.067
D’Kitty	0.968 ± 0.020	0.955 ± 0.006
A.2 Comparative Performance Analysis using Median Scores
In addition to the maximum scores discussed in the main paper, we also present the median  $(50^{th}$  percentile) scores across all seven tasks. The best design in the offline dataset, denoted as  $\mathcal{D}(\text{best})$ , along with the mean and median rankings are provided for comprehensive comparison.
Performance in Continuous Tasks. Table 5 illustrates the performances of ICT compared with other methods in continuous tasks. It is noteworthy that ICT exhibits performance on par with the best-performing methods. Compared with the vanilla gradient ascent (Grad), ICT demonstrates superior performance, thus affirming its effectiveness in addressing out-of-distribution issues. Moreover, ICT is generally better than the mean ensemble (Mean), which demonstrates the effectiveness of our strategy. These results support the use of ICT as a robust baseline for offline MBO.