SlowGuess/NeurIPS_2023_OCR huggingface dataset:
ExPT details For each domain, we pretrain ExPT for 10,000 iterations with 128 synthetic functions in each iteration, corresponding to a total number of 1,280,000 synthetic functions. For each function, we randomly sample 228 input  $x^{\prime}s$  from the unlabeled dataset  $\mathcal{D}_{\mathrm{unlabeled}}$  and generate the values  $y^{\prime}s$  from a Gaussian Process with an RBF kernel. To increase the diversity of synthetic data, we randomize the two hyperparameters, length scale  $\ell \sim \mathcal{U}[5.0,10.0]$  and function scale  $\sigma \sim \mathcal{U}[1.0,1.0]$ , when generating each function. Additionally, we add Gaussian noises  $\epsilon \sim \mathcal{N}(0,0.1)$  to each input  $x$  sampled from  $\mathcal{D}_{\mathrm{unlabeled}}$  to enlarge the pretraining inputs. For each generated function, we use 100 points as context points and the remaining 128 as target points, and train the model to optimize (2). During the adaptation phase, we condition the pretrained ExPT model on the labeled few-shot dataset  $\mathcal{D}_{\mathrm{few-shot}}$  and the target function value  $y^{\star}$  to generate designs  $x^{\prime}s$ .
Baselines We compare ExPT with BayesOpt (GP-qEI) [53], a canonical ED method, and MINs [36], COMs [57], BDI[12], and BONET [34], four recent deep learning models that have achieved state-of-the-art performance in the offline setting. To adapt GP-qEI to the few-shot setting, we use a feedforward network trained on few-shot data to serve as an oracle, a Gaussian Process to quantify uncertainty, and the quasi-Expected Improvement [61] algorithm for the acquisition function. For the deep learning baselines, we train their models on the few-shot dataset  $\mathcal{D}_{\mathrm{few - shot}}$  using the hyperparameters reported in their original papers.
Evaluation For each considered method, we allow an optimization budget  $Q = 256$ . We report the median score, the max score, and the mean score among the 256 proposed inputs. Following previous works, we normalize the score to [0, 1] by using the minimum and maximum function values from a large hidden dataset  $y_{\mathrm{norm}} = \frac{y - y_{\mathrm{min}}}{y_{\mathrm{max}} - y_{\mathrm{min}}}$ . We report the mean and standard deviation of the score across 3 independent runs for each method.
Results Table1 shows the performance of different methods in the random setting. Most methods perform well in the random setting, where ExPT achieves the highest average score and the best average rank across all 3 performance metrics. For each of the tasks and metrics considered, ExPT is either the best or second-best performing method. Notably, in Ant, ExPT significantly outperforms the best baseline by  $18\%$ ,  $9\%$ , and  $10\%$  with respect to the median, max, and mean performance,
<sup>1</sup>We exclude domains where the oracle functions are flagged to be highly inaccurate and noisy in prior works (ChEMBL, Hopper, and Superconductor), or too expensive to evaluate (NAS). See Appendix B for more details.
Table 1: Comparison of ExPT and the baselines on the few-shot random setting of 5 Design-Bench tasks. We report median, max, and mean performance across 3 random seeds. Higher scores and lower ranks are better. Blue denotes the best entry in the column, and Violet denotes the second best.
Baseline	D'Kitty	Ant	TF Bind 8	TF Bind 10	Mean score (↑)	Mean rank (↓)
\( \mathcal{D}_{\text{few-shot}}(\text{best}) \)	0.883	0.563	0.439	0.466	—	—
Median	MINs	0.859 ± 0.014	0.485 ± 0.152	0.416 ± 0.019	0.468 ± 0.014	0.557 ± 0.050	4.0
COMs	0.752 ± 0.007	0.411 ± 0.012	0.371 ± 0.001	0.468 ± 0.000	0.501 ± 0.005	4.0
BONET	0.852 ± 0.013	0.597 ± 0.119	0.441 ± 0.003	0.483 ± 0.009	0.593 ± 0.036	2.3
BDI	0.592 ± 0.020	0.396 ± 0.018	0.540 ± 0.032	0.438 ± 0.034	0.492 ± 0.026	4.8
GP-qEI	0.842 ± 0.058	0.550 ± 0.007	0.439 ± 0.000	0.467 ± 0.000	0.575 ± 0.016	4.0
ExPT	0.902 ± 0.006	0.705 ± 0.018	0.473 ± 0.014	0.477 ± 0.014	0.639 ± 0.013	1.5
Max	MINs	0.930 ± 0.010	0.890 ± 0.017	0.814 ± 0.030	0.639 ± 0.017	0.818 ± 0.019	3.3
COMs	0.920 ± 0.010	0.841 ± 0.044	0.686 ± 0.152	0.656 ± 0.020	0.776 ± 0.057	4.0
BONET	0.909 ± 0.012	0.888 ± 0.024	0.887 ± 0.053	0.702 ± 0.006	0.847 ± 0.024	3.0
BDI	0.918 ± 0.006	0.806 ± 0.094	0.906 ± 0.074	0.532 ± 0.023	0.791 ± 0.049	4.5
GP-qEI	0.896 ± 0.000	0.887 ± 0.000	0.513 ± 0.104	0.647 ± 0.011	0.736 ± 0.029	5.0
ExPT	0.973 ± 0.005	0.970 ± 0.004	0.933 ± 0.036	0.677 ± 0.048	0.888 ± 0.023	1.3
Mean	MINs	0.624 ± 0.025	0.009 ± 0.013	0.415 ± 0.030	0.465 ± 0.015	0.378 ± 0.021	4.8
COMs	0.515 ± 0.050	0.020 ± 0.006	0.369 ± 0.003	0.471 ± 0.004	0.344 ± 0.016	4.8
BONET	0.837 ± 0.023	0.579 ± 0.024	0.448 ± 0.011	0.484 ± 0.009	0.587 ± 0.017	2.0
BDI	0.570 ± 0.032	0.385 ± 0.012	0.536 ± 0.032	0.444 ± 0.027	0.484 ± 0.026	3.5
GP-qEI	0.505 ± 0.006	0.019 ± 0.001	0.439 ± 0.001	0.473 ± 0.002	0.359 ± 0.003	4.5
ExPT	0.865 ± 0.016	0.639 ± 0.026	0.476 ± 0.010	0.474 ± 0.015	0.614 ± 0.017	1.5
Table 2: Comparison of ExPT and the baselines on the few-shot poorest setting of 5 Design-Bench tasks. We report the median, max, and mean performance across 3 random seeds. Higher scores and lower ranks are better. Blue denotes the best entry in the column, and Violet denotes the second best.
Baseline	D'Kitty	Ant	TF Bind 8	TF Bind 10	Mean score (↑)	Mean rank (↓)
\( \mathcal{D}_{\text{few-shot}}(\text{best}) \)	0.307	0.124	0.124	0.000	—	—
Median	MINs	0.480 ± 0.156	0.316 ± 0.040	0.437 ± 0.007	0.463 ± 0.003	0.424 ± 0.052	3.5
COMs	0.733 ± 0.023	0.401 ± 0.026	0.111 ± 0.000	0.459 ± 0.006	0.426 ± 0.014	4.3
BONET	0.310 ± 0.000	0.236 ± 0.047	0.319 ± 0.018	0.461 ± 0.017*	0.332 ± 0.021	4.8
BDI	0.309 ± 0.000	0.192 ± 0.012	0.365 ± 0.000	0.454 ± 0.017	0.330 ± 0.007	5.5
GP-qEI	0.883 ± 0.000	0.565 ± 0.001	0.439 ± 0.000	0.467 ± 0.000	0.589 ± 0.000	2.0
ExPT	0.922 ± 0.009	0.686 ± 0.090	0.552 ± 0.042	0.489 ± 0.013	0.662 ± 0.039	1.0
Max	MINs	0.841 ± 0.014	0.721 ± 0.031	0.962 ± 0.019	0.648 ± 0.025	0.793 ± 0.022	3.3
COMs	0.931 ± 0.022	0.843 ± 0.020	0.124 ± 0.000	0.739 ± 0.057	0.659 ± 0.025	3.3
BONET	0.929 ± 0.031	0.557 ± 0.118	0.809 ± 0.038	0.519 ± 0.039*	0.704 ± 0.057	5.0
BDI	0.939 ± 0.002	0.693 ± 0.109	0.913 ± 0.000	0.596 ± 0.020	0.785 ± 0.033	3.5
GP-qEI	0.896 ± 0.000	0.887 ± 0.000	0.439 ± 0.000	0.645 ± 0.021	0.717 ± 0.005	3.8
ExPT	0.946 ± 0.018	0.965 ± 0.004	0.873 ± 0.035	0.615 ± 0.022	0.850 ± 0.020	2.3
Mean	MINs	0.623 ± 0.051	0.015 ± 0.017	0.464 ± 0.009	0.463 ± 0.002	0.391 ± 0.020	3.3
COMs	0.607 ± 0.021	0.033 ± 0.003	0.109 ± 0.001	0.454 ± 0.004	0.301 ± 0.007	4.8
BONET	0.490 ± 0.023	0.234 ± 0.052	0.318 ± 0.018	0.459 ± 0.018	0.375 ± 0.028	4.0
BDI	0.364 ± 0.004	0.215 ± 0.021	0.369 ± 0.000	0.453 ± 0.018	0.350 ± 0.011	4.8
GP-qEI	0.533 ± 0.001	0.018 ± 0.000	0.439 ± 0.000	0.470 ± 0.002	0.365 ± 0.001	3.5
ExPT	0.871 ± 0.018	0.646 ± 0.061	0.549 ± 0.032	0.488 ± 0.011	0.639 ± 0.031	1.0
respectively. Only ExPT and BONET achieve a meaningful performance in Ant when considering the mean score. BONET is also the overall second-best method in this setting.
Table 2 shows the superior performance of ExPT in few-shot poorest, the more challenging setting. ExPT achieves the highest score in 10/12 individual tasks and metrics, and also achieves the highest score and the best rank across tasks on average. Notably, in terms of the mean score, ExPT beats the best baseline by a large margin, achieving an improvement of  $40\%$ ,  $176\%$ , and  $18\%$  on D'Kitty, Ant, and TF Bind 8, and  $70\%$  on average. The performance of most baselines drops significantly from the random to the poorest setting, including BONET, the second-best method in the random setting. This was also previously observed in the BONET paper [34]. Interestingly, the performance of ExPT, MINs, and GP-qEI is not affected much by the quality of the few-shot data, and even improves in certain metrics. We hypothesize that even though the dataset is of lower quality, it may contain specific anti-correlation patterns about the problem that the model can exploit.
Pretraining analysis In addition to the absolute performance, we investigate the performance of ExPT on downstream tasks through the course of pretraining. Figure 4 shows that the performance of ExPT in most tasks improves consistently as the number of pretraining steps increases. This shows that synthetic pretraining on diverse functions facilitates the generalization to complex real-world functions. In Ant, the performance slightly drops between 4000 and 10000 iterations. Therefore, we can further improve ExPT if we have a way to stop pretraining at a point that likely leads to the best
Figure 4: The median and mean performance of ExPT of 4 Design-Bench tasks through the course of pretraining. We average the performance across 3 seeds.
SynTO D(best)
Table 3: ExPT's performance on different objectives in D'Kitty and Ant domains. We pretrain one model for all tasks in the same domain. The performance is averaged across 3 seeds.
Task	D’Kitty	D’Kitty-45	D’Kitty-60	Ant	Ant-vy	Ant-Energy
\( {\mathcal{D}}_{\text{few-shot }}\left( \text{best}\right) \)	0.307	0.297	0.344	0.124	0.210	0.189
Median	ExPT	0.922 ± 0.009	0.611 ± 0.007	0.569 ± 0.010	0.686 ± 0.090	0.613 ± 0.009	0.635 ± 0.028
Max	ExPT	0.976 ± 0.004	0.954 ± 0.008	0.973 ± 0.004	0.965 ± 0.004	0.923 ± 0.049	0.950 ± 0.033
Mean	ExPT	0.871 ± 0.018	0.619 ± 0.016	0.584 ± 0.008	0.646 ± 0.061	0.599 ± 0.005	0.608 ± 0.025
performance in downstream tasks. However, in practice, we do not have the luxury of testing on real functions during pretraining. Alternatively, we could perform validation and early stopping on a set of held-out, out-of-distribution synthetic functions. We leave this to future work.
3.2.1 ExPT for few-shot optimization of multiple objectives
As we mention in Section 2 one advantage of unsupervised pretraining is the ability to optimize for multiple objectives during the adaptation phase. In this section, we show that the same pretrained ExPT model is capable of optimizing different objectives in D'Kitty and Ant domains. We create two variants of the original D'Kitty, namely D'Kitty-45 and D'Kitty-60, whose objectives are to navigate the robot to goals that are  $45^{\circ}$  and  $60^{\circ}$  away from the original goal, respectively. For Ant, we create  $\text{Ant-v}_y$  where the goal is to run as fast as possible in the vertical  $Y$  direction (as opposed to horizontal  $X$  direction in Ant) direction, and Ant-Energy, where the goal is to preserve energy. We detail how to construct these tasks in Appendix C. We use the same pretrained models for all these tasks. During adaptation, the model conditions on the  $\mathcal{D}_{\mathrm{few-shot}}$  and  $y^\star$  for each task for optimization.
We evaluate ExPT on these tasks in the poorest setting. Table 3 shows that ExPT performs well on all tasks, where the median and mean scores are better than the best value in  $\mathcal{D}_{\mathrm{few-shot}}$ , and the max score is close to 1. For the Ant,  $\mathrm{Ant - v_y}$ , and  $\mathrm{Ant - Energy}$  tasks, we visualize the behavior of the optimal designs that are discovered at https://imgur.com/a/zpgI4YL. When subject to the same policy-network (optimizing for horizontal  $X$  speed), the robots optimized for different objectives behave differently; the optimal Ant is capable of leaping forward to move quickly in  $X$ ;  $\mathrm{Ant - v_y}$  is able to jump up to maximize speed in  $Y$ ; Ant-Energy is capable of 'sitting down' to conserve energy.
3.2.2 ExPT for closed-loop optimization
ExPT can adapt to any objective function purely through in-context learning. This means that the model can refine its understanding of the underlying objective function given more data points in a very efficient manner. In this section, we explore an alternative optimization scheme for ExPT, namely sequential sampling, which explicitly utilizes the in-context learning ability of the model. Specifically, instead of producing  $Q = 256$  inputs simultaneously, we sample one by one sequentially. That is, we condition the model on  $\mathcal{D}_{\mathrm{few-shot}}$  and  $y^{\star}$  to sample the first point, evaluate the point using the black-box function, and add the point together with its score to the context set. We repeat this process for 256 times. This process is often referred to as closed-loop optimization in ED.
Table 4: Comparison of simultaneous (ExPT) and sequential (ExPT-Seq) sampling on Ant and D'Kitty in random (left) and poorest (right) settings. We average the performance across 3 seeds.
Baseline	D’Kitty	Ant
Dfew-shot(best)	0.883	0.563
Median	ExPT	0.902 ± 0.006	0.705 ± 0.018
ExPT-Seq	0.903 ± 0.005	0.719 ± 0.013
Mean	ExPT	0.865 ± 0.016	0.639 ± 0.026
ExPT-Seq	0.872 ± 0.010	0.669 ± 0.017
Baseline	D’Kitty	Ant
Dfew-shot(best)	0.307	0.124
Median	ExPT	0.922 ± 0.009	0.686 ± 0.090
ExPT-Seq	0.928 ± 0.012	0.822 ± 0.055
Mean	ExPT	0.871 ± 0.018	0.646 ± 0.061
ExPT-Seq	0.923 ± 0.011	0.767 ± 0.048
Table 5: Comparison of inverse modeling (ExPT) versus forward modeling (TNP-ED) on Ant and D'Kitty in random (left) and poorest (right) settings. We average the performance across 3 seeds.
Baseline	D’Kitty	Ant
Dfew-shot(best)	0.883	0.563
Median	ExPT	0.902 ± 0.006	0.705 ± 0.018
TNP-ED	0.770 ± 0.009	0.438 ± 0.007
Mean	ExPT	0.865 ± 0.016	0.639 ± 0.026
TNP-ED	0.670 ± 0.037	0.451 ± 0.018
Baseline	D’Kitty	Ant
Dfew-shot(best)	0.307	0.124
Median	ExPT	0.922 ± 0.009	0.686 ± 0.090
TNP-ED	0.309 ± 0.000	0.197 ± 0.005
Mean	ExPT	0.871 ± 0.018	0.646 ± 0.061
TNP-ED	0.405 ± 0.004	0.237 ± 0.005
Table 4 shows that ExPT with sequential sampling performs better than simultaneous sampling on D'Kitty and Ant in both random and poor settings. Especially on Ant in the poorest setting, ExPT-Sequential achieves improvements of  $20\%$  and  $19\%$  over ExPT in terms of the median and mean performance, respectively. Intuitively, as we add more data points to the context set, ExPT-Sequential is able to updates its understanding of the structure of the objective function, consequently leading to improved performance.
3.2.3 Forward modeling versus Inverse modeling
As we mentioned in Section 2.2, two possible approaches exist to pretrain ExPT on synthetic data. We take the inverse modeling approach for ExPT throughout the paper, as we train ExPT to directly produce design inputs  $x^{\prime}s$ . In this section, we empirically validate our design choices by comparing ExPT with TNP-ED, its forward counterpart. TNP-ED's architecture is similar to ExPT's in Figure 2 except that the target points now contain  $x_{m + 1:N}$  instead of  $y_{m + 1:N}$ , the decoder is a 1-layer MLP, and the predicted outputs are  $\hat{y}_{m + 1:N}$ . We call this model TNP-ED because a model named TNP [45] with a similar architecture was previously proposed in the context of meta-learning. We pretrain TNP-ED using a simple mean-squared error loss  $\mathcal{L} = \sum_{i = m + 1}^{N}(\hat{y}_i - y_i)^2$ . After pretraining, we condition TNP-ED on  $\mathcal{D}_{\mathrm{few-shot}}$  and the best inputs in this dataset, and perform gradient ascent with respect to these inputs to obtain better points.
Table 5 compares the performance of ExPT and TNP-ED on D'Kitty and Ant with respect to the median score and mean score. ExPT achieves significantly better performance in all metrics, especially in the poorest setting. This is because forward models suffer from poor out-of-distribution generalization, and performing gradient ascent on this model may result in points that have high values under the model but are very poor when evaluated using the true functions. This validates our inverse modeling approach.
Online ED The majority of existing approaches solve ED in an active setting, where the model is allowed to query the black-box function to collect more data. Many of these works are based on Bayesian Optimization [39, 46, 50, 53, 56], which typically employs a surrogate model to the black-box function and an acquisition function. The surrogate model is often a predictive model that can quantify uncertainty, such as Gaussian Processes [54], Neural Processes [20, 21, 26, 29, 51, 45], or Bayesian Neural Networks [24]. The acquisition function uses the uncertainty output by the surrogate model to trade off between exploration and exploitation for querying new points.
Offline ED Recent works have proposed to solve ED by learning from a fixed set of  $(x,y)$  pairs to bypass active data collection [58, 36, 57, 12, 34, 16, 7, 15, 65]. The Design-Bench benchmark [58] consists of several such tasks in the physical sciences and robotics and is used by many recent
works in offline ED. MINs [36] and BONET [34] perform optimization by generating designs  $x$  via conditioning on a high score value  $y$ . MINs use a GANs [25] model on  $(x, y)$  pairs and BONET casts offline ED as a sequence modeling problem. COMs [57] formulates a conservative objective function that penalizes high-scoring poor designs and uses it to train a surrogate forward model which is then optimized using gradient ascent. BDI [12] uses a bidirectional model consisting of a forward and backward models that learn mappings from the dataset to high-scoring designs and vice versa. In contrast to these works, we propose ExPT in the few-shot ED setting, where the model is given access to only the  $x' s$  during pretraining, and a handful of labeled examples for adaptation.
Synthetic Pretraining In the absence of vast amounts of labeled data, pretraining on synthetic data is an effective method for achieving significant gains in model performance. Prior works in this direction construct synthetic tasks which improve performance on diverse downstream tasks such as mathematical reasoning [64], text summarization [33], and perception tasks in vision [43]. Each synthetic task produces a dataset of labeled  $(x,y)$  values that can be used to train a model as usual for various objectives. Often, pre-training in this manner produces better results than simply pre-training on another real dataset. In this work, we demonstrate that pretraining on synthetic data generated from GPs can achieve significant generalization to downstream functions, leading to state-of-the-art performance on challenging few-shot optimization problems.
Few-shot learning Few-shot learning is a common paradigm in deep learning, where the model is pretrained on large amounts of data in an unsupervised manner. At test time, the model is given only a few examples from a downstream task and is expected to generalize [60]. This technique has found applications in text-generation (GPT-x) [9], image classification [55, 52], graph neural networks [18], text to visual-data generation [63], and neural architecture search [62, 10]. ExPT is capable of performing few-shot learning for black-box optimization in a variety of domains. Moreover, ExPT is pretrained on synthetically generated data with no prior knowledge of the downstream objective.
Inspired by real-world scenarios, this work introduces and studies the few-shot experimental design setting, where we aim to optimize a black-fox function given only a few examples. This setting is ubiquitous in many real-world applications, where experimental data collection is very expensive but we have access to unlabelled designs. We then propose ExPT, a foundation model style framework for few-shot experimental design. ExPT operates in two phases involving pretraining and finetuning. ExPT is pretrained on a rich family of synthetic functions using unlabeled data and can adapt to downstream functions with only a handful of data points via in-context learning. Empirically, ExPT outperforms all the existing methods by a large margin on all considered settings, especially improving over the second-best baseline by  $70\%$  in the more challenging setting.
Limitations and Future work In this work, we assume we have access to a larger unlabeled dataset for pretraining and the knowledge of the optimal value for optimization. While these assumptions are true in many applications and have been used widely in previous works, we would like to relax these assumptions in future work to improve further the applicability of the model. One more potential direction is to finetune the pretrained ExPT model on downstream data to further improve performance. Finally, we currently pretrain ExPT for each domain separately. We are interested in exploring if pretraining a big model that works for all domains is possible and if that helps improve performance in each individual domain.
This work is supported by grants from Cisco, Meta, and Microsoft.
[52] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[53] Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms, 2012.
[54] Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995, 2009.
[55] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[56] Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task bayesian optimization. Advances in neural information processing systems, 26, 2013.
[57] Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective models for effective offline model-based optimization. In International Conference on Machine Learning, pages 10358-10368. PMLR, 2021.
[58] Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Benchmarks for data-driven offline model-based optimization. In International Conference on Machine Learning, pages 21658-21676. PMLR, 2022.
[59] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[60] Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
[61] James T Wilson, Riccardo Moriconi, Frank Hutter, and Marc Peter Deisenroth. The reparameterization trick for acquisition functions. arXiv preprint arXiv:1712.00424, 2017.
[62] Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates. arXiv preprint arXiv:2101.07667, 2021.
[63] Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVI, pages 720-736. Springer, 2022.
[64] Yuhuai Wu, Markus N Rabe, Wenda Li, Jimmy Ba, Roger B Grosse, and Christian Szegedy. Lime: Learning inductive bias for primitives of mathematical reasoning. In International Conference on Machine Learning, pages 11251-11262. PMLR, 2021.
[65] Sihyun Yu, Sungsoo Ahn, Le Song, and Jinwoo Shin. Roma: Robust model adaptation for offline model-based optimization. Advances in Neural Information Processing Systems, 34: 4619-4631, 2021.
[66] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016.