ExPT: Synthetic Pretraining for Few-Shot Experimental Design
Authors: Tung Nguyen
Sudhanshu Agrawal
Aditya Grover
Venue: NeurIPS 2023 poster (2023)
Keywords: experimental design
few-shot
black-box optimization
synthetic pretraining
in-context learning
transformer
TL;DR: We introduce Experiment Pretrained Transformer (ExPT), a novel method that can solve challenging experimental design problems with only a handful of labeled data points.

Abstract: Experimental design is a fundamental problem in many science and engineering fields. In this problem, sample efficiency is crucial due to the time, money, and safety costs of real-world design evaluations. Existing approaches either rely on active data collection or access to large, labeled datasets of past experiments, making them impractical in many real-world scenarios. In this work, we address the more challenging yet realistic setting of few-shot experimental design, where only a few labeled data points of input designs and their corresponding values are available. We approach this problem as a conditional generation task, where a model conditions on a few labeled examples and the desired output to generate an optimal input design. To this end, we introduce Experiment Pretrained Transformers (ExPT), a foundation model for few-shot experimental design that employs a novel combination of synthetic pretraining with in-context learning. In ExPT, we only assume knowledge of a finite collection of unlabelled data points from the input domain and pretrain a transformer neural network to optimize diverse synthetic functions defined over this domain. Unsupervised pretraining allows ExPT to adapt to any design task at test time in an in-context fashion by conditioning on a few labeled data points from the target task and generating the candidate optima. We evaluate ExPT on few-shot experimental design in challenging domains and demonstrate its superior generality and performance compared to existing methods. The source code is available at https://github.com/tung-nd/ExPT.git.

ExPT: Synthetic Pretraining for Few-Shot Experimental Design
Tung Nguyen, Sudhanshu Agrawal, Aditya Grover
University of California, Los Angeles
ucla.edu, suchanshuagr270g. ucla. edu
Abstract
Experimental design for optimizing black-box functions a fundamental problem in many science and engineering fields. In this problem, sample efficiency is crucial due to the time, money, and safety costs of real-world design evaluations. Existing approaches either rely active data collection access to large, labeled datasets of past experiments, making them impractical in many real-world scenarios. In this work, address the more challenging yet realistic setting of few-shot experimental design, where only a few labeled data points of input designs and their corresponding values available. We introduce Experiment Pretrained Transformers (ExPT), foundation model for few-shot experimental design that combines unsupervised learning and in-context pretraining. In ExPT, we only assume knowledge of a finite collection of unlabelled data points from the input domain and pretrain transformer neural network to optimize diverse synthetic functions defined over this domain. Unsupervised pretraining allows ExPT to adapt to any design task at test time in an in-context fashion by conditioning on a few labeled data points from the target task and generating the candidate optima. We evaluate ExPT on few-shot experimental design in challenging domains and demonstrate its superior generality and performance compared to existing methods. The source code is available at ftttts:///// //g thub. com/ bung-nd/////n aad/ExPT.gg git
Introduction
The design of experiments to optimize downstream target objectives is a ubiquitous challenge across many science and engineering domains, including materials discovery (27). protein engineering 7
49 2] molecular
[22] design, mechanical design 38), and neural architecture optimization [66].
The key criterion of interest in experimental design (ED) is sample-efficiency. as the target objectives are often black-box functions and evaluating these objectives
any candidate design often involves expensive real-world experiments. A standard class of approaches learn surrogate to approximate the target objective and actively improve the approximation quality through online experiments [53] However, online data acquisition may be infeasible in the real world due to high costs, time constraints, or safety concerns. As an alternate, recent works have proposed offline ED 5 36 57 [2] 44)
wherein model lears to perform optimization from a fixed dataset of past experiments. While this more practical than the online setting, current offline methods and benchmarks assume access to impossible
large experimental datasets containing thousands of data points, which are hard to obtain in high-stake and emerging science problems. Even when these datasets exist, the past experiments might be of very poor quality resulting in poor surrogate learning and optimization.
In this paper, we aim to overcome these limitations for hyper-efficient experimental design that does not require large experimental datasets. To this end, we introduce few-shot experimental design, more challenging setting that better resembles real-world scenarios. We describe few-shot ED as two-phased
caditingaddatttioonn paradigm. In the pretraining phase, we only assume access to
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Robotics
Real data { Ji
Synthetic Pretraining
In-context Adaptation
X'
ExPT
ExPT
2x*
Molecules
y
DNA
Figure/ I: Experiment Pretrained Transformers (ExPT) follow pretraining-adaptatioon approach for few-shot experimental design. During pretraining (left). the model has access to unlabeled designs from domain X' without their corresponding scores. For adaptation (right), the model conditions on a small set of (design, score) pairs and the desired score y" to generate the optimal design x*.
unlabeled data, i.c., input designs without associated function values. During the adaptation phase, we have access to few labeled examples of past experiments adapt the model to the downstream task. This setup offers several advantages. First, alleviates the requirement for costly annotated data and relies mainly unlabeled inputs that are easily accessible. Second, unsupervised pretraining enables us to utilize the same pretrained backbone for adapting to multiple downstream optimization tasks within the same domain. For example, in molecule design, one may want to optimize for multiple properties, including drug-likenese synthesizability, or similarity to target molecules [7
The key question in this setup is how to make use of the unlabeled data to facilitate efficient generalization to downstream tasks during optimization. Our intuition here that, while the objective function is unknown, we can use the unlabeled inputs to generate pretraining data from other synthetic functions. a model can few-shot learn from a diverse and challenging set of functions, it should be able to generalize quickly to any target objective during the adaptation phase, in line with recent foundation models for language [9] and vision [3] This insight gives rise to our idea synthetic pretraining, wherein we pretrain the model data generated from rich family of synthetic functions that operate on the same domain as the target task. Specifically, for each function drawn from this family, sample set of points by using the unlabeled data as inputs. We divide these points into a small context set and target set, and train the model via in-context learning to perform conditional generation of the target input given the context points and the target value y. model that works well on this task should be able to efficiently capture the structures of the underlying function, i.e.. how different regions of the input space influence the function value, from small context set. By explicitly training the model to perform this task on diverse set of functions, the model can generalize efficiently to downstream functions during adaptation requiring only limited supervision. After pretraining, we can perform optimization by conditioning the model on a few labeled examples from the downstream task and generating an input that achieves the optimum y"
Inspired by recent advances in few-shot learning in language we instantiate a novel foundation model with a transformer-based architecture (59] which we call Experiment Pretrained Transformers (ExPT). ExPT is an encoder-decoder architecture, in which the encoder transformer [59] network that encodes the context points and the target value, and the decoder is a VAE 32 model that predicts the high-dimensional target input. The transformer
9 and other domains
encoder allows ExPT to perform few-shot generation and optimization purely through in-context learning in gradient-free fashion. We compare the performance of ExPT and various baselines on 2 few-shot settings created from Design-Bench (58) a standard database benchmark for ED, The two settings allow us to examine how different methods perform with respect to different quantities and qualities of few-shot data. both these settings, results show that ExPT achieves the highest average score and the highest average ranking with respect to median performance, mean performance, and best-achieved performance. Especially the more challenging setting, ExPT outperforms the second-best method by 70% in terms of the mean performance. Additionally, we explore the potential of using the same pretrained EXPT for multiple objectives, and conduct extensive ablation studies to validate the effectiveness of our design choices for synthetic data generation and ExPT architecture.

---

2 Experiment Pretrained Transformers
2.1 Problem setup
Let f X' + R be a black-box function that operates on a d-dimensional domain &' C R In experimental design (ED), the goal is find the input x* that maximizes f:
arg max fxx). x²
(1)
ZEX
Typically, is nigh-dimensional and complex function that often involves expensive physical experiments, Existing approaches either assume the ability to actively query I to collect data [53] or access large dataset of past experiments (58). Both assumptions are too strong in many real-world applications where data collection is hard or even impossible [T] Therefore, we propose few-shot ED, a more challenging yet realistic setting to overcome these limitations. In few-shot ED, the goal is to optimize any objective function in the domain X' given only a handful of examples. We approach this problem with a peetrannig--ddapaaiin
pipeline. In the pretraining phase, we assume access to an unlabeled dataset Dunlabeled = {x;), ) from the optimization domain X' C Rd, We note that unlahheedd only contains potential design inputs without their corresponding scores, for example, potential molecules in molecule optimization or different combinations of hyperparameters in neural architecture search. This means the objective function f is unspecified during pretraining.
During the adaptation phase, one can use the pretrained model to optimize any objective function in the same domain X. We now have access few-shot labeled dataset that the model can use to adapt to the downstream function Diew-shot = {(11, .1),.... (zn. yv.) ], in which yi = fx;)) and n = showw < (Duntabeteu]- After adaptation, we evaluate a few-shot optimization method by allowing it to propose Q input z's and query their scores using the black-box function S. where Q is The performance of a black-box optimizer is then measured by computing the max, median, and mean of the Q evaluations, This setup provides two key benefits. First, it resembles many real- real-world scenarios, where the potential design inputs cheap and easy to obtain while their target function values are expensive to evaluate. For example, in molecular optimization, we have databases of millions of molecules 30 5 43 but only the properties of a handful are known 28 47 2
often called the optimization budget (58 36
57
12
[4]
Second, unsupervised pretraining allows us to train a general backbone that we can adapt to multiple optimization tasks in the same domain.
2.2 Synthetic Pretraining and Inverse Modeling for Scalable Experimental Design
Intuitively, the adaptation phase in resembles a few-shot learning problem, in which a model is tasked to produce the optimal input x by conditioning on few labeled examples in Prew-shot- To perform well in this task, model has to efficiently capture the structure of a high-dimension function f, i.e. what regions of the function lead higher values and vice versa, from very few examples in Diew-shot- Given this perspective, the question now is how to make use of the unlabeled dataset D Dunlabeled to pretrain a model that achieves such efficient generalization to the objective function f. Our key insight is, if a model learns to perform in- in-context learning on diverse and challenging set of functions, it should be able to adapt quickly to any objective function at test time. While the function values are unknown during pretraining, we can use the unlabeled inputs x s's to generate pretraining data from other functions. This gives rise to our idea of synthetic pretraining, wherein we pretrain the model to perform few-shot learning on family of synthetic functions F that operate on the same input domain X' of the objective f. We discuss in detail our mechanism for synthetic data generation in Section 2.3 For each function ʃ generated from F. we sample set of function evaluations { y4) b that we divide into a small context set { v4) } and a target set { 91) 3 We train the model to predict the target points conditioning on the context set.
There are two different approaches to pretraining a model on this synthetic data. The first possible approach is forward modeling, where the model is trained to predict the target outputs +m++1:V given the context points and the target inputs 1m+1:N. This is similar to the approach followed by TNPs 451, model recently proposed in the context of meta-learning. During adaptation, we can condition the model on the labeled examples Diew-shot and perform gradient ascent updates to improve an existing design input 2;, However, as commonly observed in previous works (58] 36 (7). this approach susceptible to producing highly suboptimal inputs. This is because performing gradient ascent with respect to an imperfect forward model may result in points that have high values
3

---

ExPT
Decoder
Decoder
hy
Transformer Encoder
J
Synthetic
81
J2
Bin
y/x
Pretraining
context points
target points
Adaplation
Few-shot examples
optimal value
Figure 2: The pretraining-adaapation phases for ExPT. We sample synthetic data from f and pretrain the model to maximize log p( p(mm-1:N Zi:m: Vimm Bm+11N) ). At adaptation, the model conditions on Deewsshot and y" to generate candidates. ExPT employs a transformer encoder that encodes the context points and target outputs and relevant decoder that predicts the target inputs.
under the model but are poor when evaluated using the real function. Instead, we propose to perform inverse modeling, where the model learns to predict the inputs 1m+1:N given the output values 3m++1:N and the context points. As the model learns directly generate input x's, it is less vulnerable to the aforementioned problem. Another advantage of inverse modeling is after pretraining, we can simply condition on Diew-shot and the optimal value y" generate the candidate optima. Our loss function for pretraining the model is:
0 arg max R. WwN. f(tiN))
[log p(#m+11: Fl:m; F1am Mun 1:N )]
N
(2)
arg max E. F. NN N Duntaee..n (ZtiN)
Σ lim++1
log p(a Dioms Fioms vi)
where assume the target points independent given the context set and the target output. Figure illustrates the proposed pretraining and adaptation pipeline. Typically, we use small context size m during pretraining resemble the scenario.
After pretraining, ExPT can adapt to any objective ff in the domain in a gradient-free fashion. Samples in the few-shot dataset Drew-sho become the context points and the model conditions only one target y". which is the optimal value of f. generate candidate optima. Note that we only assume the knowledge of y² and not x². This assumption is common in many prior works RE In practice. y" might be known based on domain knowledge. For example, in molecule design, there physical limits on the value of certain properties such relaxed energy, in robot applications, the optimal performance can be computed from the cost function, and in neural architecture search, we know the theoretical limits on the highest possible accuracy for classifiers.
46
K
14]
Next, we present the details of synthetic data generation and our proposed model architecture, the two components that constitute proposed foundation model, which we refer to Experiment Pretrained Transformers (ExPT).
2.3 Data generation
We need family of functions to generate synthetic data for pretraining ExPT. A good family of functions should be easy sample from and should be capable of producing diverse functions. Many possible candidates exist for synthetic function families, such as Gaussian Processes (GPs), randomly constructed Gaussian Mixture Models, or randomly initialized or pretrained neural networks. Among these candidates, we choose to generate synthetic data from Gaussian Processes with an RBF kernel. This for several reasons. First, they are natural choice as they represent distributions over functions. Second, it is easy and cheap to sample data from prior GPs. And third, GP with an RBF

---

kernel is a universal approximator to any function
44 Specifically. f is sampled as follows,
<(x,x') =0² exp (- (xr x²)²
f f~ (P(,
2
in which and are the two hyperparameters of the RBF kernel. The variance scales the magnitudes of the covariance matrix. A larger variance results in a wider range of function values, while a smaller variance restricts the function values narrower range. On the other hand, the length scale & determines how strongly the covariance matrix varies with respect to the distance between & and a'. very small length scale l means the kernel is sensitive to the distance hetween and a', leading to sharp transitions between neighboring points and a lack of smoothness in the functions. In contrast, if l is too large, the covariance between points will be similar for both near and distant points, leading to function values that are very similar. In other words, too large length scale reduces the diversity of the synthetic functions. In practice, we randomize both o and E to increase the diversity of the pretraining data. Appendix demonstrates the empirical importance of these hyperparameters.
2.4 Model architecture
To optimize the loss function in Equation 2 we need model architecture that can condition on a few examples drawn from an underlying function to make predictions for other points, This resembles the idea of in-context learning that has proven very successful in language [9] [2]
and other domains [44 445 19 37). The key to success in these works is a transformer architecture that performs in-context learning efficiently via the attention mechanism [59] Inspired by this, we instantiate ExPT with a wransformer-base architecture. Figure 2 iillssraate the ExPT overall architecture. Specifically, ExPT employs transformer encoder that encodes the context points ((r,, y4) y;) and the target inputs Um+1I:N and outputs hidden vectors hm+1-N- To inform the model that x; and yi are the input and the corresponding output from f. we concatenate them to form a token. This results in the sequence (1)1,,)).... (ymes Jn 3mm+1...........................
yNN ). We then embed these tokens using two 1 I-layer MLP networks, one for the pairs and one for the target inputs, before feeding the sequence to the transformer layers. We implement a masking mechanism that prevents the context points from attending the target points, as they do not contain information about the underlying function f.
Each hidden vector h; output by the transformer encoder encompasses the information of the context points and the target input yr- Therefore, given h,, the conditional probability (o 1imm Vi: Mi) reduces to po (a hs). As æ; is high-dimensional, we can utilize existing generative models to model the conditional distribution p(æ; lx.). In this work, we train a conditional VAE model (32
alongside the transformer encoder because of its training stability, light hyperparameter tuning, and
good empirical performance. For discrete tasks, we follow the same procedure as Trabucco al. [58]
that emulates logit values by interpolating between uniform distribution and the one hot values. We train the entire model by maximizing the lower bound of the conditional likelihood log p(z; h:):
log po (Ixx kc) 2 Bag (slxs.hh)) [log 2, h+)]
(L(ye(2 Ix, h=)ll((=)),
(4)
in which g0(2 n,, h2) is the encoder of the conditional VAE and p(2) is a standard Gaussian prior.
Experiments
3.1 Synthetic experiments
We first evaluate the performance of ExPT in a synthetic experiment, where we train ExPT on data generated from Gaussian Processes (GPs) with an RBF kernel and test the model on four out-of-distributio functions drawn from four different kernels: Matern, Linear, Cosine, and Periodic. Figure ssoows the performance of ExPT on four test functions through the course of training. The model performs well on all four functions, achieving scores that are much higher than the max value the few-shot dataset, and approaching the true optimal value, even for kernels that are significantly different from RBF like Cosine and Periodic. Moreover, the performance improves consistently as we pretrain, showing that pretraining facilitates generalization to ouiti--distributtin functions with very few examples. See Appendix A for detailed setup of this experiment.
5

---

Matem
Liner
Cosine
Portodic
1
1.0
1.0
Bazt value
0
0,5
0.5
0.5
0.0
0.0
0.0
Pretraining step (1e3)
Pretraining step (1e3)
Pretraining step (1c3)
Pretraining step (1e3)
SynTO
(Xbest)
Figure 3: The performance of ExPT on 4 out-oo-disstriutioo synthetic tasks through the pretraining phase. We average the performance across 3 seeds.
3.2 Design-Bench experiments
Tasks We consider tasks from Design-Beneh 158 D'Kitty and Ant are continuous tasks with input dimensions of 56 and 60, respectively, In D'kitty and Ant, the goal is to optimize the morphological structure of two simulated robots, Ant (6] to run as fast as possible, and D' D'kitty [I] to reach fixed target location. TF Bind 8 and TF Bind 10 are two discrete tasks, where the goal is to find the length-8 and length-10 DNA sequence that has a maximum binding affinity with the SIX6 REF R1 transcription factor. The design space in these two tasks consists of sequences of one of four categorical variables, corresponding to four types of nucleotide. For each task, Design-Bench provides public dataset, larger hidden dataset which is used to normalize the scores, and an oracle. We have an exact oracle to evaluate the proposed designs in all tasks we consider.
Few-shot settings We create 2 few-shot settings from the above tasks, which call random and poorest. random, we randomly subsample 1% of data points in the public set of each task as the few-shot dataset Sten The poorest setting is more challenging, where use 1% of the data points which have the lowest scores. The two settings examine how sensitive different methods are to the quantity and quality of the data. In both settings, use r's s in the public dataset unnlabeled-
ExPT details For each domain, we pretrain ExPT for 10,000 iterations with 128 synthetic functions in each iteration, corresponding to a total number of 1,280,000 synthetic functions. For each function, we randomly sample 228 input x's from the unlabeled dataset 'D Punlaheled and generate the values y's from Gaussian Process with RBF kernel. To increase the diversity of synthetic data, randomize the two hyperparameters, length scale ~ [[500, 10.0] and function scale a ~ [[[00, 1.0], when generating each function. Additionally, we add Gaussian noises (O, 0.1) each input sampled from Dunlabeled to enlarge the pretraining inputs. For each generated function, we use 100 points context points and the remaining 128 as target points, and train the model to optimize ② During the adaptation phase, we condition the pretrained ExPT model on the labeled few-shot dataset D Dtew-shor and the target function value y" to generate designs x's.
Baselines We compare ExPT with BayesOpt (GP-qEI) 53 a canonical ED method, and MINs Kd.. COMs [57] [2] and BONET 34 four recent deep learning models that have achieved state-of-the-arr performance in the offline setting. To adapt GP-qEI to the few-shot setting, we use feedforward network trained on few-shot data to serve as oracle, Gaussian Process to quantify uncertainty, and the quasi-Expected Improvement [6]] algorithm for the acquisition function. For the deep learning baselines, we train their models on the few-shot dataset D Piew-shor using the hyperparameters reported in their original papers.
Evaluation For each considered method, we allow an optimization budget Q = 256. We report the median score, the max score, and the mean score among the 256 proposed inputs. Following previous works, we normalize the score to [0, 1] by using the minimum and maximum function values from Vhiin
large hidden dataset Ynorm Muss Muis 3 independent runs for each method.
We report the mean and standard deviation of the score across
Results Table [ shows the performance of different methods in the random setting. Most methods perform well in the random setting, where ExPT achieves the highest average score and the best average rank across all 3 performance metrics. For each of the tasks and metrics considered, ExPT is either the best or second-bes performing method. Notably, in Ant, ExPT significantly outperforms the best baseline by 18%, 9%, and 10% with respect to the median, max, and mean performance,
We exclude domains where the oracle functions are flagged to be highly inaccurate and noisy in prior works (ChEMBL, Hopper, and Superconductor), too expensive evaluate (NAS). See Appendix B for more details.

---

Table 1: Comparison of ExPT and the baselines on the few-shot random setting of Design-Bench tasks. We report median, max, and mean performance across 3 random seeds. Higher scores and lower ranks are better. Blue denotes the best entry in the column, and Violet denotes the second best. TF Bind
Baseline
D'Kiuy
Ant
Bind
Mean score (1)
(J)
(ioesta(best)
0,883
0.563
0.439
0.406
MINs
0.859 1: 0.0144
0.455 1 0.152
0.416 1: 0.019
0.468 1 0.001
0.537 2 0.050
4.0
COMs
0.752 +: 0.007
0.411 + 0.012
0.371 + -00001
0.468 0.000
0.501 4 0.005
4.0
BONET
0.852 0.013
0.597 0.119
0.441 0.000
0.483 0.000
0.593 0.036
2.3
Median
BDI
0.582 2 0.020
0.396 + 0.018
0.540 +: 0.032
0.436 0.031
0.192 0.026
4.8
GP-qEI
0.842=0.000
0.550 0.007
0.439 + 0.000
0.467 0.000
0.576
EXPT
0.902
0.006
0.705 + 0.018
0.473 + 0.014
0.477 0.014
0.639 + 0.013
1.5
MINs
0.930
0.010
0.890
0.017
0.814 + 0.030
0,639
0.017
0.818 + 0.019
3.3
COMs
0.920 0.010
0.841 1 0.044
0.686 1. 0.152
0.656 1 0.020
0.776 1 0.057
4.0
BONET
0.909 0.012
0.858 + 0.024
0.887 - 0.053
0.702 0.0000
0,847 = 0.024
3.0
Max
BDI
0.918 = 0.006
0.800 + 0.094
0.906 + 0.074
0.532 0.023
0.791 + 0.049
4.5
GP-qEI
0.896 2 0,000
0.887
+ 01,00
0.513 + 0,2004
0,647 0.011
0,736 2 0.029
5.0
EXPT
0.005
0.970
0.004
0.933 +
0.077
MINS
0.621 0.025
0.003
0.415 1 0,030
0,465 0.005
0.378 0
COMs
0.515 + 0.050
0.020 + 0.006
0.369 + 0.003
0.471 + 0.004
0.34 0.016
4.8
BONET
0. 1.837 0.023
0.579 + 0.024
0.418 + 0.011
0.481 0.009
0.587 = 0.017
2.0
Mean
BDI
0.570 032
0.385 + 0.012
0.536 + 0 0.032
0.444 0.027
0,484 0.026
3.5
GP-qEI
0.505
0.006
0.019
+ 0.001
0.439
+ 0.001
0.473 + 0.002
0.359 + 0.003
4.5
EXPT
0.865 0.016
0.639 0.026
0.476 0.010
0.015
0.017
Table 2:
Comparison of EXPT and the baselines on the few-shot poorest setting of 5 Design-Bench tasks. We report the median, max, and mean performance across random seeds. Higher scores and lower ranks are better. Blue denotes the best entry in the column, and Violet denotes the second best.
Baseline
D'Kiny
Ant
TF Bind 8
TF Bind 10
Mcan score (1)
Mean rank (1)
(bbss))
0.307
0.124
0.124
0.000
MINs
0.480 1 0,156
0.316 1: 0.040
0.137 = 0.007
0,463 =: 0.003
0.124 = 0.062
3.5
COMs
0.733 + 0.023
0.401
0.026
0.111 +: 0.000
0.459 0.006
0.426
0.014
4.3
BONET
0.310 1: 0.000
0.236
1: 0.047
0.319 1: 0.018
0.461 = 0.017
0,332 :: 0.021
4.8
Median
BDI
0.309 + 0.000
0.192
0.012
0.365 + 0.000
0.454 + 0.017
0.330
0.007
5.5
GP-gIII
0.565
0.001
0.439
0.467
0.000
0.589
2.0
EXPT
0.922 + 0.009
0.686
0.090
0.552 + 0.042
0.489 0.013
0.662
0.039
1.0
0.811 0.014
0.721
0.031
0.962
0.019
0.793
3.3
COMs
0.331 1 0.022
0.843 1. 0.020
0.124 1 0.000
0.739
0.057
0.659
0.025
3.3
BONET
0.929 1 0.031
0.357 : 0,118
0,809 0.038
0.519
0.039'
0,704 0.055
5.0
0.939
0.002
0.603 0,109
0.913
0.000
0.596
0.020
0.785
0.033
3.5
GP&EI
0.896 1 0.00%)
0.887 1 0. 0.000
0.439 + 0.0010
0.645 = 0.021
0,717 = 0.000
3.8
EXPT
0.946 0.018
0.965
0.004
0.873
0.035
0.615 0.022
0.850
0.020
2.3
MINS
0.623 0.051
0.015 0.007
0,466 + 0.008
0.463 + 0.002
0.311 0.023
3.3
COMs
0.807 + 0.021
0.033 = 0.003
0.109 = 0.001
0,154 = 0.001
0 0.007
1.8
BONET
0.490 + 0.023
0.234 + 0.052
0.318 + 0.018
0.459 0.018
0.375 0.028
4.0
Mcan
0.361 0.001
0.215 0.021
0,153
0.011
GP-qEI
0.533 + 0.001
0:018 :: 0.000
0-139 = 0.000
0.470 = 0.002
0.365 0.001
3.5
EXPT
0.871 0.018
0.646 0.061
0.549
0.032
0.488
0.011
0.639 0.031
1.0
respectively. Only ExPT and BONET achieve a meaningful performance in Ant when considering the mean score. BONET is also the overall second-bess method in this setting.
Table 2 shows the superior performance of ExPT in few-shot poorest, the more challenging setting. ExPT achieves the highest score in 10/12 12 individual tasks and metrics, and also achieves the highest score and the best rank across tasks on average. Notably, in terms of the mean score, ExPT beats the best baseline by large margin, achieving an improvement of 40%, 176%, and 18% on D'Kitty, Ant, and TF Bind and 70% on average. The performance of most baselines drops significantly from the random to the poorest setting, including BONET, the second-best method in the random setting. This was also previously observed in the BONET paper RE Interestingly, the performance of ExPT, MINs, and GP-qEI is not affected much by the quality of the few-shot data, and even improves in certain metrics. We hypothesize that even though the dataset is of lower quality, it may contain specific anti-correlation patterns about the problem that the model can exploit.
Pretraining analysis In addition to the absolute performance, we investigate the performance of EXPT on downstream tasks through the course of pretraining. Figure y shows that the performance of EXPT in most tasks improves consistently as the number of pretraining steps increases. This shows that synthetic pretraining on diverse functions facilitates the generalization to complex real-world functions. Ant, the performance slightly drops between 4000 and 1 10000 iterations. Therefore, we can further improve ExPT if we have way to stop pretraining point that likely leads to the best
7

---

Dkitty
Arf
TF8
TF10
0.90
0.50
Modii 10001-000
0.83
0.45
0.43
0.40
0.45
0.80
0.35
0.50
0.83
8.00
9001
080
0.5
10
10
9:35
10
SynTO
(Xbest
Figure 4: The median and mean performance of ExPT of 4 Design-Bench Bench tasks through the course of pretraining. We average the performance across seeds.
Table 3: ExPT's performance on different objectives in D' D'Kitty and Ant domains. We pretrain one model for all tasks in the same domain. The performance is averaged across 3 seeds.
Task
D'Kitty
D'Kiny-45 D' D'Kity-60
Ant-ty
Ant Ant-Energy
Dus
0.307
0.297
0.344
0.124
0.210
0.189
Median Max
ExPT
0.922 + 0.009
0.611 + 0.007
0.569
0.010
0,686
0.0 0.000
0.613
0.009
0.635 0.025
EXPT
0.976 + 0.004
0.954 + 0.008
0.973 + 0.004
0.965
0.004
0.923
0.049
0.950 + 0.033
Mean
ExPT
0.871 + 0.018
0.619 0.016
0.584 + 0.008
0.646
0.061
0.599
0.005
0.608 0.025
performance downstream tasks. However, in practice, we do not have the luxury of testing on real functions during pretraining. Alternatively, could perform validation and early stopping set of held-out, duittf--iittibuttin synthetic functions. We leave this to future work.
3.2.1 ExPT for few-shot optimization of multiple objectives
As we mention in Section 2 one advantage of unsupervised pretraining is the ability to optimize for multiple objectives during the adaptation phase. In this section, show that the same pretrained ExPT model is capable of optimizing different objectives D' Kitty and Ant domains. We create two variants of the original D' Kitty, namely D' Kitty-45 and D' Kitty-60, whose objectives the robot to goals that are 45 and 60 away from the original goal, respectively. For Ant, we create Ant-r, where the goal is run fast as possible the vertical direction (as opposed horizontal X direction in Ant) direction, and Ant-Energy, where the goal is to preserve energy. We detail how to construct these tasks in Appendix C We use the same pretrained models for all these tasks. During adaptation, the model conditions on the Djew-shox and y" for each task for optimization.
navigate
We evaluate ExPT on these tasks in the poorest setting. Table 3 shows that ExPT performs well on all tasks, where the median and mean scores are better than the best value in Diew-shot, and the max score is close to 1. For the Ant, Ant-uy, and Ant-Energy tasks, visualize the behavior of the optimal designs that are discovered at ftjttttss/immmeeee nolicy-network (optimizing for horizontal X speed), the robots optimized for different objectives behave differently; the optimal Ant is capable of leaping forward to move quickly in X; Ant-v, is able to jump up maximize speed in Y; Ant-Energy is capable of 'sitting down' conserve energy.
com/ a a/zpgI4YL, When subject to the same
3.2.2 ExPT for closed-loop optimization
ExPT can adapt any objective function purely through in-context learning. This means that the model can refine its understanding of the underlying objective function given more data points in very efficient manner. In this section, we explore an alternative optimization scheme for ExPT. namely sequential sampling, which explicitly utilizes the in-context learning ability of the model. Specifically, instead of producing Q = 256 inputs simultaneously, we sample one by one sequentially. That is, we condition the model on Diew-shot and y" sample the first point, evaluate the point using the black-bor function, and add the point together with its score to the context set. We repeat this process for 256 times. This process is often referred to as closed-loop optimization in ED.

---

Table 4: Comparison of simultaneous (ExPT) and sequential (ExPT-Seq) sampling on Ant and D' Kitty in random (left) and poorest (right) settings. We average the performance across 3 seeds.
Baseline
D' D'Kitty
Ant
Baseline
D'Kity
Ant
(bs-da((bea))
0.883
0.563
Dinosser (best)
0.307
0.124
Median
ExPT-Seq
ExPT
0.903 0.005
0.302 + 0.006
0.719
0.705 + 0.018
Median
ExPT-Seq
ExPT
0.928 + 0.012
0.922 + 0.009
0.822 :: 0.055
0.686 + 0.080
ExPT
0,865
0.016
0.639 +: 0.026
ExPT
0.871 - 0.018
0.646 - 0.061
Mean
ExPT Seq
0.872 0.010
0.669 1: 0.017
Mean
EXPT Seg
0.923 1 0.011
0.767 1: 0.048
Table 5: Comparison of inverse modeling (ExPT) versus forward modeling (TNP-ED) on Ant and D' Kiity in random (left) and poorest (right) settings, We average the performance across 3 seeds.
Bascline
Kitty
Ant
Baseline
D'Kitty
Ant
(bessta(bestt
0,883
0.563
(bestt
0.307
0.124
Mediam
TNP-ED
ExPT
0.902 1 0.006
0.770 + 0.009
0.705 :: 0.018
0.138 = 0.007
Median
TNP-ED
EXPT
0.922 1: 0. 0.009
0.309 +: 0.000
0.686 1: 0.100.0000 0.090
0.197 = 0,005
EXPT
0 886 + 0.016
0.639 = 0 026
EXPT
0.871 + 0.018
0.646 : 061
Mcan
TNP-ED
0.670 + 0.037
0.451 + 0.018
Mean
TNP-ED
0.405 + 0.004
0.237 0.005
Table 4 shows that ExPT with sequential sampling performs better than simultaneous sampling on D' Kitty and Ant in both random and poor settings. Especially on Ant in the poorest setting, ExPT- Sequential achieves improvements of 20% and 19% over ExPT in terms of the median and mean performance, respectively. Intuitively, as we add more data points to the context set, ExPT-Sequential able to updates its understanding of the structure of the objective function, consequently leading to improved performance.
3.2.3 Forward modeling versus Inverse modeling
As we mentioned in Section 2.2 two possible approaches exist to pretrain ExPT on synthetic data. We take the inverse modeling approach for ExPT throughout the paper, as we train ExPT to directly produce design inputs x's s. In this section, we empirically validate our design choices by comparing ExPT with TNP-ED, its forward counterpart. TNP-ED's s architecture is similar to ExPT's in Figure 2 except that the target points now contain Im+1:N instead of 1m++1NN the decoder is a 1-layer MLP. and the predicted outputs are fm+1:N. We call this model TNP-ED because a model named TNP 45 with a similar architecture was previously proposed in the context of meta-learning We pretrain TNP-ED using a simple mean-squared error loss L E (0: 3.) After pretraining, we condition TNP-ED on Diew-shot and the best inputs in this dataset, and perform gradient ascent with respect these inputs to obtain better points.
Table 5 compares the performance of EXPT and TNP-ED on D'Kitty and Ant with respect to the median score and mean score. ExPT achieves significantly better performance in all metrics, especially in the poorest setting. This is because forward models suffer from poor out-of-distribution generalization, and performing gradient ascent on this model may result in points that have high values under the model but are very poor when evaluated using the true functions. This validates our inverse modeling approach.
4 Related work
Online ED The majority of existing approaches solve ED in an active setting, where the model is allowed to query the black-box function to collect more data. Many of these works are based
on Bayesian Optimization 39 H6
56 which typically employs a surrogate model to the black-box function and an acquisition function. The surrogate model is often a predictive model that
50
53
can quantify uncertainty, such as Gaussian Processes [54]. Neural Processes (20 21 26 29
51 45).
The acquisition function uses the uncertainty output by the surrogate model to trade off between exploration and exploitation for querying new points.
or Bayesian Neural Networks 24!
Offfine ED Recent works have proposed to solve ED by learning from fixed set of (æ; y) pairs
to bypass active data collection [58 5d 57
15 65)
The Design- Bench benchmark
5 consists of several such tasks in the physical sciences and robotics and is used by many recent

---

34] perform optimization by generating designs x via works in offline ED. MINs EO and BONET
model on (z, y) pairs and BONET formulates a conservative objective function that penalizes high-scoring poor designs and uses to train surrogate forward model which is then optimized using gradient ascent. BDI [12] uses bidirectional model consisting of a forward and backward models that learn mappings from the dataset to high-scoring designs and vice versa. In contrast to these works, propose ExPT in the few-shot ED setting, where the model is given access to only the r's during pretraining, and a handful of labeled examples for adaptation.
conditioning on a high score value y. MINs uses a GANs
25
casts offline ED sequence modeling problem. COMs
57
Synthetic Pretraining In the absence of vast amounts of labeled data, pretraining on synthetic data an effective method for achieving significant gains in model performance. Prior works in this direction construct synthetic tasks which improve performance on diverse downstream tasks such as mathematical reasoning (44, text summarization [%), and perception tasks in vision [5 Each synthetic task produces dataset of labeled (x, v) values that can be used to train a model as usual for various objectives. Often, pre-training in this manner produces better results than simply pre-training on another real dataset. In this work, we demonstrate that pretraining on synthetic data generated from GPs can achieve significant generalization to downstream functions, leading to state-of-the-art performance on challenging few-shot optimization problems.
Few-shot learning Few-shot learning is a common paradigm in deep learning. where the model is pretrained on large amounts of data in an unsupervised manner. At test time, the model is given only a few examples from downstream task and expected generalize 600 . This technique has found
applications in text-generation (GPT-x) 19), image classification
55 52 graph neural networks
ExPT is capable of performing few-shot learning for black- box optimization in variety of domains. Moreover, ExPT is
[8] text to visual-data generation [8). and neural architecture search 62
pretrained on synthetically generated data with no prior knowledge of the downstream objective.
Conclusion
Inspired by real-world scenarios, this work introduces and studies the few-shot experimental design setting, where we aim to optimize a black-fox function given only a few examples. This setting is ubiquitous in many real-world applications, where experimental data collection is very expensive but we have access to unlabelled designs. We then propose EXPT, a foundation model style framework for few-shot experimental design. ExPT operates in two phases involving pretraining and finetuning. ExPT is pretrained on a rich family of synthetic functions using unlabeled data and can adapt to downstream functions with only handful of data points via in-context learning. Empirically, ExPT outperforms all the existing methods by large margin on all considered settings, especially improving over the second-best baseline by 70% in the more challenging setting.
Limitations and Future work In this work, we assume we have access to a larger unlabeled dataset for pretraining and the knowledge of the optimal value for optimization. While these assumptions are true in many applications and have been used widely in previous works, we would like to relax these assumptions in future work improve further the applicability of the model. One more potential direction is to finetune the pretrained ExPT model on downstream data to further improve performance. Finally,
currently pretrain ExPT for each domain separately. We interested in exploring if pretraining a big model that works for all domains is possible and if that helps improve performance in each individual domain.
Acknowledgements
This work is supported by grants from Cisco, Meta, and Microsoft.
10

---

References
[1] Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. Robel: Robotics benchmarks for learning with low-cost robots. In Confer- ence on robot learning, pages 1300-1313, PMLR, 2020.
[2] Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. 2020.
[3] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting, Advances in Neural Information Processing Systems, 35: 25005 25017, 2022.
[4] Felix Berkenkamp, Angela P Schoellig, and Andreas Krause. Safe controller optimization for quadrotors with gaussian processes. In 2016 IEEE international conference on robotics and automation (ICRA), pages 491-490 IEEE, 2016.
[5] Lorenz C Blum and Jean-Louis Reymond. 970 million druglike small molecules for virtual screening in the chemical universe database gdb-13, 13. Journal of the American Chemical Society, 131(25))8732-8735 2009.
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1606v/6006.01 2016.
[7] David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In International conference on machine learning, pages 773-782. PMLR, 2019.
[8] Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C. Vaucher. Guacamol: bench- marking models for de novo molecular design. Journal of chemical information and modeling, 59(3):1096-1 1108, 2019.
[9] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
[10] Kaidi Cao, Jiaxuan You, Jiaju Liu, and Jure Leskovec. Autotransfer: AutoML with knowledge transfer- an application to graph neural networks, In The Eleventh International Conference on Learning Representations, 2023. URL URL https://ppeerreeiee www./eeeoouttee htwp://wwwwwee..e.e.ffooooorrrra
[11] Ian Char, Youngseog Chung, Willie Neiswanger, Kirthevasan Kandasamy, Andrew Oakleigh Nelson, Mark D Boyer, Egemen Kolemen, and Jeff Schneider. Offline contextual bayesian optimization for nuclear fusion. In 33rd Conference on Neural Information Processing Systems (NeurIPS), pages 1-12, 2020.
[12] Can Chen, Yingxueff Zhang, Jie Fu, Xue Steve Liu, and Mark Coates. Bidirectional learning for offline infinite- width based optimization. Advances in Neural Information Processing Systems, 35:29454-29944 2022.
[13] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 34:15084-15097, 2021.
[14] Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What essential for offline rl via supervised learning? arXiv preprint 22.iv:21 2021.
[15] Clara Fannjiang and Jennifer I Listgarten. Autofocused oracles for model-based design. Advances in Neural Information Processing Systems, 33:12945-1200 -22566, 2020.

---

[16] Justin Fu and Sergey Levine. Offline model-based optimization via normalized maximum likelihood estimation, arXiv preprint arXiv:2102.07970, 2021.
[17] Wenhao Gao, Tianfan Fu, Jimeng Sun, and Connor Coley, Sample efficiency matters: benchmark for practical molecular optimization. Advances in Neural Information Processing Systems, 25:21342-21357, 2022.
[18] Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks, 2018.
[19]
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
[20] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International conference on machine learning, pages 1704-1713. PMLR, 2018.
[21] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv: 2018.
[22] Anna Gaulton, Louisa J Bellis, A Patricia Bento. Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research, 40(01):D1100-
D1107, 2012.
[23] Anna Gaulton, Anne Hersey, Michat Nowotka, A Patricia Bento, Jon Chambers, David Mendez, Prudence Mutowo, Francis Atkinson, Louisa Bellis, Elena Cibrián-Uhalte, et al. The chembl database in 2017. Nucleic acids research, 45(D1):9945-D954, 2017.
[24] Ethan Goan and Clinton Fookes. Bayesian neural networks: An introduction and survey. Case Studies in Applied Bayesian Data Science: CIRM Jean-Morlet Chair, Fall 2018, pages 45-87, 2020.
[25] Ian Goodfellow, Jean Pouget-Abadics Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1):13))111111 144, 2020.
[26] Jonathan Gordon, Wessel P Bruinsma, Andrew YK Foong, James Requeima, Yann Dubois, and Richard Turner. Convolutional conditional neural processes. In International Conference on Learning Representations.
[27] Kam Hamidich. data-driven statistical model for predicting the critical temperature of superconductor. Computational Materials Science, 154:346-354 2018.
[28] John J Irwin and Brian K Shoichet. Zinc- free database of commercially available compounds for virtual screening. Journal of chemical information and modeling, 45(1):177-182, 2005.
[29] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum. Oriol Vinyals, and Yee Whye Tch. Attentive neural processes. In International Conference on Learning Representations.
[30] Sunghwan Kim, Paul A Thiessen, Evan Bolton, Jie Chen, Gang Fu, Asta Gindulyte, Lianyi Han, Jane He, Siqian He, Benjamin A Shoemaker, et al. Pubchem substance and compound databases. Nucleic acids research, 44011)D1202-D1213,, 2016.
[31] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
[32] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1312.6 14, 2013.
[33] Kundan Krishna, Jeffrey Bigham, and Zachary C Lipton. Does pretraining for summarization require knowledge transfer? arXiv preprint 100.v22109.04953, 2021.
12

---

[34] Siddarth Krishnamoorthy, Satvik Mehul Mashkaria, and Aditya Grover. Generative pretraining for black-box optimization. arXiv preprint arXiv:2206. 2022.
[35] Siddarth Krishnamoorthy, Satvik Mehul Mashkaria, and Aditya Grover. Diffusion models for black-box optimization. arXiv preprint arXiv:2306.07180 2023.
[36] Aviral Kumar and Sergey Levine. Model inversion networks for model-based optimization. Advances in Neural Information Processing Systems, 33:5126-5137, 2020.
[37] Michael Laskin, Luyu Wang. Junbyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steiger- wald, DJ Strouse. Steven Stenberg Hansen, Angelos Filos, Ethan Brooks, et al. In-context reinforcement learning with algorithm distillation. In NeurlPS 2022 Foundation Models for Decision Making Workshop.
[38] Thomas Liao, Grant Wang, Brian Yang. Rene Lee, Kristofer Pister, Sergey Levine, and Roberto Calandra. Data-efficient learning of morphology and controller for a microrobot. In 2019 International Conference on Robotics and Automation (ICRA), pages 2488-2494, IEEE, 2019.
[39] Daniel James Lizotte. Practical bayesian optimization. 2008
[40] Uya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations.
[41] Charles A Micchelli, Yuesheng Xu, and Haizhang Zhang. Universal kernels. Journal of Machine Learning Research, 7(12), 2006.
[42] Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2791-2809, 2022.
[43] Samarth Mishra, Rameswar Panda, Cheng Perng Phoo, Chun-Fu (Richard) Chen, Leonid Karlinsky, Kate Saenko, Venkatesh Saligrama, and Rogerio S. Feris. Task2sim: Towards effective pre-training and transfer from synthetic data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9194-9204, June 2022.
[44] Samuel Müller, Noah Hollmann, Sehastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. arXiv preprint 12. 2021.
[45] Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. arXiv preprint arXiv:2207.04177 2022.
[46] Vu Nguyen and Michael A. Osborne. optimization, 2020.
Knowing the what but not the where in bayesian
[47] Raghunathan Ramakrishnan, Pavlo Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014.
[48] Lars Ruddigkeit, Ruud Van Deursen, Lorenz Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. 17. Journal of chemical information and modeling, 52(11):2864-287 2012.
[49] Karen S Sarkisyan, Dmitry A Bolotin, Margarita Meer, Dinara R Usmanova, Alexander Mishin, George V Sharonov, Dmitry N Ivankov, Nina G Bozhanova, Mikhail S Baranov, Onuralp Soylemez, et al. Local fitness landscape of the green fluorescent protein. Nature, 533 (7603):397-401, 2016.
[50] Bobak Shahriari. Kevin Swersky, Ziyu Wang, Ryan Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104 (1):148-4)) 175, 2015.
[51] Gautam Singh, Jaesik Yoon, Youngsung Son, and Sungjin Ahn. Sequential neural processes. Advances in Neural Information Processing Systems, 32, 2019.
13

---

[52] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances neural information processing systems, 30, 2017.
[53] Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms, 2012.
[54] Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian pro- cess optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995, 2009.
[55] Flood Sung. Yongxin Yang. Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[56) Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task bayesian optimization. Advances in neural information processing systems, 26, 2013.
[57] Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective models for effective offline model-based optimization. In International Conference on Machine Learning, pages 10358-10368. PMLR, 2021.
[58] Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Bench- marks for data-driven offline model-based optimization. In International Conference on Machine Learning, pages 21658-21677. PMLR, 2022.
[59] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[60] Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey few-shot learning, ACM computing surveys (csur), 53(3):1-34, 2020.
[61] James T Wilson, Riccardo Moriconi, Frank Hutter, and Marc Peter Deisenroth. The reparame- terization trick for acquisition functions. arXiv preprint arXiv: 2017.
[62] Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates. arXiv preprint 1rXiv:2101.00000 2021.
[63] Chenfei Wu, Jian Liang. Lei Ji, Fan Yang. Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre- pre-training for neural visual world creation. In Computer Vision Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XVI, pages 720-736. Springer, 2022.
[64] Yuhuai Wu, Markus N Rabe, Wenda Li, Jimmy Ba, Roger B Grosse, and Christian Szegedy. Lime: Learning inductive bias for primitives of mathematical reasoning. In International Conference Machine Learning, pages 11251-11262. PMLR, 2021.
[65] Sihyun Yu, Sungsoo Ahn, Le Song, and Jinwoo Shin. Roma: Robust model adaptation for offline model-based optimization. Advances in Neural Information Processing Systems, 34: 4619-4631, 2021.
[66] Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv: 10115577, 2016.
14


Reviews:
[{"review_id": "4XTQBbbWE5", "reviewer": "Reviewer_Tttq", "summary": "This paper investigated the problem of few-shot black-box optimization, and presented Synthetically pre-trained Transformer for Optimization  (SynTO). By combining synthetic pretraining with in-context learning to enable few-shot generalization, SynTO demonstrate its superior performance on Design-Bench. ", "questions": "- Can the problem in this paper be directly solved by few shot learning methods? If so, Some experiments may be required to compare the proposed method with the existing few shot learning methods. Else some discussion may be required to explain this issue. \n- It seems that SynTO shares similar structure with BONET. Thus, some discussion is required to explain the performance gain of SynTO w.r.t. BONET.\n- Ablation studies are required to clarify which components of SynTO explain the performance superiority.   \n\n", "limitations": "Yes", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Synthetically pre-trained Transformer for Optimization  (SynTO). \n+ Combining synthetic pretraining with in-context learning to enable few-shot generalization.\n+ Demonstrating superior performance on Design-Bench. \n+ The paper is well written. ", "weaknesses": "- Can the problem in this paper be directly solved by few shot learning methods? If so, Some experiments may be required to compare the proposed method with the existing few shot learning methods. Else some discussion may be required to explain this issue. \n- It seems that SynTO shares similar structure with BONET. Thus, some discussion is required to explain the performance gain of SynTO w.r.t. BONET.\n- Ablation studies are required to clarify which components of SynTO explain the performance superiority.   \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and for the recognition of the strong performance of SynTO and the good presentation of the paper. We answer each of the reviewer's concerns below.\n\n> Can the problem in this paper be directly solved by few shot learning methods?\n\nWhile it is true that SynTO is a few-shot learning method and makes use of only a small labeled dataset, few-shot offline BBO works in a different setting as compared to most of the commonly referenced few-shot learning works. Other few-shot learning works [1, 2, 3] are proposed for classification problems, often in the domain of computer vision and sometimes natural language processing. This is not applicable to few-shot BBO, where a model is required to either predict y which is continuous (forward approaches), or to generate high-dimensional input x (inverse approaches). The most applicable prior approach for this problem is the neural process methods, which were proposed for meta or few-shot regression. In this regard, Table 4 already shows the superior performance of SynTO compared to Transformer Neural Processes (TNPs) [4], the current state-of-the-art neural process model.\n\n[1] Koch, Gregory, Richard Zemel, and Ruslan Salakhutdinov. \"Siamese neural networks for one-shot image recognition.\" ICML deep learning workshop. Vol. 2. No. 1. 2015.\n\n[2] Snell, Jake, Kevin Swersky, and Richard Zemel. \"Prototypical networks for few-shot learning.\" Advances in neural information processing systems 30 (2017).\n\n[3] Sung, Flood, et al. \"Learning to compare: Relation network for few-shot learning.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n\n[4] Nguyen, Tung, and Aditya Grover. \"Transformer neural processes: Uncertainty-aware meta learning via sequence modeling.\" arXiv preprint arXiv:2207.04179 (2022).\n\n> It seems that SynTO shares similar structure with BONET. Thus, some discussion is required to explain the performance gain of SynTO w.r.t. BONET.\n\nWe note that while SynTO and BONET both make use of the transformers architecture, the two methods are very different in many aspects:\n- BONET was proposed for offline BBO where we assume access to thousands of labeled data samples, while SynTO is specifically designed for few-shot BBO where we only have a few hundred labeled data points. \n- BONET is trained in a supervised manner on offline data pairs (x,y), while we pretrain SynTO on synthetic data and adapt to downstream functions using in-context learning.\n- BONET first constructs trajectories of the form (x_1, x_2, …, x_N) in which the x’s are sorted by the regrets, and then models the distribution of these sequences. In contrast, SynTO models the conditional distribution of the target given context p(x_target | y_target, x_context, y_context), where the points do not follow any order as the architecture is permutation invariant to the context.\n- BONET employs an autoregressive transformers architecture, while SynTO does not.\n\nEmpirically, SynTO outperforms BONET on few-shot BBO settings. Only using labeled data for training makes BONET vulnerable to overfitting, while pretraining on a large amount of synthetic data allows better generalization for SynTO.\n\n> Ablation studies are required to clarify which components of SynTO explain the performance superiority.\n\nPretraining on a large and diverse set of synthetic functions is the key to the superior performance of SynTO in the few-shot setting. In the main results, we pretrained SynTO on data generated from GPs with an RBF kernel which we found to work well in practice. We showed the importance of the hyperparameters of the RBF kernel on the performance of SynTO in Appendix C.1. Additionally, we performed an ablation study where we pretrain SynTO on different data distributions, including different GP kernels (GP-Cosine, GP-Linear, GP-Periodic), randomly initialized 1-layer neural networks (Random MLP), and neural network checkpoints trained on the few-shot data (Trained MLP). For each neural network generated from Random MLP, we randomly select one in 6 different initialization methods (uniform, normal, xavier uniform, xavier normal, kaiming uniform, kaiming normal) to generate the network weights. For Trained MLP, we first initialize neural networks with different depths (2, 3, 4, 5, 6), different hidden dimensions (16, 32, 64, 128, 256, 512, 1024), and different initialization methods (uniform, normal, xavier uniform, xavier normal, kaiming uniform, kaiming normal), and train these networks on the few-shot data which will then be used for generating data.\n\nTable 1 and 2 in the attached pdf file show the performance of SynTO on the few-shot random and few-shot poor settings when pretrained with different data distributions. Overall, the model achieves good performance across different data distributions, with GP-RBF being the best in most settings. This ablation study shows the robustness of SynTO to the pretraining data distribution. \n \nIn addition to the pretraining data distribution, we also conducted an ablation study on the architecture of SynTO, in which we replaced the VAE model with a diffusion model (SynTO-Diffusion). We take the diffusion architecture from [5].\n\nTable 3 and 4 in the attached pdf file show that SynTO + VAE outperforms SynTO + Diffusion in all tasks and settings. We hypothesize that SynTO with a too powerful decoder may learn to only model the distribution over x_target and ignore the conditioning variables (x_context, y_context, y_target), which consequently hurts the generalization of the model.\n\n[5] Krishnamoorthy, Siddarth, Satvik Mehul Mashkaria, and Aditya Grover. \"Diffusion Models for Black-Box Optimization.\" arXiv preprint arXiv:2306.07180 (2023)."}, {"review_id": "y3aLYu3M5h", "reviewer": "Reviewer_piUM", "summary": "The paper presents a method for tackling few-shot black-box optimization problems in which the model queries a few hundred data points from the black-box function. The proposed method utilizes synthetic pretraining, where a family of synthetic functions is employed to generate data for in-context learning of a transformer-based model. After pretraining, the model adapts to downstream function using few-shot data.", "questions": "- How sensitive is the proposed approach to the choice of synthetic functions? For instance, if we change the kernel of the GP or use randomly initialized neural networks, how is downstream performance affected?\n- How does the downstream performance vary with the context size? Additionally, does the proposed method perform well when the downstream context size differs significantly from those used during pretraining?", "limitations": "The authors have addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper proposes and addresses the problem of few-shot black-box optimization, a problem with many potential real-world applications.\n- The proposed inverse modeling approach is robust to the quality of synthetic data and allows for gradient-free optimization during testing.\n- As Gaussian Processes (GPs) are used to generate synthetic functions, there are no extra costs associated with data generation.\n- Experimental results show performance comparable to previous state-of-the-art methods, while also demonstrating increased robustness to the quality of few-shot data.", "weaknesses": "- There's no guarantee that real-world downstream functions follow a Gaussian Process with a Radial Basis Function (RBF) kernel. In other words, performance might be sensitive to the similarity between downstream functions and those generated by a GP with an RBF kernel. While the authors mention the universal approximation property, it isn't certain that the pretraining stage covers a sufficient range of functions over a sufficient number of training iterations in practice. For instance, the hyperparameters of the RBF kernel used in the experiments are bounded, and the range was heuristically chosen.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and for the recognition of the significance of the paper and the strong performance of SynTO. We answer each of the reviewer's concerns below.\n\n> There's no guarantee that real-world downstream functions follow a Gaussian Process with a Radial Basis Function (RBF) kernel.\n\nWe agree with the reviewer. Ideally, the pretraining distribution should be both diverse and close to the true function. However, given a very small labeled dataset, it is hard to find such a distribution. In practice, we found that GP-generated data is sufficient for the model to perform well on most tasks, including both discrete and continuous input spaces.  \n\nOne way to increase the similarity between pretraining functions and the downstream function is to make use of the few-shot data for generation, e.g., learn multiple neural networks on the few-shot data and use those networks to generate more data. While this is against our original setting where the few-shot data is only available at test time, we still conducted this experiment to gain more insights into the performance of SynTO. We show below that this works well in practice but not better than GP-generated data. Moreover, this has two drawbacks. First, it requires training a large set of neural networks on the few-shot data before training SynTO.  Second, it does not allow for optimizing other functions in the same domain, as the pretraining functions were trained to mimic a specific function. In other words, a result such as the one in Table 3 would not be possible.\n\n> For instance, if we change the kernel of the GP or use randomly initialized neural networks, how is downstream performance affected?\n\nWe wanted to first point the reviewer to Appendix C.1 which studies the importance of the hyperparameters of the RBF kernel on the performance of SynTO. Additionally, we performed an ablation study where we pretrain SynTO on different data distributions, including different GP kernels (GP-Cosine, GP-Linear, GP-Periodic), randomly initialized 1-layer neural networks (Random MLP), and neural network checkpoints trained on the few-shot data (Trained MLP). For each neural network generated from Random MLP, we randomly select one in 6 different initialization methods (uniform, normal, xavier uniform, xavier normal, kaiming uniform, kaiming normal) to generate the network weights. For Trained MLP, we first initialize neural networks with different depths (2, 3, 4, 5, 6), different hidden dimensions (16, 32, 64, 128, 256, 512, 1024), and different initialization methods (uniform, normal, xavier uniform, xavier normal, kaiming uniform, kaiming normal), and train these networks on the few-shot data which will then be used for generating data.\n\nTables 1 and 2 in the attached file show the performance of SynTO on the few-shot random and few-shot poor settings when pretrained with different data distributions. Overall, the model achieves good performance across different data distributions, with GP-RBF being the best in most settings. This ablation study shows the robustness of SynTO to the pretraining data distribution. \n\n> How does the downstream performance vary with the context size\n\nTables 5 and 6 in the attached pdf file show the performance of SynTO when we vary the number of context points during pretraining on the few-shot random and few-shot poor settings, respectively. Overall, the model is robust to the context size and achieves good performances with a wide range of context sizes. A larger context size tends to perform better in the random setting, but a bit poorer in the poor setting.\n\n> Does the proposed method perform well when the downstream context size differs significantly from those used during pretraining?\n\nTables 3 and 4 show that when the pretraining context size is much smaller (10, 50) or much larger (200, 500) than the downstream context size (~100), SynTO still performs well, even though the performance does degrade slightly in some settings."}, {"review_id": "1hnOPAXaZa", "reviewer": "Reviewer_DuxH", "summary": "The paper tackles black-box-optimization from few-shot examples, by pretraining a transformer model on synthetic proxy tasks using in-context learning, and evaluating with the same procedure but with real data. The synthetic tasks are generated by using 1) real unlabeled data and 2) a synthetic generative process that generates different tasks (i.e. input/targets pairs $x_i$, $y_i$) given the unalbeled data (inputs) for the task of interest. The procedure studied to generate the synthetic tasks is Gaussian processes with a radial basis function kernel. During training, the model receives D input/target value pairs generated by the process, and is tasked to predict the the distribution of the input given a target value ($p(x_y|y_i)$) and the context points using a VAE. At inference time, the model receives the few-shot examples context points, and is tasked to predict the distribution for the current task.\nThe approach results in competitive performance against existing methods, which generally outperforms existing methods. ", "questions": "Despite the paper showing strong evidence that the proposed method is sound and outperforms alternatives, more insights on why GP works would make for a stronger paper. See weakness for questions.", "limitations": "Yes, the authors adequately addressed the limitations", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The method proposed is simple and sound, and incorporates recent progress from transformer-based modelling, such as in-context learning which allows to efficiently adapt the model during inference time without backpropagation.\n\n- Despite its simplicity on how to train the system and use it during inference, it achieves competitive performance on a wide range of benchmarks.\n\n- The experimental section is thorough, giving lots of insights about the method. For example, it gives insights on how the training with GP performs when the (synthetic) tasks during inference are clearly out-of-distribution (Section 4.1), as well as a thorough analysis of alternative design choices such as random vs sorted selection for context/target points, which are shown to be suboptimal.\n\n- The paper proposes pretraining on fully synthetic data, which is easy to construct and has less ethical concerns than pretraining on real data.", "weaknesses": "\nThe main point of the paper is pretarining using synthetically generated data, and more inisights on why this works at all would make for a stronger paper. My main concerns for the paper are regarding this issue:\n-  For example, regarding the following sentence: \nL50-51: \"Our key insight is, if a model learns to perform few-shot learning on a diverse and challenging set of functions, it should be able to adapt quickly to any objective function at test time.\"  Generating tasks that are \"diverse and challenging\" from the unlabelled data does not seem enough for the method to perform well, and the concepts of \"diversity and challenging\" are not well defined in the paper nor in references, neither theoretically nor experimentally. For example, one could generate a totally random process without any type of correlations, which would be both \"challenging and diverse\" and the method would likely not perform well during inference. Making this notions clearer and relating it to the choice of Gaussian processes, either theoretically or through experimental comparisons with pretraining with other types of processes would make the paper more complete.\n\n- More insights on why the selected random process (Gaussian processes) is good, and analysis with other alternatives would make for a stronger paper. For example, although the functions used for synthetic evaluation in 4.1 are much more simple, it would be good to use them as pretraining at least to illustrate this point. \n- If the only requirement is the data being \"diverse and challenging\", can real data from similar domains (or other domains that can be adapted to match the desired data statistics) be used for pretraining? Why training with a fully synthetic process is an advantage?\n- It would be good to perform an analysis of how the statistics of the pretraining data compares to the statistics during inference (e.g. precision/recall as in [1] or similar metrics), to see if the inference task are statistically different or not when compared to the pretraining tasks. This or similar would illustrate if the the diversity of the pretraining data covers all modes of the inference tasks (high recall) and/or whether the diversity is too high compared to the distribution of data for the inference task (low precision). \n\n\n[1] Assessing Generative Models via Precision and Recall. Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, Sylvain Gelly", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and for the recognition of the simplicity, soundness, strong performance of SynTO, and thoroughness of our experiments. We answer each of the reviewer's concerns below.\n\n> Generating tasks that are \"diverse and challenging\" from the unlabelled data does not seem enough for the method to perform well, and the concepts of \"diversity and challenging\" are not well defined in the paper nor in references, neither theoretically nor experimentally. For example, one could generate a totally random process without any type of correlations, which would be both \"challenging and diverse\" and the method would likely not perform well during inference.\n\nThe main idea of synthetic pretraining is to enable the model to generalize to any downstream function given very limited labeled data. Ideally, the pretraining distribution should be representative of the downstream function, however, it is hard to find such a distribution as we only have access to a very small labeled dataset. An alternative solution is pretraining on a diverse set of functions so that the model gets exposed to many different types of functions during pretraining, which facilitates generalization to the downstream function at test time. Diversity can be thought of as having a function distribution that can generate many different kinds of functions such as linear, polynomial, exponential, periodic, etc., and their combination. A Gaussian Process with an RBF kernel is perfectly suited for this purpose since it is a universal approximator to any function [1], which means in theory if we pretrain the model for long enough it will eventually see data from a synthetic function which is close to the downstream function.\n\nWe agree that the model will not perform well when we generate data from a totally random process without any type of correlation. In this case, since x and y are independent, the model cannot extract any signal from the context points (x_context, y_context) to predict x_target. Therefore, the model will ignore the conditioning and only learn to model the marginal distribution p(x), which means it will generate x’s with random returns at test time.\n\nIn practice, choosing a suitable set of hyperparameters for the RBF kernel helps avoid this degenerate case. We showed this empirically in Appendix C.1, in which a too big or too small value of either the variance and the length scale significantly hurt the performance.\n\n[1] Micchelli, Charles A., Yuesheng Xu, and Haizhang Zhang. \"Universal Kernels.\" Journal of Machine Learning Research 7.12 (2006).\n\n> More insights on why the selected random process (Gaussian processes) is good, and analysis with other alternatives would make for a stronger paper.\n\nWe performed an ablation study where we pretrain SynTO on different data distributions, including different GP kernels (GP-Cosine, GP-Linear, GP-Periodic), randomly initialized 1-layer neural networks (Random MLP), and neural network checkpoints trained on the few-shot data (Trained MLP). For each neural network generated from Random MLP, we randomly select one in 6 different initialization methods (uniform, normal, xavier uniform, xavier normal, kaiming uniform, kaiming normal) to generate the network weights. For Trained MLP, we first initialize neural networks with different depths (2, 3, 4, 5, 6), different hidden dimensions (16, 32, 64, 128, 256, 512, 1024), and different initialization methods (uniform, normal, xavier uniform, xavier normal, kaiming uniform, kaiming normal), and train these networks on the few-shot data which will then be used for generating data.\n\nTables 1 and 2 in the attached pdf file show the performance of SynTO on the few-shot random and few-shot poor settings when pretrained with different data distributions. Overall, the model achieves good performance across different data distributions, with GP-RBF being the best in most settings. This ablation study shows that the pretraining data can be generated from other distributions than GPs, as long as they generate a diverse set of synthetic functions for pretraining. This experiment also shows the robustness of SynTO to the pretraining data distribution. \n\n> Can real data from similar domains (or other domains that can be adapted to match the desired data statistics) be used for pretraining? Why training with a fully synthetic process is an advantage?\n\nYes, we can use real data from similar domains for pretraining. However, this makes a stronger assumption because real-world data is hard to collect in most engineering problems, and we need multiple such datasets to pretrain SynTO. Using synthetic data for pretraining removes this assumption and allows generalization to real functions with no extra data collection costs."}, {"review_id": "RtrQepZxg4", "reviewer": "Reviewer_5SUX", "summary": "The authors introduce a novel approach SynTO to address a challenging setting - few-shot black-box optimization. Specifically, SynTO adopts a pretraining-adaptation pipeline. SynTO can be pretrained using synthetic functions and then adapt to downstream tasks via an in-context learning manner. Comprehensive experiments demonstrate the effectiveness of the proposed method in multiple settings.", "questions": "1.\tCan you explain why SynTO degrades more on Max scores under the poorest setting?\n\n2.\tIn Table 4, can the forward and inverse modeling methods be directly compared regarding their qualitative results, considering the two approaches generate different kinds of outputs?\n", "limitations": "Yes. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed few-shot BBO is more real-world applicable and generalizable to multiple optimization tasks. Generally speaking, the paper is technically solid.\n\n2.\tIt is interesting to pretrain models with synthetic data from families of other functions. \n\n3.\tThe paper writing is clear, and the presentation is satisfying.\n", "weaknesses": "1.\tSynTO assumes the access to large amounts of unlabeled data, which may cause unfair comparisons as the other methods are trained only with a few labeled data.\n2.\tIn Table 1, albeit with the extra unlabeled data, SynTO seems to perform inferior in some settings.\n3.\tIt would be better to give the efficiency of SynTO compared with previous methods, since it further introduces an additional pretraining step.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and for the recognition of the technical contributions of SynTO and the presentation of our paper. We answer each of the reviewer's concerns below.\n\n> SynTO assumes the access to large amounts of unlabeled data, which may cause unfair comparisons as the other methods are trained only with a few labeled data.\n\nSynTO can make use of the rich unlabeled data for pretraining, while the other methods cannot. We believe this is a strength of SynTO compared to other methods. This assumption holds in many real-world applications where we know the possible designs of an engineering problem but do not know how good they are. For example, in molecular optimization, we have databases of millions of molecules [1, 2, 3],  but only the properties of a handful are known [4, 5, 6].\n\n[1] Sunghwan Kim, Paul A Thiessen, Evan E Bolton, Jie Chen, Gang Fu, Asta Gindulyte, Lianyi Han, Jane He, Siqian He, Benjamin A Shoemaker, et al. Pubchem substance and compound databases. Nucleic acids research, 44(D1):D1202–D1213, 2016.\n\n[2] Lorenz C Blum and Jean-Louis Reymond. 970 million druglike small molecules for virtual screening in the chemical universe database gdb-13. Journal of the American Chemical Society, 131(25):8732–8733, 2009.\n\n[3] Lars Ruddigkeit, Ruud Van Deursen, Lorenz C Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. Journal of chemical information and modeling, 52(11):2864–2875, 2012.\n\n[4] John J Irwin and Brian K Shoichet. Zinc- a free database of commercially available compounds for virtual screening. Journal of chemical information and modeling, 45(1):177–182, 2005.\n\n[5] Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1–7, 2014.\n\n[6] Anna Gaulton, Anne Hersey, Michał Nowotka, A Patricia Bento, Jon Chambers, David Mendez, Prudence Mutowo, Francis Atkinson, Louisa J Bellis, Elena Cibrián-Uhalte, et al. The chembl database in 2017. Nucleic acids research, 45(D1):D945–D954, 2017.\n\n> In Table 1, albeit with the extra unlabeled data, SynTO seems to perform inferior in some settings.\n\nWe note that SynTO is not inferior compared to the baselines. In Table 1, in all tasks under all metrics, SynTO is either the best or the second-best method, and the mean rank and the mean score of SynTO are also superior to the baselines.\n\n> It would be better to give the efficiency of SynTO compared with previous methods, since it further introduces an additional pretraining step.\n\nSynTO has a pretraining step but performs optimization for downstream functions via in-context learning without any gradient updates, which means there is only 1 training stage, similar to the other methods. The only difference is SynTO trains on synthetic data, while the other methods train on real data.\n\n> Can you explain why SynTO degrades more on Max scores under the poorest setting?\n\nOverall, SynTO is still the best method in terms of the Max score metric with respect to mean score and mean rank. Moreover, we believe mean and median scores are better indicators of the actual performance, since they take into account the quality of all proposals instead of only the best proposal. In discrete tasks such as tf8 and tf10, a method can propose a very good point by chance without actually performing well in expectation.\n\n> In Table 4, can the forward and inverse modeling methods be directly compared regarding their qualitative results, considering the two approaches generate different kinds of outputs?\n\nBoth the forward and inverse modeling approaches are optimizing f and hence, their final outputs used for evaluation in Table 4 are the same. That is, they both propose x's for black-box evaluation. The difference is while the inverse model generates those proposals directly conditioning on the desired y, the forward approach first learns a surrogate for f and then performs gradient ascent on the x's to propose better points.\n"}]

Author rebuttal:
We conducted additional experiments to gain more insights into the performance of SynTO. The experiments are:
- (Table 1 and 2) Pretraining SynTO on different data distributions, including different GP kernels (GP-Cosine, GP-Linear, GP-Periodic), randomly initialized 1-layer neural networks (Random MLP), and neural network checkpoints trained on the few-shot data (Trained MLP).
-  (Table 3 and 4) SynTO with different decoder architectures, i.e., diffusion vs VAE.
-  (Table 5 and 6) Pretraining SynTO with different context sizes.

In all experiments, we evaluate the performance of SynTO on all 4 tasks Dkitty, Ant, TF8, and TF10. For each setting we train 3 models with 3 seeds and average the results.

Meta-review: Reviewers agree on acceptance based on simple ideas, practical significance and experimental evaluation. The authors are encouraged to include discussions and experiments from the rebuttal in the final version.
Decision: Accept (poster)