Title: ExPT: Synthetic Pretraining for Few-Shot Experimental Design
Venue: NeurIPS 2023
Accepted: yes
Keywords: experimental design

few-shot

black-box optimization

synthetic pretraining

in-context learning

transformer
Abstract: Experimental design is a fundamental problem in many science and engineering fields. In this problem, sample efficiency is crucial due to the time, money, and safety costs of real-world design evaluations. Existing approaches either rely on active data collection or access to large, labeled datasets of past experiments, making them impractical in many real-world scenarios. In this work, we address the more challenging yet realistic setting of few-shot experimental design, where only a few labeled data points of input designs and their corresponding values are available. We approach this problem as a conditional generation task, where a model conditions on a few labeled examples and the desired output to generate an optimal input design. To this end, we introduce Experiment Pretrained Transformers (ExPT), a foundation model for few-shot experimental design that employs a novel combination of synthetic pretraining with in-context learning. In ExPT, we only assume knowledge of a finite collection of unlabelled data points from the input domain and pretrain a transformer neural network to optimize diverse synthetic functions defined over this domain. Unsupervised pretraining allows ExPT to adapt to any design task at test time in an in-context fashion by conditioning on a few labeled data points from the target task and generating the candidate optima. We evaluate ExPT on few-shot experimental design in challenging domains and demonstrate its superior generality and performance compared to existing methods. The source code is available at https://github.com/tung-nd/ExPT.git.

ExPT: Synthetic Pretraining for Few-Shot Experimental Design

Abstract

Experimental design for optimizing black-box functions is a fundamental problem in many science and engineering fields. In this problem, sample efficiency is crucial due to the time, money, and safety costs of real-world design evaluations. Existing approaches either rely on active data collection or access to large, labeled datasets of past experiments, making them impractical in many real-world scenarios. In this work, we address the more challenging yet realistic setting of few-shot experimental design, where only a few labeled data points of input designs and their corresponding values are available. We introduce Ex periment P retrained T ransformers (ExPT), a foundation model for few-shot experimental design that combines unsupervised learning and in-context pretraining. In ExPT, we only assume knowledge of a finite collection of unlabelled data points from the input domain and pretrain a transformer neural network to optimize diverse synthetic functions defined over this domain. Unsupervised pretraining allows ExPT to adapt to any design task at test time in an in-context fashion by conditioning on a few labeled data points from the target task and generating the candidate optima. We evaluate ExPT on few-shot experimental design in challenging domains and demonstrate its superior generality and performance compared to existing methods. The source code is available at https://github.com/tung-nd/ExPT.git .

1 Introduction

The design of experiments to optimize downstream target objectives is a ubiquitous challenge across many science and engineering domains, including materials discovery [27], protein engineering [7, 49, 2], molecular [22] design, mechanical design [4, 38], and neural architecture optimization [66]. The key criterion of interest in experimental design (ED) is sample-efficiency, as the target objectives are often black-box functions and evaluating these objectives for any candidate design often involves expensive real-world experiments. A standard class of approaches learn a surrogate to approximate the target objective and actively improve the approximation quality through online experiments [53]. However, online data acquisition may be infeasible in the real world due to high costs, time constraints, or safety concerns. As an alternate, recent works have proposed offline ED [7, 58, 36, 57, 12, 34], wherein a model learns to perform optimization from a fixed dataset of past experiments. While this is more practical than the online setting, current offline methods and benchmarks assume access to large experimental datasets containing thousands of data points, which are hard or even impossible to obtain in high-stake and emerging science problems. Even when these datasets exist, the past experiments might be of very poor quality resulting in poor surrogate learning and optimization.

In this paper, we aim to overcome these limitations for hyper-efficient experimental design that does not require large experimental datasets. To this end, we introduce few-shot experimental design, a more challenging setting that better resembles real-world scenarios. We describe few-shot ED as a two-phased pretraining-adaptation paradigm. In the pretraining phase, we only assume access to

X

Molecules

DNA

Robotics

ExPT

Synthetic

Pretraining

Real data

{

(

x

i

, y

i

)

}

n

i

=1

ExPT

y

/star

x

/star

In-context

Adaptation

unlabeled data, i.e., input designs without associated function values. During the adaptation phase, we have access to a few labeled examples of past experiments to adapt the model to the downstream task. This setup offers several advantages. First, it alleviates the requirement for costly annotated data and relies mainly on unlabeled inputs that are easily accessible. Second, unsupervised pretraining enables us to utilize the same pretrained backbone for adapting to multiple downstream optimization tasks within the same domain. For example, in molecule design, one may want to optimize for multiple properties, including drug-likeness, synthesizability, or similarity to target molecules [8, 17].

The key question in this setup is how to make use of the unlabeled data to facilitate efficient generalization to downstream tasks during optimization. Our intuition here is that, while the objective function is unknown, we can use the unlabeled inputs to generate pretraining data from other synthetic functions. If a model can few-shot learn from a diverse and challenging set of functions, it should be able to generalize quickly to any target objective during the adaptation phase, in line with recent foundation models for language [9] and vision [3]. This insight gives rise to our idea of synthetic pretraining , wherein we pretrain the model on data generated from a rich family of synthetic functions that operate on the same domain as the target task. Specifically, for each function drawn from this family, we sample a set of points by using the unlabeled data as inputs. We divide these points into a small context set and a target set, and train the model via in-context learning to perform conditional generation of the target input x given the context points and the target value y . A model that works well on this task should be able to efficiently capture the structures of the underlying function, i.e., how different regions of the input space influence the function value, from a small context set. By explicitly training the model to perform this task on a diverse set of functions, the model can generalize efficiently to downstream functions during adaptation requiring only limited supervision. After pretraining, we can perform optimization by conditioning the model on a few labeled examples from the downstream task and generating an input that achieves the optimum y /star .

Inspired by recent advances in few-shot learning in language [9, 42] and other domains [44, 45, 19, 37], we instantiate a novel foundation model with a transformer-based architecture [59], which we call Ex periment P retrained T ransformers (ExPT). ExPT is an encoder-decoder architecture, in which the encoder is a transformer [59] network that encodes the context points and the target value, and the decoder is a VAE [32] model that predicts the high-dimensional target input. The transformer encoder allows ExPT to perform few-shot generation and optimization purely through in-context learning in a gradient-free fashion. We compare the performance of ExPT and various baselines on 2 few-shot settings created from Design-Bench [58], a standard database benchmark for ED. The two settings allow us to examine how different methods perform with respect to different quantities and qualities of few-shot data. In both these settings, results show that ExPT achieves the highest average score and the highest average ranking with respect to median performance, mean performance, and best-achieved performance. Especially in the more challenging setting, ExPT outperforms the second-best method by 70% in terms of the mean performance. Additionally, we explore the potential of using the same pretrained ExPT for multiple objectives, and conduct extensive ablation studies to validate the effectiveness of our design choices for synthetic data generation and ExPT architecture.

2 Experiment Pretrained Transformers

2.1 Problem setup

Let f : X → R be a black-box function that operates on a d -dimensional domain X ⊆ R d . In experimental design (ED), the goal is to find the input x /star that maximizes f :

Typically, f is a high-dimensional and complex function that often involves expensive physical experiments. Existing approaches either assume the ability to actively query f to collect data [53] or access to a large dataset of past experiments [58]. Both assumptions are too strong in many real-world applications where data collection is hard or even impossible [11]. Therefore, we propose few-shot ED , a more challenging yet realistic setting to overcome these limitations. In few-shot ED, the goal is to optimize any objective function in the domain X given only a handful of examples. We approach this problem with a pretraining-adaptation pipeline. In the pretraining phase, we assume access to an unlabeled dataset D unlabeled = { x i } |D unlabeled | i =1 from the optimization domain X ⊆ R d . We note that D unlabeled only contains potential design inputs without their corresponding scores, for example, potential molecules in molecule optimization or different combinations of hyperparameters in neural architecture search. This means the objective function f is unspecified during pretraining.

During the adaptation phase, one can use the pretrained model to optimize any objective function f in the same domain X . We now have access to a few-shot labeled dataset that the model can use to adapt to the downstream function D few-shot = { ( x 1 , y 1 ) , . . . , ( x n , y n ) } , in which y i = f ( x i ) and n = |D few-shot | /lessmuch |D unlabeled | . After adaptation, we evaluate a few-shot optimization method by allowing it to propose Q input x ′ s and query their scores using the black-box function f , where Q is often called the optimization budget [58, 36, 57, 12, 34]. The performance of a black-box optimizer is then measured by computing the max, median, and mean of the Q evaluations, This setup provides two key benefits. First, it resembles many real-world scenarios, where the potential design inputs are cheap and easy to obtain while their target function values are expensive to evaluate. For example, in molecular optimization, we have databases of millions of molecules [30, 5, 48] but only the properties of a handful are known [28, 47, 23]. Second, unsupervised pretraining allows us to train a general backbone that we can adapt to multiple optimization tasks in the same domain.

2.2 Synthetic Pretraining and Inverse Modeling for Scalable Experimental Design

Intuitively, the adaptation phase in §2.1 resembles a few-shot learning problem, in which a model is tasked to produce the optimal input x /star by conditioning on a few labeled examples in D few-shot. To perform well in this task, a model has to efficiently capture the structure of a high-dimension function f , i.e., what regions of the function lead to higher values and vice versa, from very few examples in D few-shot. Given this perspective, the question now is how to make use of the unlabeled dataset D unlabeled to pretrain a model that achieves such efficient generalization to the objective function f . Our key insight is, if a model learns to perform in-context learning on a diverse and challenging set of functions, it should be able to adapt quickly to any objective function at test time. While the function values are unknown during pretraining, we can use the unlabeled inputs x ′ s to generate pretraining data from other functions. This gives rise to our idea of synthetic pretraining , wherein we pretrain the model to perform few-shot learning on a family of synthetic functions ˜ F that operate on the same input domain X of the objective f . We discuss in detail our mechanism for synthetic data generation in Section 2.3. For each function ˜ f generated from ˜ F , we sample a set of function evaluations { ( x i , y i ) } N i =1 that we divide into a small context set { ( x i , y i ) } i m =1 and a target set { ( x j , y j ) } N j = m +1 . We train the model to predict the target points conditioning on the context set.

There are two different approaches to pretraining a model on this synthetic data. The first possible approach is forward modeling, where the model is trained to predict the target outputs y m +1: N given the context points and the target inputs x m +1: N . This is similar to the approach followed by TNPs [45], a model recently proposed in the context of meta-learning. During adaptation, we can condition the model on the labeled examples in D few-shot and perform gradient ascent updates to improve an existing design input x t . However, as commonly observed in previous works [58, 36, 57], this approach is susceptible to producing highly suboptimal inputs. This is because performing gradient ascent with respect to an imperfect forward model may result in points that have high values

ExPT

D

y

1

x

1

Adaptation

y

2

x

2

y

n

x

n

few-shot examples

y

/star

optimal value

˜

F

y

1

x

1

Synthetic

Pretraining

y

m

x

m

context points

y

m

+1

y

N

target points

Transformer Encoder

h

m

+1

h

N

Decoder

Decoder

ˆ

x

m

+1

ˆ

x

N

under the model but are poor when evaluated using the real function. Instead, we propose to perform inverse modeling , where the model learns to predict the inputs x m +1: N given the output values y m +1: N and the context points. As the model learns to directly generate input x ′ s , it is less vulnerable to the aforementioned problem. Another advantage of inverse modeling is after pretraining, we can simply condition on D few-shot and the optimal value y /star to generate the candidate optima. Our loss function for pretraining the model is:

where we assume the target points are independent given the context set and the target output. Figure 2 illustrates the proposed pretraining and adaptation pipeline. Typically, we use a small context size m during pretraining to resemble the test scenario.

After pretraining, ExPT can adapt to any objective f in the domain in a gradient-free fashion. Samples in the few-shot dataset D few-shot become the context points and the model conditions on only one target y /star , which is the optimal value of f , to generate candidate optima. Note that we only assume the knowledge of y /star and not x /star . This assumption is common in many prior works [34, 46, 13, 14]. In practice, y /star might be known based on domain knowledge. For example, in molecule design, there are physical limits on the value of certain properties such as relaxed energy, in robot applications, the optimal performance can be computed from the cost function, and in neural architecture search, we know the theoretical limits on the highest possible accuracy for classifiers.

Next, we present the details of synthetic data generation and our proposed model architecture, the two components that constitute our proposed foundation model, which we refer to as Ex periment P retrained T ransformers (ExPT).

2.3 Data generation

We need a family of functions to generate synthetic data for pretraining ExPT. A good family of functions should be easy to sample from and should be capable of producing diverse functions. Many possible candidates exist for synthetic function families, such as Gaussian Processes (GPs), randomly constructed Gaussian Mixture Models, or randomly initialized or pretrained neural networks. Among these candidates, we choose to generate synthetic data from Gaussian Processes with an RBF kernel. This is for several reasons. First, they are a natural choice as they represent distributions over functions. Second, it is easy and cheap to sample data from prior GPs. And third, a GP with an RBF

kernel is a universal approximator to any function [41]. Specifically, ˜ f is sampled as follows,

in which σ and /lscript are the two hyperparameters of the RBF kernel. The variance σ scales the magnitudes of the covariance matrix. A larger variance results in a wider range of function values, while a smaller variance restricts the function values to a narrower range. On the other hand, the length scale /lscript determines how strongly the covariance matrix varies with respect to the distance between x and x ′ . A very small length scale /lscript means the kernel is sensitive to the distance between x and x ′ , leading to sharp transitions between neighboring points and a lack of smoothness in the functions. In contrast, if /lscript is too large, the covariance between points will be similar for both near and distant points, leading to function values that are very similar. In other words, too large a length scale reduces the diversity of the synthetic functions. In practice, we randomize both σ and /lscript to increase the diversity of the pretraining data. Appendix C.1 demonstrates the empirical importance of these hyperparameters.

2.4 Model architecture

To optimize the loss function in Equation (2), we need a model architecture that can condition on a few examples drawn from an underlying function to make predictions for other points. This resembles the idea of in-context learning that has proven very successful in language [9, 42] and other domains [44, 45, 19, 37]. The key to success in these works is a transformer architecture that performs in-context learning efficiently via the attention mechanism [59]. Inspired by this, we instantiate ExPT with a transformer-based architecture. Figure 2 illustrates the ExPT overall architecture. Specifically, ExPT employs a transformer encoder that encodes the context points { ( x i , y i ) } i m =1 and the target inputs y m +1: N , and outputs hidden vectors h m +1: N . To inform the model that x i and y i are the input and the corresponding output from ˜ f , we concatenate them to form a token. This results in the sequence { ( y 1 , x 1 ) , . . . , ( y m , x m ) , y m +1 , . . . , y N } . We then embed these tokens using two 1-layer MLP networks, one for the pairs and one for the target inputs, before feeding the sequence to the transformer layers. We implement a masking mechanism that prevents the context points from attending the target points, as they do not contain information about the underlying function ˜ f .

Each hidden vector h i output by the transformer encoder encompasses the information of the context points and the target input y i . Therefore, given h i , the conditional probability p θ ( x i | x 1: m , y 1: m , y i ) reduces to p θ ( x i | h i ) . As x i is high-dimensional, we can utilize existing generative models to model the conditional distribution p ( x i | h i ) . In this work, we train a conditional V AE model [32] alongside the transformer encoder because of its training stability, light hyperparameter tuning, and good empirical performance. For discrete tasks, we follow the same procedure as Trabucco et al. [58] that emulates logit values by interpolating between a uniform distribution and the one hot values. We train the entire model by maximizing the lower bound of the conditional likelihood log p ( x i | h i ) :

in which q φ ( z | x i , h i ) is the encoder of the conditional V AE and p ( z ) is a standard Gaussian prior.

3 Experiments

3.1 Synthetic experiments

We first evaluate the performance of ExPT in a synthetic experiment, where we train ExPT on data generated from Gaussian Processes (GPs) with an RBF kernel and test the model on four out-of-distribution functions drawn from four different kernels: Matern, Linear, Cosine, and Periodic. Figure 3 shows the performance of ExPT on four test functions through the course of training. The model performs well on all four functions, achieving scores that are much higher than the max value in the few-shot dataset, and approaching the true optimal value, even for kernels that are significantly different from RBF like Cosine and Periodic. Moreover, the performance improves consistently as we pretrain, showing that pretraining facilitates generalization to out-of-distribution functions with very few examples. See Appendix A for a detailed setup of this experiment.

3.2 Design-Bench experiments

Tasks We consider 4 tasks from Design-Bench 1 [58]. D'Kitty and Ant are continuous tasks with input dimensions of 56 and 60 , respectively. In D'kitty and Ant, the goal is to optimize the morphological structure of two simulated robots, Ant [6] to run as fast as possible, and D'kitty [1] to reach a fixed target location. TF Bind 8 and TF Bind 10 are two discrete tasks, where the goal is to find the length8 and length10 DNA sequence that has a maximum binding affinity with the SIX6_REF_R1 transcription factor. The design space in these two tasks consists of sequences of one of four categorical variables, corresponding to four types of nucleotide. For each task, Design-Bench provides a public dataset, a larger hidden dataset which is used to normalize the scores, and an oracle. We have an exact oracle to evaluate the proposed designs in all 4 tasks we consider.

Few-shot settings We create 2 few-shot settings from the above tasks, which we call random and poorest . In random , we randomly subsample 1% of data points in the public set of each task as the few-shot dataset D few-shot. The poorest setting is more challenging, where we use 1% of the data points which have the lowest scores. The two settings examine how sensitive different methods are to the quantity and quality of the data. In both settings, we use x ′ s in the public dataset as D unlabeled .

ExPT details For each domain, we pretrain ExPT for 10 , 000 iterations with 128 synthetic functions in each iteration, corresponding to a total number of 1 , 280 , 000 synthetic functions. For each function, we randomly sample 228 input x ′ s from the unlabeled dataset D unlabeled and generate the values y ′ s from a Gaussian Process with an RBF kernel. To increase the diversity of synthetic data, we randomize the two hyperparameters, length scale /lscript ∼ U [5 . 0 , 10 . 0] and function scale σ ∼ U [1 . 0 , 1 . 0] , when generating each function. Additionally, we add Gaussian noises /epsilon1 ∼ N (0 , 0 . 1) to each input x sampled from D unlabeled to enlarge the pretraining inputs. For each generated function, we use 100 points as context points and the remaining 128 as target points, and train the model to optimize (2). During the adaptation phase, we condition the pretrained ExPT model on the labeled few-shot dataset D few-shot and the target function value y /star to generate designs x ′ s .

Baselines We compare ExPT with BayesOpt (GP-qEI) [53], a canonical ED method, and MINs [36], COMs [57], BDI[12], and BONET [34], four recent deep learning models that have achieved state-of-the-art performance in the offline setting. To adapt GP-qEI to the few-shot setting, we use a feedforward network trained on few-shot data to serve as an oracle, a Gaussian Process to quantify uncertainty, and the quasi-Expected Improvement [61] algorithm for the acquisition function. For the deep learning baselines, we train their models on the few-shot dataset D few-shot using the hyperparameters reported in their original papers.

Evaluation For each considered method, we allow an optimization budget Q = 256 . We report the median score, the max score, and the mean score among the 256 proposed inputs. Following previous works, we normalize the score to [0 , 1] by using the minimum and maximum function values from a large hidden dataset y norm = y -y min y max -y min . We report the mean and standard deviation of the score across 3 independent runs for each method.

Results Table 1 shows the performance of different methods in the random setting. Most methods perform well in the random setting, where ExPT achieves the highest average score and the best average rank across all 3 performance metrics. For each of the tasks and metrics considered, ExPT is either the best or second-best performing method. Notably, in Ant, ExPT significantly outperforms the best baseline by 18% , 9% , and 10% with respect to the median, max, and mean performance,

respectively. Only ExPT and BONET achieve a meaningful performance in Ant when considering the mean score. BONET is also the overall second-best method in this setting.

Table 2 shows the superior performance of ExPT in few-shot poorest , the more challenging setting. ExPT achieves the highest score in 10 / 12 individual tasks and metrics, and also achieves the highest score and the best rank across tasks on average. Notably, in terms of the mean score, ExPT beats the best baseline by a large margin, achieving an improvement of 40% , 176% , and 18% on D'Kitty, Ant, and TF Bind 8, and 70% on average. The performance of most baselines drops significantly from the random to the poorest setting, including BONET, the second-best method in the random setting. This was also previously observed in the BONET paper [34]. Interestingly, the performance of ExPT, MINs, and GP-qEI is not affected much by the quality of the few-shot data, and even improves in certain metrics. We hypothesize that even though the dataset is of lower quality, it may contain specific anti-correlation patterns about the problem that the model can exploit.

Pretraining analysis In addition to the absolute performance, we investigate the performance of ExPT on downstream tasks through the course of pretraining. Figure 4 shows that the performance of ExPT in most tasks improves consistently as the number of pretraining steps increases. This shows that synthetic pretraining on diverse functions facilitates the generalization to complex real-world functions. In Ant, the performance slightly drops between 4000 and 10000 iterations. Therefore, we can further improve ExPT if we have a way to stop pretraining at a point that likely leads to the best

performance in downstream tasks. However, in practice, we do not have the luxury of testing on real functions during pretraining. Alternatively, we could perform validation and early stopping on a set of held-out, out-of-distribution synthetic functions. We leave this to future work.

3.2.1 ExPT for few-shot optimization of multiple objectives

As we mention in Section 2, one advantage of unsupervised pretraining is the ability to optimize for multiple objectives during the adaptation phase. In this section, we show that the same pretrained ExPT model is capable of optimizing different objectives in D'Kitty and Ant domains. We create two variants of the original D'Kitty, namely D'Kitty-45 and D'Kitty-60, whose objectives are to navigate the robot to goals that are 45 ◦ and 60 ◦ away from the original goal, respectively. For Ant, we create Antv y where the goal is to run as fast as possible in the vertical Y direction (as opposed to horizontal X direction in Ant) direction, and Ant-Energy, where the goal is to preserve energy. We detail how to construct these tasks in Appendix C. We use the same pretrained models for all these tasks. During adaptation, the model conditions on the D few-shot and y /star for each task for optimization.

We evaluate ExPT on these tasks in the poorest setting. Table 3 shows that ExPT performs well on all tasks, where the median and mean scores are better than the best value in D few-shot, and the max score is close to 1 . For the Ant, Antv y , and Ant-Energy tasks, we visualize the behavior of the optimal designs that are discovered at https://imgur.com/a/zpgI4YL . When subject to the same policy-network (optimizing for horizontal X speed), the robots optimized for different objectives behave differently; the optimal Ant is capable of leaping forward to move quickly in X ; Antv y is able to jump up to maximize speed in Y ; Ant-Energy is capable of 'sitting down' to conserve energy.

3.2.2 ExPT for closed-loop optimization

ExPT can adapt to any objective function purely through in-context learning. This means that the model can refine its understanding of the underlying objective function given more data points in a very efficient manner. In this section, we explore an alternative optimization scheme for ExPT, namely sequential sampling , which explicitly utilizes the in-context learning ability of the model. Specifically, instead of producing Q = 256 inputs simultaneously, we sample one by one sequentially. That is, we condition the model on D few-shot and y /star to sample the first point, evaluate the point using the black-box function, and add the point together with its score to the context set. We repeat this process for 256 times. This process is often referred to as closed-loop optimization in ED.

Table 4 shows that ExPT with sequential sampling performs better than simultaneous sampling on D'Kitty and Ant in both random and poor settings. Especially on Ant in the poorest setting, ExPTSequential achieves improvements of 20% and 19% over ExPT in terms of the median and mean performance, respectively. Intuitively, as we add more data points to the context set, ExPT-Sequential is able to updates its understanding of the structure of the objective function, consequently leading to improved performance.

3.2.3 Forward modeling versus Inverse modeling

As we mentioned in Section 2.2, two possible approaches exist to pretrain ExPT on synthetic data. We take the inverse modeling approach for ExPT throughout the paper, as we train ExPT to directly produce design inputs x ′ s . In this section, we empirically validate our design choices by comparing ExPT with TNP-ED, its forward counterpart. TNP-ED's architecture is similar to ExPT's in Figure 2, except that the target points now contain x m +1: N instead of y m +1: N , the decoder is a 1-layer MLP, and the predicted outputs are ˆ y m +1: N . We call this model TNP-ED because a model named TNP [45] with a similar architecture was previously proposed in the context of meta-learning. We pretrain TNP-ED using a simple mean-squared error loss L = ∑ N i = m +1 (ˆ y i -y i ) 2 . After pretraining, we condition TNP-ED on D few-shot and the best inputs in this dataset, and perform gradient ascent with respect to these inputs to obtain better points.

Table 5 compares the performance of ExPT and TNP-ED on D'Kitty and Ant with respect to the median score and mean score. ExPT achieves significantly better performance in all metrics, especially in the poorest setting. This is because forward models suffer from poor out-of-distribution generalization, and performing gradient ascent on this model may result in points that have high values under the model but are very poor when evaluated using the true functions. This validates our inverse modeling approach.

4 Related work

Online ED The majority of existing approaches solve ED in an active setting, where the model is allowed to query the black-box function to collect more data. Many of these works are based on Bayesian Optimization [39, 46, 50, 53, 56], which typically employs a surrogate model to the black-box function and an acquisition function. The surrogate model is often a predictive model that can quantify uncertainty, such as Gaussian Processes [54], Neural Processes [20, 21, 26, 29, 51, 45], or Bayesian Neural Networks [24]. The acquisition function uses the uncertainty output by the surrogate model to trade off between exploration and exploitation for querying new points.

Offline ED Recent works have proposed to solve ED by learning from a fixed set of ( x, y ) pairs to bypass active data collection [58, 36, 57, 12, 34, 16, 7, 15, 65]. The Design-Bench benchmark [58] consists of several such tasks in the physical sciences and robotics and is used by many recent

works in offline ED. MINs [36] and BONET [34] perform optimization by generating designs x via conditioning on a high score value y . MINs uses a GANs [25] model on ( x, y ) pairs and BONET casts offline ED as a sequence modeling problem. COMs [57] formulates a conservative objective function that penalizes high-scoring poor designs and uses it to train a surrogate forward model which is then optimized using gradient ascent. BDI [12] uses a bidirectional model consisting of a forward and backward models that learn mappings from the dataset to high-scoring designs and vice versa. In contrast to these works, we propose ExPT in the few-shot ED setting, where the model is given access to only the x ′ s during pretraining, and a handful of labeled examples for adaptation.

Synthetic Pretraining In the absence of vast amounts of labeled data, pretraining on synthetic data is an effective method for achieving significant gains in model performance. Prior works in this direction construct synthetic tasks which improve performance on diverse downstream tasks such as mathematical reasoning [64], text summarization [33], and perception tasks in vision [43]. Each synthetic task produces a dataset of labeled ( x, y ) values that can be used to train a model as usual for various objectives. Often, pre-training in this manner produces better results than simply pre-training on another real dataset. In this work, we demonstrate that pretraining on synthetic data generated from GPs can achieve significant generalization to downstream functions, leading to state-of-the-art performance on challenging few-shot optimization problems.

Few-shot learning Few-shot learning is a common paradigm in deep learning, where the model is pretrained on large amounts of data in an unsupervised manner. At test time, the model is given only a few examples from a downstream task and is expected to generalize [60]. This technique has found applications in text-generation (GPT-x) [9], image classification [55, 52], graph neural networks [18], text to visual-data generation [63], and neural architecture search [62] [10]. ExPT is capable of performing few-shot learning for black-box optimization in a variety of domains. Moreover, ExPT is pretrained on synthetically generated data with no prior knowledge of the downstream objective.

5 Conclusion

Inspired by real-world scenarios, this work introduces and studies the few-shot experimental design setting, where we aim to optimize a black-fox function given only a few examples. This setting is ubiquitous in many real-world applications, where experimental data collection is very expensive but we have access to unlabelled designs. We then propose ExPT, a foundation model style framework for few-shot experimental design. ExPT operates in two phases involving pretraining and finetuning. ExPT is pretrained on a rich family of synthetic functions using unlabeled data and can adapt to downstream functions with only a handful of data points via in-context learning. Empirically, ExPT outperforms all the existing methods by a large margin on all considered settings, especially improving over the second-best baseline by 70% in the more challenging setting.

Limitations and Future work In this work, we assume we have access to a larger unlabeled dataset for pretraining and the knowledge of the optimal value for optimization. While these assumptions are true in many applications and have been used widely in previous works, we would like to relax these assumptions in future work to improve further the applicability of the model. One more potential direction is to finetune the pretrained ExPT model on downstream data to further improve performance. Finally, we currently pretrain ExPT for each domain separately. We are interested in exploring if pretraining a big model that works for all domains is possible and if that helps improve performance in each individual domain.

References

Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. Robel: Robotics benchmarks for learning with low-cost robots. In Conference on robot learning , pages 1300-1313. PMLR, 2020.

Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. 2020.

Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems , 35: 25005-25017, 2022.

Felix Berkenkamp, Angela P Schoellig, and Andreas Krause. Safe controller optimization for quadrotors with gaussian processes. In 2016 IEEE international conference on robotics and automation (ICRA) , pages 491-496. IEEE, 2016.

Lorenz C Blum and Jean-Louis Reymond. 970 million druglike small molecules for virtual screening in the chemical universe database gdb-13. Journal of the American Chemical Society , 131(25):8732-8733, 2009.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In International conference on machine learning , pages 773-782. PMLR, 2019.

Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling , 59(3):1096-1108, 2019.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.

Kaidi Cao, Jiaxuan You, Jiaju Liu, and Jure Leskovec. Autotransfer: AutoML with knowledge transfer - an application to graph neural networks. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=y81ppNf_vg .

Ian Char, Youngseog Chung, Willie Neiswanger, Kirthevasan Kandasamy, Andrew Oakleigh Nelson, Mark D Boyer, Egemen Kolemen, and Jeff Schneider. Offline contextual bayesian optimization for nuclear fusion. In 33rd Conference on Neural Information Processing Systems (NeurIPS) , pages 1-12, 2020.

Can Chen, Yingxueff Zhang, Jie Fu, Xue Steve Liu, and Mark Coates. Bidirectional learning for offline infinite-width model-based optimization. Advances in Neural Information Processing Systems , 35:29454-29467, 2022.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems , 34:15084-15097, 2021.

Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.10751 , 2021.

Clara Fannjiang and Jennifer Listgarten. Autofocused oracles for model-based design. Advances in Neural Information Processing Systems , 33:12945-12956, 2020.

Justin Fu and Sergey Levine. Offline model-based optimization via normalized maximum likelihood estimation. arXiv preprint arXiv:2102.07970 , 2021.

Wenhao Gao, Tianfan Fu, Jimeng Sun, and Connor Coley. Sample efficiency matters: a benchmark for practical molecular optimization. Advances in Neural Information Processing Systems , 35:21342-21357, 2022.

Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks, 2018.

Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems , 35:30583-30598, 2022.

Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International conference on machine learning , pages 1704-1713. PMLR, 2018.

Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01622 , 2018.

Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research , 40(D1):D1100D1107, 2012.

Anna Gaulton, Anne Hersey, Micha ł Nowotka, A Patricia Bento, Jon Chambers, David Mendez, Prudence Mutowo, Francis Atkinson, Louisa J Bellis, Elena Cibrián-Uhalte, et al. The chembl database in 2017. Nucleic acids research , 45(D1):D945-D954, 2017.

Ethan Goan and Clinton Fookes. Bayesian neural networks: An introduction and survey. Case Studies in Applied Bayesian Data Science: CIRM Jean-Morlet Chair, Fall 2018 , pages 45-87, 2020.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

Jonathan Gordon, Wessel P Bruinsma, Andrew YK Foong, James Requeima, Yann Dubois, and Richard E Turner. Convolutional conditional neural processes. In International Conference on Learning Representations .

Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of a superconductor. Computational Materials Science , 154:346-354, 2018.

John J Irwin and Brian K Shoichet. Zinc- a free database of commercially available compounds for virtual screening. Journal of chemical information and modeling , 45(1):177-182, 2005.

Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations .

Sunghwan Kim, Paul A Thiessen, Evan E Bolton, Jie Chen, Gang Fu, Asta Gindulyte, Lianyi Han, Jane He, Siqian He, Benjamin A Shoemaker, et al. Pubchem substance and compound databases. Nucleic acids research , 44(D1):D1202-D1213, 2016.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Kundan Krishna, Jeffrey Bigham, and Zachary C Lipton. Does pretraining for summarization require knowledge transfer? arXiv preprint arXiv:2109.04953 , 2021.

Siddarth Krishnamoorthy, Satvik Mehul Mashkaria, and Aditya Grover. Generative pretraining for black-box optimization. arXiv preprint arXiv:2206.10786 , 2022.

Siddarth Krishnamoorthy, Satvik Mehul Mashkaria, and Aditya Grover. Diffusion models for black-box optimization. arXiv preprint arXiv:2306.07180 , 2023.

Aviral Kumar and Sergey Levine. Model inversion networks for model-based optimization. Advances in Neural Information Processing Systems , 33:5126-5137, 2020.

Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan Brooks, et al. In-context reinforcement learning with algorithm distillation. In NeurIPS 2022 Foundation Models for Decision Making Workshop .

Thomas Liao, Grant Wang, Brian Yang, Rene Lee, Kristofer Pister, Sergey Levine, and Roberto Calandra. Data-efficient learning of morphology and controller for a microrobot. In 2019 International Conference on Robotics and Automation (ICRA) , pages 2488-2494. IEEE, 2019.

Daniel James Lizotte. Practical bayesian optimization. 2008.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations .

Charles A Micchelli, Yuesheng Xu, and Haizhang Zhang. Universal kernels. Journal of Machine Learning Research , 7(12), 2006.

Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2791-2809, 2022.

Samarth Mishra, Rameswar Panda, Cheng Perng Phoo, Chun-Fu (Richard) Chen, Leonid Karlinsky, Kate Saenko, Venkatesh Saligrama, and Rogerio S. Feris. Task2sim: Towards effective pre-training and transfer from synthetic data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9194-9204, June 2022.

Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. arXiv preprint arXiv:2112.10510 , 2021.

Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. arXiv preprint arXiv:2207.04179 , 2022.

Vu Nguyen and Michael A. Osborne. Knowing the what but not the where in bayesian optimization, 2020.

Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data , 1(1):1-7, 2014.

Lars Ruddigkeit, Ruud Van Deursen, Lorenz C Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. Journal of chemical information and modeling , 52(11):2864-2875, 2012.

Karen S Sarkisyan, Dmitry A Bolotin, Margarita V Meer, Dinara R Usmanova, Alexander S Mishin, George V Sharonov, Dmitry N Ivankov, Nina G Bozhanova, Mikhail S Baranov, Onuralp Soylemez, et al. Local fitness landscape of the green fluorescent protein. Nature , 533 (7603):397-401, 2016.

Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE , 104 (1):148-175, 2015.

Gautam Singh, Jaesik Yoon, Youngsung Son, and Sungjin Ahn. Sequential neural processes. Advances in Neural Information Processing Systems , 32, 2019.

Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems , 30, 2017.

Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms, 2012.

Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995 , 2009.

Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018.

Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task bayesian optimization. Advances in neural information processing systems , 26, 2013.

Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective models for effective offline model-based optimization. In International Conference on Machine Learning , pages 10358-10368. PMLR, 2021.

Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Benchmarks for data-driven offline model-based optimization. In International Conference on Machine Learning , pages 21658-21676. PMLR, 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur) , 53(3):1-34, 2020.

James T Wilson, Riccardo Moriconi, Frank Hutter, and Marc Peter Deisenroth. The reparameterization trick for acquisition functions. arXiv preprint arXiv:1712.00424 , 2017.

Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates. arXiv preprint arXiv:2101.07667 , 2021.

Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVI , pages 720-736. Springer, 2022.

Yuhuai Wu, Markus N Rabe, Wenda Li, Jimmy Ba, Roger B Grosse, and Christian Szegedy. Lime: Learning inductive bias for primitives of mathematical reasoning. In International Conference on Machine Learning , pages 11251-11262. PMLR, 2021.

Sihyun Yu, Sungsoo Ahn, Le Song, and Jinwoo Shin. Roma: Robust model adaptation for offline model-based optimization. Advances in Neural Information Processing Systems , 34: 4619-4631, 2021.

Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 , 2016.

Peer reviews:
comments: SUMMARY:
The authors introduce a novel approach SynTO to address a challenging setting - few-shot black-box optimization. Specifically, SynTO adopts a pretraining-adaptation pipeline. SynTO can be pretrained using synthetic functions and then adapt to downstream tasks via an in-context learning manner. Comprehensive experiments demonstrate the effectiveness of the proposed method in multiple settings.

STRENGTHS:
1.	The proposed few-shot BBO is more real-world applicable and generalizable to multiple optimization tasks. Generally speaking, the paper is technically solid.

2.	It is interesting to pretrain models with synthetic data from families of other functions. 

3.	The paper writing is clear, and the presentation is satisfying.

WEAKNESSES:
1.	SynTO assumes the access to large amounts of unlabeled data, which may cause unfair comparisons as the other methods are trained only with a few labeled data.
2.	In Table 1, albeit with the extra unlabeled data, SynTO seems to perform inferior in some settings.
3.	It would be better to give the efficiency of SynTO compared with previous methods, since it further introduces an additional pretraining step.

QUESTIONS:
1.	Can you explain why SynTO degrades more on Max scores under the poorest setting?

2.	In Table 4, can the forward and inverse modeling methods be directly compared regarding their qualitative results, considering the two approaches generate different kinds of outputs?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles black-box-optimization from few-shot examples, by pretraining a transformer model on synthetic proxy tasks using in-context learning, and evaluating with the same procedure but with real data. The synthetic tasks are generated by using 1) real unlabeled data and 2) a synthetic generative process that generates different tasks (i.e. input/targets pairs $x_i$, $y_i$) given the unalbeled data (inputs) for the task of interest. The procedure studied to generate the synthetic tasks is Gaussian processes with a radial basis function kernel. During training, the model receives D input/target value pairs generated by the process, and is tasked to predict the the distribution of the input given a target value ($p(x_y|y_i)$) and the context points using a VAE. At inference time, the model receives the few-shot examples context points, and is tasked to predict the distribution for the current task.\nThe approach results in competitive performance against existing methods, which generally outperforms existing methods.\n\nSTRENGTHS:\n- The method proposed is simple and sound, and incorporates recent progress from transformer-based modelling, such as in-context learning which allows to efficiently adapt the model during inference time without backpropagation.\n\n- Despite its simplicity on how to train the system and use it during inference, it achieves competitive performance on a wide range of benchmarks.\n\n- The experimental section is thorough, giving lots of insights about the method. For example, it gives insights on how the training with GP performs when the (synthetic) tasks during inference are clearly out-of-distribution (Section 4.1), as well as a thorough analysis of alternative design choices such as random vs sorted selection for context/target points, which are shown to be suboptimal.\n\n- The paper proposes pretraining on fully synthetic data, which is easy to construct and has less ethical concerns than pretraining on real data.\n\nWEAKNESSES:\nThe main point of the paper is pretarining using synthetically generated data, and more inisights on why this works at all would make for a stronger paper. My main concerns for the paper are regarding this issue:\n-  For example, regarding the following sentence: \nL50-51: \"Our key insight is, if a model learns to perform few-shot learning on a diverse and challenging set of functions, it should be able to adapt quickly to any objective function at test time.\"  Generating tasks that are \"diverse and challenging\" from the unlabelled data does not seem enough for the method to perform well, and the concepts of \"diversity and challenging\" are not well defined in the paper nor in references, neither theoretically nor experimentally. For example, one could generate a totally random process without any type of correlations, which would be both \"challenging and diverse\" and the method would likely not perform well during inference. Making this notions clearer and relating it to the choice of Gaussian processes, either theoretically or through experimental comparisons with pretraining with other types of processes would make the paper more complete.\n\n- More insights on why the selected random process (Gaussian processes) is good, and analysis with other alternatives would make for a stronger paper. For example, although the functions used for synthetic evaluation in 4.1 are much more simple, it would be good to use them as pretraining at least to illustrate this point. \n- If the only requirement is the data being \"diverse and challenging\", can real data from similar domains (or other domains that can be adapted to match the desired data statistics) be used for pretraining? Why training with a fully synthetic process is an advantage?\n- It would be good to perform an analysis of how the statistics of the pretraining data compares to the statistics during inference (e.g. precision/recall as in [1] or similar metrics), to see if the inference task are statistically different or not when compared to the pretraining tasks. This or similar would illustrate if the the diversity of the pretraining data covers all modes of the inference tasks (high recall) and/or whether the diversity is too high compared to the distribution of data for the inference task (low precision). \n\n\n[1] Assessing Generative Models via Precision and Recall. Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, Sylvain Gelly\n\nQUESTIONS:\nDespite the paper showing strong evidence that the proposed method is sound and outperforms alternatives, more insights on why GP works would make for a stronger paper. See weakness for questions.\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method for tackling few-shot black-box optimization problems in which the model queries a few hundred data points from the black-box function. The proposed method utilizes synthetic pretraining, where a family of synthetic functions is employed to generate data for in-context learning of a transformer-based model. After pretraining, the model adapts to downstream function using few-shot data.\n\nSTRENGTHS:\n- The paper proposes and addresses the problem of few-shot black-box optimization, a problem with many potential real-world applications.\n- The proposed inverse modeling approach is robust to the quality of synthetic data and allows for gradient-free optimization during testing.\n- As Gaussian Processes (GPs) are used to generate synthetic functions, there are no extra costs associated with data generation.\n- Experimental results show performance comparable to previous state-of-the-art methods, while also demonstrating increased robustness to the quality of few-shot data.\n\nWEAKNESSES:\n- There's no guarantee that real-world downstream functions follow a Gaussian Process with a Radial Basis Function (RBF) kernel. In other words, performance might be sensitive to the similarity between downstream functions and those generated by a GP with an RBF kernel. While the authors mention the universal approximation property, it isn't certain that the pretraining stage covers a sufficient range of functions over a sufficient number of training iterations in practice. For instance, the hyperparameters of the RBF kernel used in the experiments are bounded, and the range was heuristically chosen.\n\nQUESTIONS:\n- How sensitive is the proposed approach to the choice of synthetic functions? For instance, if we change the kernel of the GP or use randomly initialized neural networks, how is downstream performance affected?\n- How does the downstream performance vary with the context size? Additionally, does the proposed method perform well when the downstream context size differs significantly from those used during pretraining?\n\nLIMITATIONS:\nThe authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper investigated the problem of few-shot black-box optimization, and presented Synthetically pre-trained Transformer for Optimization  (SynTO). By combining synthetic pretraining with in-context learning to enable few-shot generalization, SynTO demonstrate its superior performance on Design-Bench.

STRENGTHS:
+ Synthetically pre-trained Transformer for Optimization  (SynTO). 
+ Combining synthetic pretraining with in-context learning to enable few-shot generalization.
+ Demonstrating superior performance on Design-Bench. 
+ The paper is well written.

WEAKNESSES:
- Can the problem in this paper be directly solved by few shot learning methods? If so, Some experiments may be required to compare the proposed method with the existing few shot learning methods. Else some discussion may be required to explain this issue. 
- It seems that SynTO shares similar structure with BONET. Thus, some discussion is required to explain the performance gain of SynTO w.r.t. BONET.
- Ablation studies are required to clarify which components of SynTO explain the performance superiority.

QUESTIONS:
- Can the problem in this paper be directly solved by few shot learning methods? If so, Some experiments may be required to compare the proposed method with the existing few shot learning methods. Else some discussion may be required to explain this issue. 
- It seems that SynTO shares similar structure with BONET. Thus, some discussion is required to explain the performance gain of SynTO w.r.t. BONET.
- Ablation studies are required to clarify which components of SynTO explain the performance superiority.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09