Generative Pretraining for Black-Box Optimization
Authors: Siddarth Krishnamoorthy
Satvik Mehul Mashkaria
Aditya Grover
Venue: Submitted to ICLR 2023 (2023)
Keywords: decision making
generative modelling
transformers

Abstract: Many problems in science and engineering involve optimizing an expensive black-box function over a high-dimensional space. For such black-box optimization (BBO) problems, we typically assume a small budget for online function evaluations, but also often have access to a fixed, offline dataset for pretraining. Prior approaches seek to utilize the offline data to approximate the function or its inverse but are not sufficiently accurate far from the data distribution. We propose BONET, a generative framework for pretraining a novel black-box optimizer using offline datasets. In BONET, we train an autoregressive model on fixed-length trajectories corresponding to runs of implicit black-box function optimizers. We design a sampling strategy to synthesize trajectories from offline data using a simple heuristic of rolling out monotonic transitions from low-fidelity to high-fidelity samples. Empirically, we instantiate BONET using a causally masked Transformer and evaluate it on Design-Bench, where we rank the best on average, outperforming state-of-the-art baselines.

Under review as a conference paper at ICLR 2023
GENERATIVE PRETRAINING FOR BLACK-BOX OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Many problems in science and engineering involve optimizing an expensive black- box function over high -igh-dimensiona space. For such black- box optimization (BBO) problems, we typically assume a small budget for online function evalu- ations, but also often have access to fixed, offline dataset for pretraining. Prior approaches seek to utilize the offline data to approximate the function or its in- verse but are not sufficiently accurate far from the data distribution. We propose BONET, generative framework for pretraining novel black-box optimizer us- ing offline datasets. In BONET, we train an autoregressive model on fixed-length trajectories derived from offline dataset. We design sampling strategy synthesize trajectories from offline data using a simple heuristic of rolling out monotonic transitions from low-fidelity to high-fidelity samples. Empirically, we instantiate BONET using a causally masked Transformer (Radford et al. (2199) and evaluate it on Design-Bench (Trabucco al. (2022), where we rank the best on average, outperforming state-of-the-are baselines.
INTRODUCTION
Many fundamental problems in science and engineering, ranging from the discovery of drugs and materials the design and manufacturing of hardware technology, require optimizing an expensive black-box function in a large search space (Larson et al.) 2019; (Shahriari et al. 2016). The key challenge here is that evaluating and optimizing such a black box function typically expensive, as it often requires real-world experimentation and exploration of high-dimensional search space.
Fortunately, for many such black-box optimization (BBO) problems, we often have access to an offline dataset of function evaluations. Such an offline dataset can greatly reduce the budget for online function evaluation. This introduces us the setting of offline BBO. A key difference exists between the offline BBO setting and its online counterpart; in offline BBO, to actively query the black-box function during optimization, unlike online BBO where most approaches (Snoek et al. 2012 Shahriari al. 2016) utilize iterative online solving. One natural approach for offline BBO would be to train a surrogate (forward) model that approximates the black- box function using the offline data. Once learned, can perform gradient ascent the input space to find the optimal point. Unfortunately. this method does not perform well in practice hecause the forward model can incorrectly give sub-optimal and out-of-domain points a high score (see Figure la). To mitigate this issue, COMs (Trabucco et al. 2021) learns a forward mapping that penalizes high scores on points outside the dataset, but this can have the opposite effect of not being able to explore high fidelity points that are far from the dataset. Further, another class of recent approaches
not allowed
(Kumar Levine 2020 Brookes et al. 2019 Fannjiang & Listgarten (020) propose conditional
generative approach that learns an inverse mapping function values to the points. For effective generalization, such mapping needs to be highly multimodal for high-dimensional functions, which in itself presents a challenge for current approaches.
We propose Black-box Optimization Networks (BONET), a new generative framework for pretrain- ing black-box optimizers on offline datasets. Instead of approximating the surrogate function (or its inverse), we seek to approximate the dynamics of online black-box optimizers using an autoregres- sive sequence model. Naively, this would require access to several trajectory runs of different black- box optimizers, which is expensive or even impossible in many cases. Our key observation is that we can synthesize synthetic trajectories comprised of offline points that mimic empirical characteristics

---

Under review as a conference paper at ICLR 2023
y-ftx)
True mundrn
-01
Fanctioo Ub
-03
Modro Diteee
8
Drennn Sil EaraerpCanne oulbloonhior
20
35
Xa
Horizon
(a)
(b)
(c)
Figure 1: (a) Example of offline BBO on toy ID problem. Here, the domain ends the red dashed line. Thus, the correct optimal value is x", whereas gradient ascent the fitted function will output out of the domain point (b) Example trajectory on the 2D-Branin function. The dotted lines denote the trajectories in our offline dataset, and the solid line refers to model trajectory, with low-quality blue points and high-quality red points. (c) Function values of trajectories generated by simple gaussian process (GP) based BayesOpt model several synthetic functions.
of online BBO algorithms, such BayesOpt. While could design many characteristic proper- ties, we build off an empirical observation related to the function values of the proposed points. In particular, averaged over multiple runs, online black-box optimizers (c.g., BayesOpt) tend show improvements in the function values of the proposed points (Bijl et al. (2016). as shown in Fig- ure Tc While not exact, we build on this observation to develop a sorting heuristic that constructs synthetic trajectories consisting of offline points ordered monotonically based on their ascending function values. Even though such heuristic does not apply uniformly for the trajectory runs of all combinations of black-box optimizers and functions, we show that it is simple, scalable, and quite effective practice.
Further, augment every offline point in our trajectories with regret budget, defined as the cu- mulative regret of the trajectory starting at the current point until the end the trajectory. We train BONET to generate trajectories conditioned on the regret budget of the first point of the trajectory. Thus, at test time, we can generate good candidate points by rolling out a trajectory with a low regret budget. Figure 1b shows illustration.
We evaluate our method on several real-world tasks in the Design- Binggh-Beechh (Trabucco et al. 2022) dataset. These tasks are based real-world problems such robot morphology optimization, DNA sequence optimization, and optimizing superconducting temperature of materials, all which requires searching 0.772 and an average rank of 2.4 across all tasks, outperforming the next best baseline, which achieves a rank of 3.7.
high-dimensional search space. We achieve normalized mean of
PRETRAINING BLACK-BOX OPTIMIZERS VIA BONET
2.1 PROBLEM STATEMENT
Let f X + IR be a black-box function, where X' C I an arbitrary d-dimensional domain. In black-box optimization (BBO), are interested finding the point x" that maximizes f:
X E arg max (x)
XEX
Typically, f expensive evaluate and we not assume direct access to it during train- ing. Instead, we have access to offline dataset of N previous function evaluations D ((x1.81). (XN.93, B.)), where y: = f(x)). For evaluating a black-box optimizer post-training, allow it to query the black-box function f for small budget of Q queries and output the point with the best function value obtained. This protocol follows prior works offline BBO (Trabucco)
2021 2022 al.
Kumar& & Levine 2020 Brookes et al.) 2019 Fannjiang & Listgarten 2020
Overview of BONET We illustrate our proposed framework for offline BBO in Figure 2 and Algo- rithm T. BONET consists of 3 sequential phases: trajectory construction, autoregressive modelling.
2

---

Under review as a conference paper at ICLR 2023
Offline dataset
X
D
SORT-SAMPLE
Autoregressive Transformer
(Phase 1)
Dirai
R2 budget point
R
Training (Phase 2)
Trajectory dataset
R
RP
XP
XP+1
Evaluation (Phase 3)
prefix
predictions
Figure 2: Schematic for BONET. In Phase I, we construct trajectory dataset Piraj using SORT- SAMPLE. In Phase 2, we learn an autoregressive model for Duraj- In Phase 3, we condition the model on offline prefix sequence and unroll it further obtain candidate proposals 8p41:11.
roll-out evaluation. In Phase 1 (Section 2.2
we transform the offline dataset 'D into trajectory dataset Danj- This followed by Phase 2 (Section 2.3 where we train an autoregressive model on Dirj- Finally, we evaluate the model by rolling out candidate points in Phase 3 (Section) 2.4.
2.2 PHASE 1: CONSTRUCTING TRAJECTORIES
Our key motivation BONET is to train a model to mimic the sequential behavior online black- box optimizers. However, the diffieulty is that do not have the ability generate trajectories by actively querying the black-box function during training. In BONET, we overcome this difficulty by synthesizing trajectories purely from an offline dataset based on two guiding desiderata.
Algorithm Black-box Optimization Networks (BONET)
Input Offline dataset DD Evaluation Regret Budget Ri, Prefix length P. Query budget Q, Trajec- tory length T, num_ num_trajs, Smoothing parameter K, Temperature т, Number of bins NB
Output A set of proposed candidate points X with the constraint |X| Q
1: Phase SORT-SAMPLE
2: Construct bins (...... ... .BNE) } from DD each bin covering equal y-range, as described in 2.2
3: Calculate the scores (n1 m2. ... RNgB) ) for each bin using K and r
4: Duaj f-
5: for = I,... num trajs do
6: Uniformly randomly sample n, points from B, and concatenate them to construct T
7: Sort T the ascending order of the function value
8: Represent T as (R1, X1, R2, x., Rr, XT), following equation and append T Duraj
9: end for
Phase 2: Training
11: Train the model to maximize the log- log-likelihood of Danaj using the loss in equation
Phase 3: Evaluation
13: Construct a trajectory T'' from D following Phase 1, and delete the last T - P points Calculate R, and feed (R, x,) to 9o sequentially, Vt 1, 1.......
15: Roll-out go autoregressively while feeding R₂ = R, Vt P P+1....,7
16: X is the set of last min(Q, P) rolled-out points.
First, the procedure for synthesizing trajectories should efficiently scale to high-dimensional data points and large offline datasets. Second, each trajectory should mimic characteristic behaviors commonly seen in online black-box optimizers. We identify such characteristic of interest. In particular, we note that the moving average of function values of points proposed by such black-bor box optimizers tends to improve the course of their runs barring local perturbations (e.g., due to
3

---

Under review as a conference paper at ICLR 2023
exploration). While exceptions and do exist, this phenomena is commonly observed in practice for optimizers such as BayesOpt (Bijl et al. 2016). We also illustrate this behavior in Figure Ic for some commonly used test functions optimized via BayesOpt.
Sorted Trajectories We propose to satisfy the above desiderata in BONET by following a sorting heuristic. Specifically, given set of T offlin points, construct trajectory of length T by simply sorting the points in ascending order from low to high function values. We note that sorting is just heuristic we use for constructing synthetic trajectories from the offline dataset, and this behavior may not be followed by any general optimizer over any arbitrary functions. We also perform ablations on different heuristics Appendix C.T Further, we note that sorting does not provide any guidance on the rate or the relative spacing between the points i.c. how fast the function values increase. This rate is important for controlling the sample budget for black-box optimization. Next, we discuss a sampling strategy for explicitly controlling this rate.
Sampling Strategies for Offiin Points So far, we have proposed simple heuristic for transitioning set of offline points into sorted trajec- tory. To obtain these offline trajec- tory points from the offline dataset, one default strategy is to sample uni- formly at random T points from D and sort them. However, we found this strategy to not work well in prac- tice. Intuitively, we might expect a large volume of the search space to consist of points with low-function values.
0.00
6003
430
-30
-00
180 0208
-300
-300
Fanction
Punction Valoo
(a)
(h)
Figure 3: Plots showing the distribution of function values in the offline dataset D (left) and the trajectories in Duri (right) for the Ant Morphology benchmark (Trabucco et al. (002). Notice how the overall density points with high function values is up-weighted post our re-weighting.
Thus, if our offline dataset is uni- formly distributed across the domain, the probability of getting a high qual- ity point will be very low with a uni- form sampling strategy. To counter this challenge, we propose 2 step sampling strategy based on binning followed by importance reweighting. Our formulation is motivated by a similar strat- egy proposed by Kumar & Levine (2020) for loss reweighting. First, partition the offlin dataset DD into NB bins of equal-width, i.e., each bin covers a range of equal length. Next, for each bin, we assign different sampling probability, such that (a) bins where the average function value is high are more likely to be sampled, and (b) bins with more points are sampled more often. The former helps minimize the budget, whereas the latter ensures diversity in trajectories. Based on these two criteria, the score for a bin B; is given as:
the function values to
B.J B K exp
(- yb,
8;
T
where ý is the best function value the dataset D. [B|| refers to the number of points in the zth bin, and yb, is the midpoint of the interval corresponding to the bin B,. Here, the first term [B,|+K allows us to assign a higher weight to the larger bins with smoothing. The second term gives higher weight to the good bins using an exponential weighting scheme. More details about K and T can be found in Appendix B Finally, we use these scores 8; to proportionally sample n. points from bin B, where n4 = T'
|B,]
for i E {2, .... ,NB and ni = T Ei>1 T4i making the overall length of the trajectories equal to T. In Figure 3 we illustrate the shift in distribution of function values due to sampling strategy. We refer to the combined strategy of sampling and then sorting as CORT-SAMPLL in Figure 2 and Algorithm
Augmenting Trajectories With Regret Budgets Our sorted trajectories heuristically reflect roll- outs of implicit black-box optimizers. However they do not provide us with information on the rate at which trajectory approaches the optimal value. A natural choice for such knob would be the cumulative regret. Moreover, as we shall show later, cumulative regret provides BONET a simple and effective knob to generalize outside the offline dataset during the evaluation phase.
4

---

Under review as a conference paper at ICLR 2023
Hence, propose augment each data point x, in trajectory with a Regret Budget (RB). The RB RS at timestep defined as the cumulative regret of the trajectory, starting at point x;; f(x,)). Intuitively, high (low) value for R2 is likely to result in a high (low) budget for the model to explore diverse points. Note, we are only assuming knowledge of an estimate for f(x") (and not x*). Thus, each trajectory in our desired set Diaj can be represented
R. = (fx)
T (R,, X1, R2, X2, RT. XT)
We will refer to Ri as Initial Regret Budget (IRB) henceforth. This will be of significance for evaluating our model in Phase (Section 2.4p. as we can specify a low IRB induce the model to output points close to the optima.
2.3 PIIASE 2: TRAINING AN AUTOREGRESSIVE GENERATIVE MODEL
Given trajectory dataset, design our BBO agent conditional autoregressive model and train to maximize the likelihood of trajectories in Dus- More formally, we denote our model parameterized by 0 go (x, xxt,, Ret), where by kic mean the set (ki,... -1-1). Here, x; are the sequence of points in a trajectory, and R3 refers to the regret budget at timestep i.
Building recent advances in sequence modeling (Vaswani et al. 2017 Brown et al. 2020 Rad- ford et al. 2019) we instantiate our model with a causally masked transformer and train it to maximize the likelihood of our trajectory dataset Dirj.
[(0: Dmj) =
Σ log go (x, Rgs)
(4)
T T-Diai
In practice, we translate this loss to the mean squared error loss for continuous X' (equivalent to a Gaussian ge with fixed variance), and cross entropy loss for discrete X.
2.4 PHASE 3: EVALUATION ROLLOUT OF FINAL CANDIDATES
Once trained, we can use our BBO agent to directly output new points its candidate guesses for maximizing the black-box function. We do so by rolling out evaluation trajectories from our model. Each trajectory will be subdivided into a prefix subsequence and a prediction subsequence. The prefix subsequence consists of P < T points sampled from our offline dataset as before. These prefix points provide initial warm-up queries to the model. Thereafter, we rollout the prediction subsequence consisting of T - P points by sampling from our autoregressive generative model.
Setting Regret Budgets One key question relates to setting the regret budget at the start the suffix subsequence, It is not preferable to set it to Rp+1 of the sampled trajectory, as doing will lead to a slow rate of reaching high-quality regions similar to the one observed This will not allow the suffix to generalize beyond the offline dataset.
training trajectories.
Alternatively, we initialize it to a low value BONET to accelcrate the trajectory towards good points following prefix subsequence. We refer to this low value Evaluation RB and denote it as R in Figure 2 and in Section 3 Thereafter, we keep the RB for the suffix subsequence fixed (f), as the agent is expected to be already in good region. Moreover, updating the RB here would require sequential querying to the function ʃ. which
be prohibitive. Thus, our evaluation protocol can generate a set of candidate queries purely in an offline manner. In practice, we also find it helpful to split the Q candidates among few (and not 1) small R values, each with different prefix.
3 EXPERIMENTAL EVALUATION
We first empirically evaluate BONET for optimizing a synthetic 2D function, Branin, in order to analyze its working and probe the various components. Next, perform scale benchmarking and experiments on Design-Benn Bench (Trabucco et al. 2022), suite of offline BBO tasks based on real-world problems.
5

---

Under review as a conference paper at ICLR 2023
3.1 BRANIN TASK
Branin is well-known benchmark function for evaluating optimization methods. is 2D function evaluated on the ranges 2 E [-5, 10] and 22 E [0, 15]:
(0r a(æ2 - bxq + cœ - r)² + s(1 1) cos σ +
(5)
where a -1 6 4,. c 5 r 6 and 1. In this square region, for has three global maximas. (-T, 12.275) (T,, 2.275), and (9.42478, 2.475); with the maximum value of -0.397887. Figure [bb shows an illustration of the function contours. For offline optimization we uniformly sample N 5000 points in the domain, and remove the top 10%-ile (according to the function value) from this set to remove points close to the optima to make the task more challenging. We then construct 400 trajectories of length 64 each according to the PORT-SAMPLE strategy. During the evaluation, we ini- tialize four trajectories with prefix length of 32 and un- roll them for additional 32 steps, and output the best result, thus consuming a query budget of 128. As we see in Table BONET successfully gener- alizes beyond the best point in our offline dataset. We also re- port numbers for gradient as- cent baseline, which uses the of-
Best function value achieved by each method on Branin task. We report mean and standard deviation averaged over runs. Gradient ascent performs poorly because on many initial- ization points, the trajectories escape out of the square domain.
Table 1:
OPTIMA
D (best)
BONET
Grad. Ascent
-0.398
-6.119
-1.79 + 0.843 -3.953 + 4.258
fline dataset to train forward model (a 2 layer NN) mapping to y and then performs gradient ascent on x to infer its optima. Next, we perform ablations to understand the effect of the Evaluation RB ft and prefix length P on our rolled-out trajectories.
Impact of R Figures 4a and 4b shows rolled-out trajectories for our model for different Å values, with prefix lengths 16 and 32. We see that low ît rolls out higher quality points compared to high R.. To verify our semantics of regret budget as knob for controlling the rate at which the model accelerates high-quality points, the Figure 4c we also plot trajectories where we update the RB values in the suffix. We stop the roll-out if RB becomes non-positive. It is evident that for smaller R. the agent quickly accelerates to high-quality regions, whereas for high R, it gradually shifts to high-quality points. This shows how it controls the rate of transition from low to high-quality points.
To further check whether our model has learned to generate a sequence having cumulative regret close the initial RB Ri. plot R1 vs the cumulative regret of full rolled-out sequence in Figure [a We observe that the curve is close to the desired idcal line y = x. Notice that the range of Rí values of Duraj is quite narrow, but BONET is able generalize well to much wider range, allowing it to propose points even better than the dataset. During training, the model has only seen low RB values towards the end of the trajectories. However, the powerful stitching ability (Chen) et al. 2021) of the model allows it to roll out a novel trajectory having low cumulative regret when conditioned on low unseen Rt values. Finally, since in BBO our goal is to find the best point, we also plot the best rolled-out point across a trajectory versus i in Figure 5b while keeping the prefix sequence fixed. As expected, we observe a decreasing trend, justifying our choice of small Å values.
Impact of Prefix Length Figure 5c shows the obtained best function values for different prefix lengths, averaged over multiple R values, with same query budget Q 32. As expected, we see an increasing trend in the best function value. We also observe a decreasing variance, indicating that the trajectory roll-outs are more stable when augmented with history of points. Note that prefix lengths larger than 32 doesn't perform very well in practice because they have fewer than 32 shots to propose good point in single trajectory. Empirically, the trajectory length to perform well across the experiments. We provide more details about the ablations, experimental setup, and model hyper-parameters in the Appendix Bnd C
found prefix length equal to half of

---

Under review as a conference paper at ICLR 2023
Purtiton Seluu
Funcion Value
Purriton
4.0
4.0
6.0
2.0
2.0
4.01
0.0
0.0
2.0
Prefix Sequence
Prefix Sequence
Profix Sequence
Timestep
Timestep
Timestep
(a)
(b)
(c)
Figure 4: Rolled-out trajectories for Branin task for multiple R values (averaged over 5 runs). Figure (b) shows trajectories with prefix length 32, without updating RB (default evaluation setting). In Figure (a), we change the prefix length to 16 while in Figure (c), we update RB in the suffix.
-15
-22
Cummatt va Blgge
523
Functoo Volue
23.
Dertt nometton
-40
50
Meal
25
Datasee Mirimum
Estasec Maximum
20
Intt Regret Runnee
15 100
150 u2
Addiional Budcet
18.0
233
(a) Initial RB vs Cumulative Regret
(b) Variation with ft
(c) Variation with Prefix length
Figure 5: Ablation studies for Branin task. All the values averaged over 5 runs.
3.2 DESIGN-BENCH TASKS
Next, we evaluate BONET on 7 complex real-world tasks of Design-Bench (Trabucco et al. 2022 TF-Bind-8 and TF-Bind-10
discrete tasks where the goal is to optimize for a DNA sequence which has maximum affinity to bind with particular transcription factor. The sequences are of length (10) for TF-Bind-8 (TF-Bind-10), where each element the sequence one of 4 bases. ChEMBL is a discrete task where the aim is to design a drug with certain qualities. NAS is a discrete task where we want optimize a NN for performance on CIFARIO (Krizhevsky et al., 2010). In D'Kitty and Ant morphology tasks, we optimize the morphology of two robots: Ant from OpenAI gym (Brockman et al. 2016) and D' Kitty from ROBEL (Ahn al. 2019). Superconductor task, the aim is to find a chemical formula for superconducting material with high critical temperature. D' Kitty, Ant and Superconductor are continuous tasks with dimensions 56, 60, and 86 respectively. For the first four tasks, we have query access to the exact oracle function. For Superconductor, only have an approximate oracle, which is random forest regressor trained on a much larger hidden dataset. These tasks considered challenging due high dimensionality, low quality points in the offline dataset, approximate oracles in some cases, and highly sensitive landscapes with narrow optima regions (Trabucco al. 2022).
Baselines We compare BONET with multiple canonical baselines like gradient ascent, REIN- FORCE (Sutton et al. 1999). BayesOpt (Snoek et al.) 2012) and CMA-FS (Hansen) 2006). We also compare with more recent methods like MINs (Kumar & Levine 2020). COMs (Trabucco) al. (021)) and CbAS (Brookes al. 2019 For inherently active methods like BayesOpt, since we cannot query the oracle function f(x) during optimization (due to being in an offline setting), we follow the procedure used by Trabucco al. (2022), and perform BayesOpt on surrogate model
f(x) (a feedforward NN) trained on the offline dataset. For the BayesOpt baseline, we use a Gaus- sian Process to quantify uncertainty and use the quasi-Expected Improvement (Wilson et al. 2017)
haven't included Hopper since domain buggy we found that oracle function used to evaluate the task was highly inaccurate and noisy. An expanded discussion can he found in Appendix B.7
We take the haseline implementations from https///ttta..mmmmaaaalatatteeet

---

Under review as a conference paper at ICLR 2023
Table 2: 100th percentile comparative evaluation of BONET over tasks averaged over 5 runs. The error bars refer to the standard deviation across the 5 seeds. We report normalized scores with Q = 256 (except for NAS, where we use Q = 128 due to compute restrictions) and highlight the top 2 results in each column. Blue denotes the best entry in the column, and Violet denotes the second best. We observe that BONET is the top 2 in 5 out of tasks, and is also consistently able to outperform the best offline dataset point in all tasks.
BASELINE TRRIND
TERIND10
SEPSROONICCTOR
ANT
I'KIITY
CHEMMIE
NAS
BEANSCORES (T)
MIAAN BANK (T)
CSAS
0300 I1008
B1ST & ILDIT
0.43 (108S
0,855 (1855
1
5-4
0.844 2100
0..11
00,00
0.0.00
CMN-ES
0.070
0.401=-0.001
1.930
0.721=0.000
0-985-20..770
0.540:00:550
Gualcal Noon
0.81120.015
0,029 0.000
1:000:00.00
0.342-30344
00001/0017
0.617
=.000
0.415.400.00
0.525+00.015
6.0
03495 11005
0.211-0012
14463 1:541-7
0:351 11
0.24220.20
01535
-1.06 1.000
B,222 1.044
CUME
[L96] 0:12
0.354.1.222 0.12
1/113
03883 (166)
0.918 0.000
01683
11166
0.039:100:13
0.720 0/131
(0un)
0.004
0.085
0.002
0.064 0.012
0.054 0.011
0.734:-00005
0.772::0.000
Table 3: 50th percentile evaluations all the tasks. Similar to Table 2 find that BONET achieves both the best average rank and the best mean score on all tasks. This shows that BONET consistently outputs good candidate points.
RASELINE
TFBINDS
IPRIND10
SUPERCONDUCTOOR
ANI
D'KIFTY
CHEMBL
NAS
MEAN SCORE
MEAN BANK
ChAS
6,478 0:001
0.311 0:000
0.804+-0000
0 +-003
0.633
-00000
0.202-0.077
0:535 0:004
3.7
-0.00
0:272--00806
4-0003
10,09
CMA-ES
0563-1000007
6.653=0:011
0.37=00061
-0.601
0.033 10:0-00.000
0.591
0.965-0000 4-0000
3.4
Grdden Auces
6:670=D0000 0:00
0:463 - 0.002
0::61
0.637 +:0.11
0663- -0000
0:331 0-330-0.000
0.55 0- 0.035
4.3
RRINPORCE
0.550---0600
6.172 0.000
0.470 0.017
0.186-1-::00
0.317 4-000-
0.633 -0000
-1.555-0.000
0 0-003
4.9
COVs
0.852:4-0.000
0,179 0.012
0.365
0.375 0:565-
0.885 0-003
0.633
4.0173
0.509 0:003
3.8
BONET
0
0.495 =: 0.087
0,300 = 0.005
0.810 2: 0.092
0,907 =: 0.030
0,030 0:000
0,871
0.014 0.035
2.8
acquisition function for optimization, similar to prior work (Trabucco et al. 2022 2021). MINs (Ku-
mar & Levine (2200) also train a forward model to optimize over the conditioning parameters. Our method does not need separate forward model and thus, is not dependent on the quality of the fit of the forward model. We provide other variants of BayesOpt baseline in Appendix D.2
Evaluation We allow a query budget of Q 256 for all the baselines, except for NAS, where we use reduced budget of Q = 128 due to compute restrictions. For BONET, across all tasks, we roll out 4 trajectories, each with a prefix and prediction subsequence length of 64 each. The prediction subsequence is initialized with one of 4 candidate low R values 0, 0.01, 0.05, 0.1. For each R, then roll out for 64 timesteps and choose the best point. We report the mean and standard deviation over trials for each of the models and tasks in Table 2 Following the procedure used by (Tabbcco al.. 2021 2022 Yu et al. (02T), the results of Table Jare linearly normalized between the minimum and maximum values of a large hidden offline dataset. In Table 3 we also present the median function value of the the proposed output points for each method, averaged over runs.
Results Overall, we obtain a mean score of 0.772 and an average rank of 2.4, which is the best among all the baselines. We also achieve the best results on three tasks. Additionally, we are among the top? 2 for five out of seven tasks, We show significant improvements over generative methods such as MINs (Kumar & Levine 2020) or CbAS (Brookes et al. 2019), and forward mapping methods such as COMs (Trabucco et al. 2021) on TF-BBind8, TF-Bind 10, Ant and D' D'Kitty. We also note that while BONET is placed second in Ant, it shows a much lower standard deviation (0.012) compared to the to the best performing method CMA-ES, which has a much larger standard deviation of 0.928. We also have a lowest mean standard deviation across all tasks, suggesting that BONET is less sensitive bad initializations compared to other methods. We report the un- normalized results, ablations and other experimental details for Design- Bench tasks in Appendix (C) BONET also performs best 50th percentile evaluation, showing that it has better set of proposed points compared to other methods, and that the performance not by randomly finding good points.
Besign-Benchh
RELATED WORK
Active BBO The majority of prior work in BBO have been in the active setting, where surro- gate models are allowed query the function during training. This includes long bodies of work
8

---

Under review as a conference paper at ICLR 2023
in Bayesian Optimization, e.g., (Snoek et al.] 2012;
Swersky al. 2013 Srinivas al. 2010
Nguyen & Osborne| 2020) and bandits, e.g., GGarivier et al. [016:] Riqueime et al. 2018 Joachims et al.] 2018b a Swaminathan & Joachims 2013 Jacg al. (019). Such methods usually employ
surrogate models such as Gaussian Processes (Srinivas et al. 2010), Neural Processes (Garnelo
et al. 20T8a bj Gordon et al.] 2019 Anonymous 2022 /Singh al., 2019) or Bayesian Neural
Networks (Chang) [022]] |oaan & Fookes 2020) to approximate the black-box fuucction, and an
uncertainty-aware acquisition strategy for querying new points.
Offiine BBO Recent works have made use of such datasets and shown promising results (Kumar
& Levine 2020 Trabucco et al. 2021 Brookes al., 2019 Fannjiang & Listgarten 2020; Fu &
Levine |021 Yu et al.] 2021).
Kumar & Levine (2020) train a stochastic inverse mapping from
the outputs y to inputs x using generative model similar to conditional GAN (Goodfellow al.)
2014 MMrza & Osindero 2014,
Arjovsky et al. 2017; Nowozin et al. 2016). They then optimize over y to find a good design point. Training GANs however can be diffiultt due to issues like mode collapse (Arjovsky
al. 2017). Other methods make use of gradient ascent to find an optimal solution. Trabucco et al. (202T) and Yu et al. (202T) train a model to be robust to outliers by regularizing the objective such that they assign low score to those points. Fu & Levine (2021) train normalized maximum likelihood estimate of the function. We instead offer a fresh perspective based on generative sequence modelling and show strong results in comparison to many of these prior
works in Section 3
Offiine reinforcement learning (RL) While both RL and BBO sequential decision-making problems, the key difference is that RL is stateful while BBO is not. In the offline setting (Schmid- huber 2019 Jacq et al., 2019), both problems require models or policies that can generalize beyond the offline dataset to achieve good performance. Related to our work, autoregressive transformers have been successfully applied on trajectory data obtained from offline RL (Chen al. 2021 Janner et al. (021). However, there are important differences between their setting and offline BBO, For example, the data offline BBO not sequential in nature, unlike in offline RI. where the offline data is naturally in the form of demonstrations. One of our contributions is to design notion of 'high-to-low' sequences to BBO for autoregressive modeling and test-time generalization.
Learned optimizers Our work also bears resemblance to work on meta-learning optimizers like those of (Andrychowicz et al. 2016) and (Chen et al. 2017). However, a key difference between these learned optimizers and BONET is that they require access to gradients either during training time or during both training and evaluation time, whereas BONET has no such restriction, meaning it can work in situations where access gradient information is not practical (for instance with non- differentiable black-box functions). Furthermore, we concentrate on the offline setting, in contrast to learned optimizer work which usually looks an active optimization setting.
DISCUSSION
We presented BONET, novel generative framework for pretraining black-box optimizers using offline data. BONET consists of a 3 phased process. In the first phase, we use a novel SORT- SAMPLE strategy to generate trajectories from offline data that use the sorting heuristic to mimic the behavior of online BBO optimizers. In phases 2 and 3, we train our model using an autore- gressive transformer and use generate candidate points that maximize the black-box function. Experimentally, we verify that BONET is capable of solving complex high- nigh-dimensional tasks like the ones in Design- -ench,, achieving an average rank of 2.4 with a mean score of 0.772.
Limitations and Future Work BONET assumes knowledge of the approximate value of the optima f(x") to computes the regrets of points. Though we show Appendix C.5 that different reasonable estimates of f(x") can give similar results, this is still something we would like to address in future work. We are also interested in extending BONET to an active setting where our model can quantify uncertainty and actively query the black-box function after pretraining on offline dataset. On a practical side, we would also be interested in analyzing the properties of domains that dictate where BONET can strongly excel (e.g.. D' D'Kitty) struggle (e.g., Superconductor) relative to other approaches. Finally, we aim to expand our scope to a meta-tase task setting, where instead of a single function, our offline data consists of past evaluations from multiple black-box functions.

---

Under review as a conference paper at ICLR 2023
6 ETHICAL AND SOCIAL RESPONSIBILITY
Offline black-box optimization can play a critical role in improving efficiency and safety in many real-world settings like nuclear reactors the pharmaceutical industry, where active optimization may prove to be computationally inefficient (requiring too many queries) or even dangerous. How- ever, while we do not anticipate anything inherently malicious about our work, is possible to utilize our proposed method (and other optimizers in general) in malicious settings (e.g.. optimizing for drugs that have harmful effects). This is something to remember when deploying algorithms such ours for high stake real-world applications.
REPRODUCIBILITY
Throughout the paper, we maintained a high standard of rigor for reproducibility. We report the detailed pseudocode of our method in Algorithm [ and we also provide the link to our code via an anonymized link here We provide more details on our training setup and choice of hypeparammeters in Appendix B We report results on multiple datasets in Design-Bench (Trabucco et al. 2022), each with different properties and benchmark our method over multiple baselines from different families of approaches. Our results averaged over 5 seeds, and also provide the standard deviations. We also conduct several ablations to evaluate sensitivity of BONET to different parameters.
REFERENCES
Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. Robel: Robotics benchmarks for learning with low-cost robots. 2019. doi:
0044850//RXII.1990.110
httpss/ org/ wwwpooggg/bbbbb.ee httpg//b//199... 11639
Marcin Andrychowicz. Misha Denil, Sergio Gómez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning to learn by gradient de- scent by gradient descent. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Asso- ciates, Inc., 2016. URL httttts::/prrceeeeeeee 98875587825559228884442222228
neuriperpe wwteeeuiss..cpppppeeeeeeeeee cccpapper//11///1eee.. pdf
Anonymous. Neural bootstrapping attention for neural processes. In Submitted to The Tenth In- ternational Conference Learning Representations, 2022. URL httts::///peeeeeeeeeeett
net htte//orummicc. Setiimiid--77Viia 27VhFVRVgeU under review.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial net- works. In Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Con- ference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 214 223. PMLR, 06-11 Aug 2017. URL https::/proceet ar arjovsky17a html
ml vrCss/v70/
Hildo Bijl, Thomas B. Schön, Jan- Willem van Wingerden, and Michel Verhaegen. A sequential monte carlo approach thompson sampling for bayesian optimization, 2016. URL https:
//arxiv. wwpxii.org//bb//10000 00169
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learming, volume 97 of Proceedings of Machine Learn- ing Research, pp. 773-782. PMLR, 09-15 Jun 2019. URL httpps:/pproceeeee
mmr.
html
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh,
10

---

Under review as a conference paper at ICLR 2023
Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL https::/pprocceiingg..oo
http://w........ccccco/////////1 1457c0d6bf 116.74888008c11426664444 Pawp:/eeAaaPaap..pppp. Paper.
Daniel T. Chang. Bayesian neural networks: Essentials, 2021. URL httts::/arxiv.cr
00s/22006.1599)
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.01345, 2021.
Yutian Chen, Matthew W. Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Timothy P. Lilli- crap, Matt Botvinick, and Nando de Freitas. Learning learn without gradient descent by gra- dient descent. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Reseurch, pp. 748-756 PMLR, 06-11 Aug 2017. URL https: htttt::://proceedingg ml mwr.press//70. chen17e.hir
Clara Fannjiang and Jennifer Listgarten. Autofocused oracles for model-based design, 2020. URL https: http::://axxvv.vrr www.r////////aa abaaaaass/200 08052
Justin Fu and Sergey Levine. Offline model-based optimization via normalized maximum likelihood
estimation. In International Conference on Learning Representations, 2021. URL https:
//openzeview. wwwp://wwieee.ne/ffoeoeee wwt//eet//oooee (aees//forrmmmdddmmtteeeeeeee d=MM...4
Aurélien Garivier, Emilic Kaufmann, and Tor Lattimore. On explore-then strategies. In Proceedings of the 30th International Conference on Neural Information Processing Sys- tems, NIPS' 16, pp. 784-792, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional neural pro- cesses. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Con- ference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1704-1713, PMLR, 10-15 Jul 2018a. URL httph::t////t htttte:///rrccedinggs mlr ott.reedddnng.....peess////// garnelo18a html
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01622 2018b.
Ethan Goan and Clinton Fookes. Bayesian neural networks: An introduction and survey. In Case Studies in Applied Bayesian Data Science, pp. 45-87. Springer International Publish- ing, 2020. doi: 10. 11211007/788---00-4555555 3. URI.
www.oor...... 10../1..100. 1007%
2F978-3-030-2255-
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, www.enwwww......eeeeeeeeeeeeeeee
Inc., 2014. URL htttss://proceeddngs neurips cc cc//paper/201//E11e 5e289901224411110335cchhhaaaaaee
Jonathan Gordon, Wessel P Bruinsma. Andrew YK Foong, James Requeima, Yann Dubois, and Richard E Turner. Convolutional conditional neural processes. arXiv preprint arXiv:1910.13555 2019.
Nikolaus Hansen. The CMA Evolution Strategy: A Comparing Review, pp. 75-102. Springer Berlin Heidelberg, Berlin, Heidelberg, 2006. ISBN 978- 97-35543492- doi: 10.1007 000---000------ 4. URL http::/::..o..ooo/0000000/0//-30

---

Under review as a conference paper at ICLR 2023
Alexis Jacq, Matthieu Geist, Ana Paiva, and Olivier Pietquin. Learning from learner. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2990-2999 PMLR, 09-15 Jun 2019. URL htttps://procseddinn .algg.mlll.... /ttp jacql 19a
html
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
T. Joachims, A. Swaminathan, and M. de Rijke. Deep learning with logged bandit feedback. In International Conference on Learning Representations (ICLR), 2018a.
Thorsten Joachims, Adith Swaminathan, and Maarten de Rijke. Deep learning with logged bandit feedback. In International Conference on Learning Representations, 2018b. URL https:// openreview net
-Xb
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced search). URL http://www cs. toronto. edu/kriz/cifar html, 5(4): 2010.
Aviral Kumar and Sergey Levine. Model inversion networks for model-based optimiza- tion. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. ciates, Inc., 2020. URL httpps::/pproceeddiiiit neurips wwtpaapppp..cppppee///2200 cccppaper//22///iieeee Paweoc
Curran Asso-
Jeffrey Larson, Matt Menickelly, and Stefan M. Wild. Derivative-free optimization methods. Acta Numerica, 28:287-404, may 2019, doi: 10. 00.1077/s092249911900000 URL hattps://dot org/10. /tt0//100007122209922222933 9000000
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets, 2014. URL https:
7/ 7/arxiv.org/
11884
Vu Nguyen and Michael A. Osborne. Knowing the what but not the where in Bayesian optimiza- tion. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Con- ference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 7317-7326. PMLR, 13-18 18 Jul 2020. URL https: https::/prroceednggs mlr press/v119/
nguyen20d.htm.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran As- sociates, Inc., 2016. URL https: httppssspppcceeddnngg. newe cedebb6e8 8 72 E53 wee39beeec333f33 8c38 991 988819988111111 74e88774477-- Paper.
neur ips. wwtppppeee/01///500
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Carlos Riquelme, George Tucker, and Jasper Snoek. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. arXiv preprint arXiv:1802.001177, 2018.
Juergen Schmidhuber. Reinforcement learning upside down: Don't 't predict rewards just map them to actions, 2019. /ttps::::///axx
abs 191..1991..00 02875
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan Adams, and Nando de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1): 148-175, 2016. doi: 10. 10.100///R0..2215.222222
Gautam Singh, Jaesik Yoon, Youngsung Son, and Sungjin Ahn. Sequential neural processes. arXiv preprint arXiv: 10264, 2019.
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms, 2012. URL https:
/ tttttsss//aaxii..///aaaaa.e
2944
12

---

Under review as a conference paper at ICLR 2023
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias W. Seeger. Gaussian Process Opti- mization in the Bandit Setting: No Regret and Experimental Design. In Proceedings of the 27th International Conference Machine Learning, pp. 1015-1022. Omnipress, 2010.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradi- ent methods for reinforcement learning with function approximation. In S. Solla, T. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999. URL stttpss://rroeediiggg.eeeee (54g828b85b0bben 9 otw.aaaeeeeeeeeeeeeeeee00000a--- pdf
Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. Journal of Machine Learning Research, 16(52):1731-1755
2015. /ttp:tt::/// http://///....oooo////////////// htt////waaaaaaaaaaaaaaaaaaaaaaae html
Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task bayesian optimiza- tion. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems. volume 26. Curran Asso- ciates, Inc., 2013, URL https: htt.::::/pproceddagss. new.pps......ipi........ wttp.c/ppppp///////..... F33ba 5effa5ef f
pdf
Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective mod- els for effective offline mosel-based optimization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning. volume 139 of Pro- ceedings of Machine Learning Research, pp. 10358-10368. PMLR, 18-24 Jul 2021. URL bitts::/prrceest
ml www.rppressseee 39/ trabuccr2 html
Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Resign-benche Benchmarks
for data-driven offline model-based optimization. CoRR, abs/2202.08450, 2022. URL https:
wlazxiw.org//bs//22222 08450
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL httt:::://proccedinng.neee http://nng..nnrrssccc/////////// pctpsc///ppp///1///eeee..
pdf
James T. Wilson, Riccardo Moriconi, Frank Hutter, and Mare Peter Deisenroth. The reparameteriza- tion trick for acquisition functions, 2017. URL htttht:///axxiitt wttp/bsg/a////////e/ 00424 00424
Sihyun Yu, Sungsoo Ahn, Le Song, and Jinwoo Shin. RoMA: Robust model adaptation for of-
fline model-based optimization. In A. Beygelzimer, Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: //openreview, net fort//ommmm 1d-iididn M.d---CTTTnnUUU
13

---

Under review as a conference paper at ICLR 2023
A NOTATIONS & THEORETICAL ANALYSIS
A. NOTATIONS
For ease of reference, we list the notations in Table
Table 4: Important notations used in the paper
SYMBOL
MEANING
Black box function
Support of f
X
Optima (taken to be maxima for consistency)
Offline dataset
N
Size of offline dataset
Dinaj num_trajs
Trajectory dataset
Number of trajectories in Deraj
T
A trajectory
T
Length of a trajectory
Q
Query budget for black-box function Prefix length
R.
Regret Budget at timestep
R1
Initial Regret Budget
Evaluation Regret Budget
90
Autoregressive generative model with parameters 0
K,T
SORT-SAMPE hyperparameters
NB
Number of bins
C
Context Length
A.2 THEORETICAL ANALYSIS
One crucial component in the SORT-SAMPLE is sorting the trajectories in the increasing order of the function values. Although our primary motivation for sorting derived from the empiri- cal observations from online black-box optimizers (Figure [c). we note that for certain class of functions that are non-trivial to solve from the perspective of optimization (maximization for our paper), lower (higher) function values occupy a larger (smaller) domain. Thus, intuitively we can relate lower function values with exploration and higher function values with exploitation. With this perspective, sorting can be seen moving from high-diversity region to low-diversity region a behavior typically seen in online black-box optimizers (Bijl et al. 2016) Garivier et al. 2016). In this section, try formally prove such properties for this constrained class of functions.
We consider the simplified case of differential 1D functions with certain assumptions for simplicity, and further extend this notion more general D-dimensional case. First, define notion of e-high points.
Definition 1. e-high values. Let the range of f be denoted as [ymin Ymaz). Then, a function value y in this range is e-high if 2 ymin + elymaa Dmin).
Intuitively, the above definition implies that is e-high if it is in the top 1 1 fraction of the range of f. The following result characterizes the relative diversity of regions consisting of e-high points for 1-D functions.
Proposition 1. Let f: [a, + R, a. E R be a real-valued, continuous and differentiable function such that the (a) and f(b) not e-high. Let H C [a, b) be
measuuabbl set of values for which f(x e-high, with e > 0.5. Let H [a, b) \ H. Without loss of generality, let's 's assume that f(a) f(b). the Lipchitz constant L of f is upper bounded by
2(e(ymaz Mmin) + Ymin f(a))
(6)
b
then H < (H%, where denotes volume of w.r.t. the Lebesgue Measure.
14

---

Under review as a conference paper at ICLR 2023
Proof of Proposition| 1
Note that no point in the domain [a, b] achieves e-high function value, then the statement holds trivially true. So, assume that there is atleast one point which has e-high function value. Let I1 and Z2 be the smallest and largest such points in the domain. Since the boundary points doesn't t
have c-high values, Ho (I1 a) (b 22) and H S (22 .1). Thus, if prove that
(z)-a)+ x2) 2 (I2 -2), then we are done. To prove this, we try to prove (æ+ a) z (x2 21).
Assume, on the contrary, that (ar a) (x2 z1). Rearranging, we get
(7)
Now, by the definition of Lipchitz constant, we have:
f(æ1) - f(a)
<
2( (( [(a))
0
L
(8)
D2 - a 2(E * (Umax - Ymin) + Umin - f(a)) S L b b-a
Last inequality holds because 22 completing our proof for Proposition
S
This inequality contradicts the bound 6 L
!
Now, show extension of this proposition for D-dimensional functions with hypercube do- mains.
Proposition Let f X x be a D-dimensional, real-valued, continuous, and differentiable function with hypercube domain X' = [a1, bn] x [0z, ba)...., x bpl. such that none of the boundary points are e-high, for some fixed e. Here by boundary points, we mean the points on the surface of the domain hypercube. Let Yman and ywin be the maximum and the minimum values attained by J. Let H C X be a Lebesgue-measuruue
set of points for which f(x) is e-high. Let H xx= H. If the Lipchitz constant L of f is upper bounded by
2(E (ymaz Ymin) + Umin
max f(a1, (I,I)
"g."
(9)
b
01
then [H| [1%. where]
denotes volume of a set w.t. the Lebesgue Measure.
Proof.
We prove this proposition by induction on the number of dimensions D. Notice that for D the statement reduces to Proposition [ which have already proved. Next, we assume that the statement holds for (D - 11) dimeennsion functions and prove it for D dimensions, with D 1.
Let's define HDD. FD + BD to be functional that maps any D-dimensional function, say f. to Lebesgue-measurabbee subset of dg that corresponds to the set of points where f(x) is e-high. Here, FD and Bp denote the set of all D-dimensional functions and the set of all Lebesgue-meant measurable subsets of RR" respectively.
We similarly define the mapping H. to be functional mapping a function f the complement of its c-high region. Thus, H HD.((f) and H (f). Now, by definition,
HHD.Cf((.. :.).,,)]=-- =
(10)
And similarly,
( - (f dED
(11)
Consequently, "Ho- ,3)D) (/ ",",,,"""""))))," (D)) for every aD (ap, bp).
to prove |ND.lf(....
If(. ,))b
we prove
15

---

Under review as a conference paper at ICLR 2023
To do this, we first fix the Dth dimension to be In other words, dimensional slice of f(11,-.. .DD) with 2D = c. Let be such a slice with g 1( IDD11) =
considering the (D 1)-
f 1 First need e for which e -igh value for g is e-high for f:
e ( Vonnn) vmnn E(ymax y/min) + Mmmin
(12)
where Vinar and Vmni are the minimum and maximum values respectively achieved by g. By 1 this choice of e", we are ensuring that point (I1, " c) is e-high w.r. if and only ( is hh-high w.r.t g. In other words,
1 (g) = -D--I, - 1ee (g) (f. .))
(13)
Let the Lipchitz constant of g L First we show that lg S L.. By definition of Lipchitz constant,
for x = (I1,"
1D-1), " (21, = in the domain of g.
[ -D 1) 9(2 DD11
Lg sup x
D-1
Σ
(x. (x=-2) (m)²
[f c. f(21, ... 22
(14)
sup
x²2
D-1
Σ (z; 2)²² (c c)²
1 Where last inequality is by definition of w.r. f. Combining this with our bound on in 9 we get
2(e= (ymax Yminn) Ymin
max T2, ,ID))
Lg x
22. 3
01
2(E * (Ymaz Umián + Umin max
22,1,22..... .c))
22 2D-1
(fixing tth dimension) (15)
b max
2(c (ylnn yini ) + yn
02, (D-))
22. DD-1
b 01
Thus, the Lipchitz bound assumption is followed by g with = ed, Also, the boundaries are not e9-
high w.r.t g because of the choice of e. This implies, by inductive assumption, that HDD1 (g) < p- (g).. This, combined with equality [3 proves that that [7DD1e((f( ... .))) <
(f (...... (()))),)),,),))) Since this is true for all E [ap, by equations 10 and 11 our proof for (s,, : < (Hs (f),),))))).).)))))))))) is complete.
B EXPERIMENTAL DETAILS
B. SORT-SAMPLE
In SORT-SAMPLE, the score of each bin is calculated according the formula
B.l
( yh,
8;
exp
B
K
The two variables K and here act smoothing parameters. K controls the relative priority given to the larger bins (bins with more points). Higher value K assigns higher relative weight to these large bins compared smaller bins, whereas low value of K the weight assigned to large and small bins would be similar. In the extreme case where K 0, the weight due to [B,1KK will [BB
always be 1, regardless bin size. For very large value of K, the weight will be approximately linearly proportional the bin size [B,]. The later case is not desirable because if there is bin
16

---

Under review as a conference paper at ICLR 2023
As N1=1
N6 16-64
Value
300
270.85
223,06
Best Function Value
230
250
830
200
200
001 Rig
30
Ria
0
00
D'Kitty
D'Kitty
(a) Varying K and
(b) Varying Na
Figure 6: We plot the best achieved function value for different combinations of K, 7 and Nn- As expected, we don't see much sensitivity the choice of K and For NB. expected, we significant decrease for Ng = 1, but NB = 32 is comparable to NB 64
which has very large number of points (which might be low quality bin), then most of the total weight will be given to that bin because of the linear proportionality.
Temperature controls how harshly the bad bins are penalized. Lower the value, lower the relative score of the low quality bins (bins with high regret) and vice versa.
In our experiments, we don't tune the values of K, T, and number of bins NB. In all the tasks, we use K = 0.03 x N and т = Rio. where R10 is the 1hth percentile regret value in D. For Branin task, we use NB g 32 and for the Design-Bence tasks, we use NB 64. Empirically, as we show in Figure 6a we didn't 't observe much effect on K in the range [0.01, 0.1]. and for from the 50th to the 1hth percentiles of R. Figure 6b) shows variation with NB. keeping all other parameters fixed. As expected, NB = doesn't perform well, having just one bin is equivalent having no re- re-weighting. Beyond 32, we don't see much variation with the value of NB.
B.2 MODEL ARCHITECTURE IMPLEMENTATION
Architecture We use a GPT RRdford et al. 2019) like architecture, where each timestep refers to two tokens R₂ and x,. Similar to Chen et al. (2021), we add new learned timestep embedding (in addition to the positional embedding already present transformers). Each token Rg and x, that goes input
model is first projected into 128 dimensional embedding space using linear embedding layer. To this embedding, we also add the positional and timestep embeddings. This passed through causally masked transformer. The prediction head for R2 predicts X,, which is then used to compute the loss. The output of the prediction head for x, is discarded. At each timestep, we feed in the last C timesteps the model, where C' here refers to the context length. For continuous tasks, the prediction head for R, outputs d-dimensional prediction & For discrete tasks, the prediction head outputs a V d-dimensional prediction, where V refers the number of classes in the discrete task. Thus, each dimension in X' corresponds to a V-dimensional logits vector.
Code Our code (available at the anonymized link here) is built upon the code from minGPT and Chen et al. 2022 All code we use is under the licence.
Training The parameter details for all the tasks are summarized in the Table 5 Note that almost all of the parameters are same across all the Design-Bench tasks. Number of Tayers is higher for batch size of 128
continuous tasks, they are of higher dimensionalities. For all the tasks, and fixed learning rate 10 for 75 epochs. All training is done using 10 Intel(R) Xcon(R) CPU cores (E5-2698 v4443 @ 2.20GHz) and one NVIDIA Tesla V100 SXM2 GPU.
httpas//giithbb..com/////////
17

---

Under review as a conference paper at ICLR 2023
B.3 EVALUATION
For all the Design-Bench tasks (except NAS), we use a query budget Q - 256. For NAS, we use query budget of Q 128 due compute restrictions. Since prefix length of 64, this means that we can roll-out four different trajectories. There are two variable parameters during the evaluation: Evaluation RB (R) and the prefix sub-sequence. We empirically observed that ft has more impact the variability of rolled-out points compared to prefix sub- sequence. Hence, we roll-out trajectory for 4 different low R values (0.0, 0.01, 0.05, 0.1). These values are kept fixed across all the tasks and are not tuned. They are chosen to probe the interval [0.0, 0.1], while giving slightly more importance to the low values by choosing 0.01. Evaluation strategies with lower query budget Q available are discussed in the section
a trajectory length of 128 and
Table 5: Important parameters for all the tasks
TASK
Type
HEADS
LAYERS
T
NB
num_trajs
Branin (toy)
Continuous
8
64
32
32
32
400
TFBind8
Discrete
8
128
64
800
TFBind10
Discrete
8
8
128
64
64
64
800
ChEMBL
Discrete
8
128
64
64
64
800
NAS
Discrete
8
128
64
800
D'Kitty
Continuous
32
128
64
64
64
800
Ant
Continuous
32
128
64
64
64
800
Supervonductor
Continuous
32
128
64
64
64
800
B.4 FIXED VS. UPDATED RB DURING EVALUATION
During the evaluation of the prediction sequence, we proposed to keep the Regret Budget (RB) fixed. One alternative strategy is to sequentially update RB after every iteration, i.c. predict x; from Ri, and compute Re+1 R4 - (f(x*) - fxx)).. However, there are two issues with such an update rule:
1. Updating RB adds a sequential dependency on our model during evaluation, as we must query f(xx) to compute R4+1- Thus, generating the Q candidate points is not purely offline.
2. While updating the regret budget RG it is possible that some timestep Rg becomes negative. Since the model has never seen negative RB values during training, this is unde- sirable.
Hence, we do not update RB during evaluation, and instead provide fixed R value at every timestep after the prefix length. This way, point proposal is not dependent on sequential evaluations of f. making it much faster the evaluations f can then be parallelized. Furthermore, by not updating R we sidestep the issue of negative RBs. Empirically, we see in Figure 7 there is not much difference across different strategies, which justifies our choice of not updating RB, allowing our method to be purely offline.
B.5 BASELINES
For the gradient ascent baseline of Branin task, train 2 layer neural network (with hidden layer of size 128) as forward model for 75 epochs with a fixed learning rate of 10 10-4 44 For gradient ascent the learnt model during evaluation, we report results with step size of 0.1 for 64 steps. We average over seeds, and for each seed we perform two random restarts.
For the baselines in the Design- Bench tasks, 2022) and report results with the default parameters for a query budget of 256.
the baseline code F provided in Trabucco et al.
We were not able reproduce the results of Fu & Levine 2021 and YU et al. 2021 of Bench.
the latest version
18

---

Under review as a conference paper at ICLR 2023
update
Update
285.32
Update (capped)
Best Function Value
250
150
100
50
-02
D'Kitty
Figure 7: Comparison between 3 strategies: (1) do not update the RB at all (blue), (2) update the RB but do not handle the case when R, becomes negative (orange) and (3) don't make the update if the update going to make RB negative and update otherwise (green), In all three cases, we see similar performance.
B.6 DESIGN-BENCH BENCH TASKS
For the Design-Bench Bench tasks, we pre-process the offline dataset to normalize the function values before constructing the trajectory dataset Duraj- Normalized Vnorm values computed Ynorm y-Wmin where Ymin and Ymazz are minimum and maximum function values of much larger Ymos Umnin.
hidden dataset. Note that only use the knowledge of Ymin and Ymar from this hidden dataset,
and not the corresponding values. With this normalization procedure, we use 1.0 as an estimate of
f(x) while constructing trajectories Dusj The results we report in Table 2 are also normalized
using the same procedure, similar to prior works (Trabucco et al. 2022 202 unnormalized results Table
We also report
Table 6: Task details
TASK
SIZE
DIMENSIONS
TASK MAX
TFBind8
32898
8
1.0
IFBind
10000
10
2.128
ChEMBL
1093
31
443000.0
NAS
1771
64
69.63
D'Kitty
10004
56
340.0
Ant
10004
60
590.0
Superconductor
17014
86
185.0
B.7 HOPPER TASK
We didn't include the Hopper task in our results offline dataset values and the oracle outputs. Hopper data consist of 3200 points, each with 51 26 dimensions. The lowest and highest function values are 87.93 and 1361.61. Figure g shows the distribution of the normalized function values. This distribution is extremely skewed towards low function values. Only 6 points out 3200 have normalized function value greater than 0.5.
of the inconsistency between the
We noticed that the oracle of Hopper is highly inaccurate for points with higher function values. Figure ? shows the function values in the dataset vs. the oracle output for the top 10 best points in the data, clearly showing the inconsistency between the two. In fact, the oracle minima and maxima for the dataset are just 56.26 and 786.78 79, respectively, far from the actual dataset values. Due to such discrepancies, we have decided not include the Hopper task in analysis.
19

---

Under review as a conference paper at ICLR 2023
1400
Dataset Values
Unnormalized Function Value
100
Oracle Mean Value
1200
Frequency
200
1000
150
800
600
50
Normalized Function Values
id
Figure 8: Histogram of normalized function val- in the Hopper dataset. The distribution is highly skewed towards low function values.
Figure 9: Dataset values vs Oracle values for top 10 points. Oracle being noisy. show and standard deviation over 20 runs.
C ADDITIONAL EXPERIMENTAL RESULTS
C.I ABLATIONS ON SORT-SAMPAR STRATEGY
SORT-SAMPLE algorithm has two main components: Our sorting heuristic is primarily motivated by typical runs of online optimizers. To show this, we run an online GP to optimize the three synthetic functions, namely the negative Branin, negative Goldstein-Price and negative Six hump camel functions and plot the function values for the proposed points. Figure shoss sample trajectories of the function values of the proposed maxima after each function evaluation. We can see, on average, the function values tend increase over time as the number of queries increases, Such behavior has also been reported for other black-box functions and setups, see e.g. (Bijl et al. 2016
Sampling after re-weighting and sorting.
MFY
Function Value
-0.1
-0.2
-0.3
-0.4
-0.5
Branin SixHupCaamll
-0.6
GoldsteinPrice
15
Horizon
Figure 10: Mean and standard deviations of 10 trajectories unrolled by simple GP-based BayesOpt algorithm on the synthetic functions.
However, in this section, we do perform ablations to see the effects these components. To this end, we construct trajectories using 4 strategies:
Random: Uniformly randomly sample a trajectory from the offline dataset.
2. Random + Sorting: Uniformly randomly sample trajectory from the offline dataset and sort it in ascending order of the function values.
3. Re-weighting + Partial Sorting: Perform re-weighting, uniformly randomly sample n, number of points from each bin, and concatenate them from lowest quality bin to the high-
20

---

Under review as a conference paper at ICLR 2023
est quality bin. This way, the trajectory will be partially sorted, i.e. the order of the bins themselves is sorted, but the points sampled from a bin will be randomly ordered. this case, the trajectories not entirely monotonic w.r.t the function values. Intuitively. this intermingles exploration and exploitation phases within and across bins respectively.
4. Re-weighting + Sorting (default in BONET): Sort the trajectory obtained strategy 3. This is the default setting
in our experiments.
Figure 11 contains the results obtained by each of the four strategies. Note that while going from strategy to 2, we keep the points sampled a trajectory the same, so the only difference be- tween them is sorting. Figure [T shows that strategy clearly outperforms strategy 2. This means that sorting has a significant impact on the results. Next, note that strategy 2 and 4 differ only in their sampling strategy, and strategy outperforms strategy 2, which shows the effectiveness of re- re-weighting. This experiment justifies our choice for both re-weighting and sorting.
500
Random Sorting
Best Function Value
400
Re-weighting + Partial Sorting
Re-weighting Sorting (default)
285.11
270.85
254.45
256.7
200
100
D'Kitty
Figure 11: Results with various trajectory construction strategies for D' Kitty task, averaged over runs. Comparing blue and orange bars, evident that sorting is improving the results. Similarly, comparison of orange bar with red bar shows that re-weighting further improves the results.
C.2 ANALYSIS ON QUERY BUDGET Q
So far, we have been discussing the results with query budget Q = 256. Here, we describe the evaluation strategy use when lower query budget is available. Our strategy will be to give higher preference lower Å values when lower budget is available. For example, when Q 192, only roll-out and evaluate for R (0.0, 0.01, 0.05). For 192 Q S 256, we will roll-out for R (0.0, 0.01, 0.05, 0.1}, evaluate the entire predicted sub-sequences of lengths 64 for {0.0, 0.01, 0.05). and evaluate the first Q 192 points in the predicted sub-sequences for fi 0.1. In the Figure 12 we present the results for different query budgets for our method compared to important baselines, for the D' "Kity task. We outperform the baselines for almost all the query budget values.
C.3 ADDITIONAL ABLATIONS
Here we present ablations similar to Section 3 on the D' D'Kitty task, and observe similar trends to what we see in the Branin ablations.
C.4 EFFECT OF PREFIX SEQUENCES
During the evaluation, the unrolled trajectory depends on two factors affecting the unrolled output: Evaluation Regret Budget î and the prefix sequence. Empirically, we found that f has a larger impact the unrolled trajectory than the prefix sequence. To show this, first evaluate the Branin task for 10 different randomly sampled prefix sequences for fixed R and then do the same with
21

---

Under review as a conference paper at ICLR 2023
Best Function Value
200
100
BONET
COMs
MINs
Grad. Ascent
32
120 192256
Query Budget Q
Figure 12: Results for various query budget values Q for D'Kitty task, averaged over runs. We match or outperform other baselines on almost all the values of Q
200
Function Value
160
Function Value
-200
-372
-90
dataset trajectory
dateset trajectory
-238
-200
122
Timestep
Timestep
(a) P = 64, without updating RB
(b) P 64, with updating RB
Figure 13: Figures 13a and 13b show plots of the trajectories generated on DKitty for different values of evaluation RB (0, 8 and 10). In Figure [3a we show results without updating RB, and in Figure 13b we show results with updating. All the trajectories are averaged over runs.
Cumulative Regrat
240
Function
15
220
200
10
Obtalnee
180
Dacaset Minimum
360
Dataset Maximum
140
15
Initial Rearet Budaet
Additional Recret Budaet
(a) Initial RB vs Cumulative Regret
(b) Effect of
Figure 14: Ablations on D'Kitty, averaged over runs.
10 different Å values sampled from the range (0.0, 0.5) for the same prefix sequence. Figure 15 shows the standard deviation of the minimum regret of the 10 different unrolled trajectories for
22

---

Under review as a conference paper at ICLR 2023
Table Unnormalized 100th percentile results
BASELINE
TFBINDS
TFHIND10
SUPERCONDUCTOR
D'KITTY
CHEMBL
NAS
(DeA)
DU00G32
46300000000
ChAS
0458 018---100 0.018
0.301 0.01-1 0.067
85.178
15.572
168.711 905
215.917 19.965
560000000000 5000000
60.355 02.79
GP-E1
0.821 0.285
0.655 0.015
82,888 3.944
480,019 5-0.000
213.816:t: 0.0000
387950 0:0 0.000
60 00.2 2.0.59
UMM ES
01333 11023
0.548 0.136
[8.82] 4: 0.961
1016. 4mn 905 407
4,700 1,2000
(012 40000000
69 473 0..70
Graãies Ascenn
0.981 2.0.000
0.770 0.1166
95.252
3: 0.886
-54.356 4: 662
226.491 22.120
200066
000 200000000
68.772 000000
REINFORCE
00550:: 1.0013
0.002 0.113
85.027 2: 5.058
-151,007 41,303
-301 864 246.284
358400,
21000.000
20.724 2.0000
000:7
0:770
3.217
277-0775
Std. deviation of minimum regret
std. wrt prefix seq.
0.008
std. wrt Evaluation RB
0.007
0.006
0.005
0.002
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Figure 15: Standard deviation of the minimum regret of the unrolled trajectories with 10 different prefix sequences for fixed R and different R for fixed prefix sequence.
fixed it and prefix sequences, respectively. The standard deviation for the variation in prefix length consistently lower than that for the variation it explaining our choice of spending query budget on different R rather than different prefix sequences.
C.5 EFFECT OF ESTIMATING Yиex
A key assumption of method is the knowledge of Vmax- Though in many problems this is not issue, there are many other problems where the value of ymax may not be known. A simple solution could be just estimate Ynax- In Figure 16 we evaluate BONET on D' Kitty multiple varying values of Vmax starting from just beyond the dataset maxima. We find that the value ymaz initially affects performance alot, but beyond point, plateaus..
250
240
function
500
550
Yeax
Figure 16: Best points for different values of Umax D Kitty.
23

---

Under review as a conference paper at ICLR 2023
Table 8: Results for when random 2% subsection of the offline dataset was withheld during training from BONET
10% %
50 %
90%
99%
D (best)
74.21 286.60 + 1.47
74.20 284.74 23.68
73.99
65.71 241.17 = 18.07
BONET
274.11 7.57
Table 9: Results for when the top 2% of the offline dataset was withheld during training from BONET
10%
50%
90 %
99% %
(best)
61.14
-40.40
-545.36 211.56 + 16.74
-548.89 193.27 + 5.51
BONET
267.68 + 2.28
261.04 + 28.09
C.6 ABLATION ON MODEL PARAMETERS
In this experiment we study the effect of changing the number parameters BONET. We do this by altering the number of layers and heads in BONET on D' D'Kitty. We find that increasing the number of parameters helps up to a point, beyond which the model over-fits.
important to note that we present this study only to understand the impact of model size our performance. We don't 't actually tune over these parameters in our experiments. They are kept fixed across all the discrete and continuous tasks (refer to Table 5
Abdetias namberr lyen Crty
Verying the number heads Diiic
Mater Eayers
282
Karber boak
187%
2850
280
2825
272
233.
272.5
272.0
(a) Varying the number of layers in BONET. (b) Varying number of heads in BONET. Lay- ers are fixed at 16
Heads are at
We show the performance of various models with differing number of layers and heads on D' D'Kitty to see their effect on BONET. We find that increasing the number of parameters helps upto point, beyond which it overfits.
Figure 17:
C.7 ABLATIONS DATASET SIZE
To test the limits of BONET we run experiments where we withhold offline training data from BONET and evaluate the performance. We have two settings, one where we withhold an x% size random subsection of data in Table & and another where we withhold the top percentile of data during training and evaluation we don't see as sharp of a drop off in performance compared to when we withhold the good points in the dataset. This leads us to believe that the dominating effect is not the size of the dataset exactly, but the quality of points in the dataset. Further, note that even here the points proposed are significantly larger than the maximum point in the dataset, which rules out the possibility of memorization for BONET.
Table 9 We see that with just reducing the number of points,
24

---

Under review as a conference paper at ICLR 2023
C.8 NOISE ABLATION
We rum experiment where we add progressive larger amounts of noise to the y values in our offline dataset while training our model, to test how robust BONET is to noisy data. We report the results in Table foo D' Kittt. We find that, expected, increasing noise reduces performance, and BONET is in fact reasonably robust to noise, and the largest drop-offf occurs when the magnitude of noise is equal to the magnitude of values.
Table 10: Ablation on adding various magnitudes of noise training data
NOISE SCALE
SCORE 0% of max
285.110
2% of max
279.746
20% of max
255.925
100% of max
137.485
C.9 RANDOM BASELINE
One might argue that BONET just memorizes the best points in the offline dataset and outputs random points close to those best points during evaluation. To rule out this possibility, we perform a simple experiment for the D "iitty task. We choose a small hypercube domain around the optimal point in the offline dataset and uniformly randomly sample 256 points in that domain. In Table [1 we show the results for different widths of this hypercube. width means only the best point in the dataset.
For smaller hypercubes around the best points in the offline dataset, we see that the best point found by 256 random searches is roughly 225, which significantly lower than what BONET finds (291.08). For larger hypercubes, the points are highly diverse. These observations suggest that this optimization problem cannot be solved by just randomly outputting points around the best point in the dataset. If we look at the 256 points output by BONET, they are consistently good (mean is 220), with comparatively very low variance. This suggests that BONET is not simply outputting random points around the best points in the dataset.
D ADDITIONAL ANALYSIS
D.1 VIZUALIZATION OF PREDICTED POINTS
Here we try to visualize the predicted points of BONET compared to the points in the offline data to study the nature of the points proposed by the model. As shown in Figure [8] BONET generalizes well on the unseen maxima regions of the function and produces low regret points.
D.2 ACTIVE GP EXPERIMENT
We also run experiment to compare BONET with active BBO method. Namely, compare BONET with active BayesOpt, using the same GP prior and acquisition function Improvement) as mentioned in Section 3 The difference between the active method and the offline method we compare with in Table 2 is that while the active method directly optimizes the ground truth function, the offline method first trains a surrogate neural network on the data, and then per- forms bayesian optimization on the surrogate instead of the ground truth function. This is done to make the BayesOpt baseline fully offline, and is the same procedure followed by Trabucco et al. 2022 2021). Note that this would result in unfair comparison since the method is both online and queries the oracle function resulting in using more queries than our budget. As shown in Table 12 We find that using oracle actually doesn't 't necessarily improve performance across all tasks, and there are other tasks where the performance doesn't 't change at all. And our model does outperform the oracle GP-qEI method several tasks.
25

---

Under review as a conference paper at ICLR 2023
Results of using a simple sampling strategy randomly from small hypercube centered around the optima. We find that BONET considerably beats this baseline, indicating that general- ization occurring with BONET is not fortuitous.
Table 11:
Width of hypereube
Max. function Value
Mean function Value
Std. Deviation
0.0
199.23
199.23
0.0
0.005
212.66
190.68
9.28
0.01
222.44
182.13
12.21
0.05
226.10 10
-169.62
331.00
0.1
209.00
368.71
261.37
BONET
291.08
221.00
24.43
dataset points
BONET predictions
-65
-95
-125
8
X
-55
6
-885
245
2-
0
10
Xo
Figure 18: Vizualization of 32 points unrolled by a sample evaluation trajectory of BONET com- pared 2000 points randomly sampled from the offline dataset. Three maxima regions don't contain any dataset points because of the removal of the top 10%-ile of uniformly sampled points, as de- scribed in the section 3.1 However, almost all the unrolled points fall into maxima region, clearly showing the generalization capability of BONET.
Table 12: Comparison with GP-qEI oracle function
TFBIND8
TFBIND10
SUPERCONDUCTOR
ANT
D'KITTY
GP-qBI (active)
0.945 1 0.018
0.922 1 0.231
94.587 1 2.137
480.049 0.000
213.816 0.000
GP-qEI
0.824 1: 0.086
0.675 1 0.043
92,686 1: 3.944
450.049 1 0.000
213.816 1: 0.000
BONET
0.975 + 0.004
0.855 + 0.1
80.84 + 4.087
567.042 + 11.653
285.110 + 13.130
D.3 T-SNE PLOTS ON D'KITTY
We show t-SNE plots DKitty for the datasets of differing sizes, with the removal of randomly sampled 3% of the data (setting one described the previous section). The blue points represent points proposed by model, and the red points represent points in the dataset. In general, blue points do not overlap the red points, indicating that the points proposed by BONET are from a different region.
26

---

Under review as a conference paper at ICLR 2023
(a) 99%
(b) 90%
(c) 50%
10%
Figure 19: We show (SNE plots on DKitty for the datasets of differing sizes with the removal of randomly sampled x% of the data. The blue points represent points proposed by our model, and the red points represent points in the dataset. We find that in general, blue points do not overlap the red points, indicating that the points proposed by BONET are from a different region.
27


Reviews:
[{"review_id": "uHa9LzPgE2p", "reviewer": "Reviewer_oyMp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new generative framework to pretrain the trajectory towards the optimal value of an objective function on an offline dataset, then use this generative model (more specifically, an autoregressive model) in evaluation to select data points so as to find the optimal value of the objective function. To synthesize the trajectories from the offline data, the paper presents a sampling strategy based on a simple heuristic of sorting the function values of the data points in the offline dataset and performing a binning process that weighs more on data points with higher function values. Experiments are conducted on a synthetic function and various real-world problems to understand the performance of the proposed approach.", "summary_of_the_review": "Even though I think the main idea of the paper is interesting and new, I have various serious concerns regarding the rigorousness and soundness of the proposed method, and also the experimental evaluation. Please see my comments above.", "strength_and_weaknesses": "Strengths:\n+ Overall, I think the idea of learning the trajectory towards the optimal value using offline dataset is interesting and might have potential.\n+ The paper's writing is also clear and easy to understand.\n+ The experiments include real-world problems.\n\nWeaknesses:\nI have various concerns regarding the paper. Please see my comments below.\n+ The proposed sorting strategy seems to not take into account the fact that the observed data can be noisy. And the regrets in the trajectory training dataset also assume that we can know exactly the noiseless values of the observed data. But in practice, most of the time, the objective function values will be corrupted by noise. How will noise affect the performance of the proposed method?\n+ The assumption of having an estimation of the objective function's optimal value f(x*) is very strong. How does the performance of the proposed method change w.r.t. this value? The sensitivity analysis conducted in Section C.5 is not enough to understand how this value affects the performance of the proposed method (it's just for one simple problem and the range of this estimated value is limited). Besides, what are the values f(x*) used in all the experiments? I seem to not be able to find this information in the main paper. Note that if the value f(x*) is known, there are other existing works developing new BO methods to use this known value to improve the efficiency of the optimization process, e.g., Knowing the what but not the where in Bayesian optimization by Nguyen and Osborne (ICML 2020).\n+ A big concern is that there is a lack of comparison with warm-start BO methods, e.g., Initializing Bayesian Hyperparameter Optimization via Meta-Learning by Feurer et al (AAAI 2015). These warm-start BO methods also make use of the knowledge learned from similar datasets and then transfer this knowledge into a new optimization problem. It has been shown that these methods can help speeding up the BO process. And note that the learned datasets in these warm-start methods do not necessarily come from the same problem we need to optimize.\n+ In the experiments, the offline datasets are generated from the same problem as the problem during evaluation, so I don't think the task of learning the trajectory towards the optimal value is challenging anymore. For the Branin function, the offline datasets are constructed from 5000 data points in the domain, this is really big, and to me, this offline dataset can already help to learn the objective function quite correctly. For other real-world tasks of Design-Bench, I'm not sure if I missed it but I can't find information regarding the size of the offline datasets of these tasks in the main paper. How much data used in the offline datasets of these tasks?\n+ In Tables 2 & 3, what do 100th percentile comparative evaluation and 50th percentile evaluations mean? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lcwRfJVB-A", "reviewer": "Reviewer_Vfqb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies black-box optimization and proposes BONET, which is a generative pre-trained model from offline datasets. An autoregressive model on fixed-length trajectories is trained, and a sampling strategy is designed to synthesize trajectories from offline data using a simple heuristic of rolling out monotonic transitions.", "summary_of_the_review": "This paper proposes an interesting method to deal with black-box optimization problems. However, it is unclear what real-world applications are suitable where off-line datasets are suitable for black-box optimization, the proposed method did not outperform baselines in the experiments, and some state-of-the-arts were not compared.", "strength_and_weaknesses": "Strength:\n1.\tThe idea to pre-train a generative model is interesting.\n2.\tThe paper is well-structured.\nWeakness:\n1.\tThe setting that offline datasets are available is doubtful.\n2.\tExperiments are not convincing enough to illustrate the effectiveness of the proposed framework.\n3.\tSome statements are unclear and difficult to understand.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GFUzudclin", "reviewer": "Reviewer_GVeM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed BONET, a method to optimize expensive black-box function with offline data. BONET consists of three phases, 1) synthesize trajectories from offline data using a simple heuristic, 2) fit an autoregressive model based on the trajectories and regret budgets and 3) roll out the evaluation to output predictions for the maxima of the black-box function. Empirical study is conducted on one synthetic 2D example and 7 real-world tasks, which showed promising results. ", "summary_of_the_review": "The paper presented a neat idea in maximizing black-box functions using offline data only. Although the proposed method BONET is somehow based on exiting ideas such as importance reweighing and causally masked transformer, the simplicity of the proposed method brings value to be applicable in practice. \n\n", "strength_and_weaknesses": "Strength: \n* The paper is well written and easy to follow. \n* The proposed idea is neat and easy to use in practice. \n\n\nWeakness:\n* The analysis regarding \\hat{R} are for the Branin task, does it hold in general? \nThe optimal value for \\hat{R} seems to be 0 from Figure 4, how much value does it add when fitting the autoregressive model with regret information? \n* In phase one, what is the choice of N_B and how it affects the performance? \n* more clarification in the experimental section may improve the paper. For example, have a table of dimensionality for the 7 real world datasets, clarify the definition of normalised score in Table 2 and Table 3, and correct typos such as ''stitching'' -> ``stitching''. \n* It would be interesting to see more evidence in the benefits of the model in the case of high-dimensional multi-modal scenarios. It is not clear from the experiments that the problems are high-dimensional and multi-modal. \n* how does the model perform when observations are sparse? \n* is there a way to extend to the case when the functional evaluations are noisy?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "23dPFqIlgt", "reviewer": "Reviewer_nSto", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the optimization of a black-box expensive to evaluate function in the low data regime where the goal is to take advantage of offline related datasets. Specifically, when the offline and online data have different distributions, this work proposes to build a a generative model for pretraining a novel black-box optimizer based on the offline data. ", "summary_of_the_review": "This paper addresses and interesting problem. However, comparison to other related works can be improved. Without such comparison, an accurate judgement is hard to make.", "strength_and_weaknesses": "Instead of a  heuristic for transitioning between low and high fidelity, it is interesting to see if the transition can be modeled and learned.\n\nIt is also interesting to see how this approach compares to multi-task learning where different tasks include different data distributions amongst offline and online datasets.\n This approach seems to have commonalities with Thompson Sampling (TS) and batch Thompson Sampling (q-TS) from Bayesian optimization literature. Although this work cites the related papers, a more through comparison/analysis can be valuable.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject